An Open-Source Platform for Collaborative Annotation of Physiological Waveforms

Lucas McCullum1, Hasan Saeed2, Benjamin Moody1, Diane Perry3, Eric Gottlieb4, Tom Pollard1, Xavier Borrat Frigola5, Qiao Li6, Gari Clifford7, Roger Mark1, Li-wei Lehman1
1Massachusetts Institute of Technology, 2University of Michigan, 3Beth Israel Medical Center, 4Brigham and Women's Hospital, 5Hospital Clinic de Barcelona, 6Emory University, 7Georgia Institute of Technology


Aim: Electrocardiographic (ECG) monitoring is becoming increasingly commonplace, with modern ambulatory devices allowing long-term, continuous capture from a broad population. To develop robust algorithms for automated diagnosis and characterisation of medical conditions such as ventricular tachycardia (VT), researchers require high-quality annotations. These annotations typically need to be provided by human experts. Currently there is a lack of freely available, high-quality software to enable collaborative, human annotation of physiological waveforms such as ECG.

Method: We developed a software platform to support collaborative, expert annotation of physiological waveforms. The software enables experts to quickly annotate waveform records using a standard web browser. The software is simple to install, following best practice in Python packaging, and it offers a range of features, including: user management and task customization; a programmatic interface for data import and export; and a leaderboard for annotation progress tracking. Using the platform, we carried out a pilot study to assess the quality of 1,980 VT alarms from several commercial hospital monitors. Four expert cardiac clinicians were recruited and randomly assigned batches of alarms to annotate.

Results: Our pilot project demonstrated the utility of the annotation software and provided important feedback for improvement. Interestingly, of the 1,980 VT alarms in the source data, only 23% were considered "true” VT events by the human experts.

Conclusion: We developed a flexible, generalizable, web-based platform to enable multiple users to collaboratively annotate physiological waveforms. In a pilot study using the platform, we found that a significant proportion of VT alarms in commercial platforms were false positives.