An Open-Source Platform for Collaborative Annotation of Physiological Waveforms

Lucas McCullum1, Hasan Saeed2, Benjamin Moody1, Diane Perry3, Eric Gottlieb4, Tom Pollard1, Xavier Borrat Frigola5, Dana Moukheiber1, Qiao Li6, Gari Clifford7, Li-wei Lehman1, Roger Mark1
1Massachusetts Institute of Technology, 2University of Michigan, 3Beth Israel Medical Center, 4Brigham and Women's Hospital, 5Hospital Clinic de Barcelona, 6Emory University, 7Georgia Institute of Technology


Abstract

Aim: Electrocardiographic (ECG) monitoring is becoming increasingly commonplace, with modern ambulatory devices allowing long-term, continuous capture from a broad population. To develop robust algorithms for automated diagnosis and characterisation of medical conditions such as ventricular tachycardia (VT), researchers require high-quality annotations. These annotations typically need to be provided by human experts. Currently there is a lack of freely available, high-quality software to enable collaborative, human annotation of physiological waveforms such as ECG.

Method: We developed a software platform to support collaborative, expert annotation of physiological waveforms. The software enables experts to quickly annotate waveform records using a standard web browser. The software is simple to install, following best practice in Python packaging, and it offers a range of features, including: user management and task customization; a programmatic interface for data import and export; and a leaderboard for annotation progress tracking. Using the platform, we carried out a pilot study to assess the quality of 1,980 VT alarms from several commercial hospital monitors. Four expert cardiac clinicians were recruited and randomly assigned batches of alarms to annotate.

Results: Our pilot project demonstrated the utility of the annotation software and provided important feedback for improvement. Interestingly, of the 1,980 VT alarms in the source data, only 23% were considered “true” VT events by the human experts.

Conclusion: We developed a flexible, generalizable, web-based platform to enable multiple users to collaboratively annotate physiological waveforms. In a pilot study using the platform, we found that a significant proportion of VT alarms in commercial platforms were false positives.