Using Consumer Camera and Custom Firmware to Monitor Heart Rate in Terminally Ill Children During Music Therapy

Maurice Rohr1, Monika Hoog Antink2, Sebastian Dill1, Christoph Hoog Antink3
1Technische Universität Darmstadt, 2Hochschule für Musik und Theater, 3TU Darmstadt


Abstract

Background: Photoplethysmography Imaging (PPGI) is considered a prospective non-contact replacement for existing methods of heart rate (HR) and HR variability monitoring. However, "in the wild" studies are rare. Music therapy for children with severe disabilities aims to alleviate pain and increase their wellbeing. As they are not able to communicate via spoken language, changes in facial expressions and vital signs may be used to quantitatively assess effects of music therapy. Here, cameras can be used to capture both, hints of the emotional expressions and, via PPGI, HR. From a technical perspective, the high variability in HR, the interaction with the therapist, as well as natural lighting form an ideal scenario to stress-test existing algorithms. Methods: We used a consumer camera with custom firmware to record raw videos of children during music therapy. For face extraction and skin segmentation we use the pre-trained deep learning models BlazeFace and Deeplab respectively. To enforce temporal coherence, we apply temporal filtering to the skin masks inside a 5-frame window. The PPG signal is extracted with CHROM. We use ridge detection to estimate the HR from the spectrogram in the 0.4 to 4 Hz range in 30 second sliding windows. We evaluate the estimated HR using a reference ECG. Preliminary results: For an initial subset of the data we achieved a mean absolute / root-mean-squared error of 1.62 / 2.31 BPM and a correlation of 0.94. These values improved to 1.56 / 2.17 and 0.95 by applying temporal filtering to the skin mask. Conclusion: PPGI methods show promising results for music therapy and can provide accurate HR over a wide range. Enforcing temporally coherence of skin segmentation may be an efficient method to improve PPGI estimation. All results are preliminary for few subjects but will be extended to a larger population.