LLM-Based Automated Audit of Holter ECG Reports for Guideline Adherence and Quality Control

Mikhail Chmelevsky1 and Margarita Budanova2
1Division of Cardiology, Fondazione Cardiocentro Ticino, 2Federal Almazov National Medical Research Center


Abstract

Interpretation of long-term ambulatory electrocardiographic monitoring reports is a cognitively demanding expert task characterized by high information density, multi-parameter integration and susceptibility to diagnostic, terminological and logical inconsistencies. Manual quality control is labor-intensive and poorly scalable. We developed a web-based platform for automated audit of Holter ECG reports using large language models within a formalized quantitative framework for report quality assessment. The study aimed to evaluate the feasibility of an LLM-based system for automated assessment of Holter ECG report quality with respect to report completeness, adherence to guidelines, internal logical consistency and clinically relevant interpretation errors. The pipeline combined document parsing, section segmentation, semantic extraction of arrhythmological findings, rule-based validation of mandatory report fields and LLM-based expert-level assessment of interpretation quality. For each report i, the number of detected errors was denoted by ni, the presence of at least one error by Ei∈{0,1} and the severity grade of error j by gij∈{1,…,5}. Quality was quantified by the proportion of error-free reports, P0=Nno error/N, mean errors per report, eˉ=(1/N)∑ini, mean errors among erroneous reports, eˉ+=(∑ini)/(∑iEi), and a normalized Quality Index, QI=1−(∑i∑jgij)/Mmax, integrating both frequency and severity of errors. This framework transformed expert qualitative assessment into reproducible quantitative metrics suitable for continuous audit and between-physician comparison. In an automated audit of 100 Holter ECG reports, complete processing and quality evaluation were performed in 3 min 42 s, whereas manual expert review was estimated to require approximately 4-5 h. The dominant error class was diagnostic misinterpretation of arrhythmias (about 60-65% of detected issues) followed by quantitative omissions such as pauses or episode duration (30-35%), terminological errors (25-30%), logical inconsistencies between presented events and final conclusions (20-25%) and technical defects including unreadable ECG fragments (15-20%). LLM-based automated audit of Holter ECG reports is feasible and enables rapid, scalable and clinically meaningful quality control.