Aims: Early identification of cognitive impairment from routine sleep studies could support timely intervention and improved long-term outcomes. Inspired by Andrew Karpathy's autoresearch approach, we aimed to develop an agentic machine-learning workflow to predict cognitive impairment from polysomnography-derived metadata and algorithmic sleep annotations, with explicit emphasis on cross-site robustness.
Methods: Our pipeline has three components: (1) an expert-curated feature pool combining demographics and CAISR-derived sleep/event descriptors, (2) an autonomous experimentation environment powered by ChatGPT 5.3-Codex and Hermes agent architecture that proposes and evaluates iterative model changes under strict leakage controls, and (3) a deterministic evaluation harness designed to approximate deployment conditions by leave-one-site-out (LOSO) validation coupled with AUC and challenge metrics on the unaltered and undersampled train data. We evaluated both a compact CAISR-driven classifier stack and a survival-aware dual-head architecture that fuses binary risk estimation with time-to-event regression.
Results: Across iterative development cycles on the public training cohort (sites S0001, I0002, I0006), the current best submission-track model achieved a LOSO macro-AUROC of 0.737 with a Challenge Score of 0.071. In the survival-aware experimental track, the latest 100-iteration agentic search achieved a balanced LOSO macro-AUROC of 0.654, balanced Challenge Score of 0.071, imbalanced macro-AUROC of 0.665, and imbalanced macro-TPR of 0.737, with reduced overfitting versus earlier iterations. The most influential features were arousal probability variability, REM proportion, limb-event load, and arousal index.
Conclusion: We presented an autoresearch methodology that couples expert-chosen sleep features with autonomous, rigorously logged experimentation under site-robust validation. This framework enables continuous, systematic model refinement while preserving a transparent research protocol. Team SECVC; unofficial-phase leaderboard score: 0.654 AUC.