We investigated several modeling strategies using structured CAISR annotation features and multimodal SleepFM token embeddings derived from overnight PSG. Early tabular baselines based on demographics and CAISR-derived sleep summaries showed that apparent performance on random local splits could be misleading when demographic missingness and site composition were not controlled. On the official hidden test set, a random forest using demographics achieved AUROC 0.423, and adding CAISR features after removing BMI-dependent shortcuts improved AUROC to 0.473, but both remained below the sample submission baseline of 0.501. These analyses revealed clear site differences across the three local sources and motivated site-aware validation as a core robustness check.
We then extended CAISR modeling beyond summary statistics to rhythm-based features computed directly from raw annotation EDFs, including stage proportions, transition rates, fragmentation, uncertainty, and event burdens for arousal, respiratory, and limb activity. This engineered feature track achieved repeated-CV AUROC/AUPRC of 0.575/0.572 with elastic net, and site-aware AUROC/AUPRC up to 0.630/0.632 with histogram gradient boosting. However, performance on the largest held-out site remained near 0.55 AUROC, indicating persistent domain shift.
In parallel, we evaluated SleepFM-based sequence models using 5 s tokenization with hierarchical aggregation, alongside lighter hierarchical attention-pooling and CAISR-only sequence variants. Although hierarchical non-recurrent pooling was more stable than GRU-based sequence models in local validation, no learned sequence model consistently exceeded the best engineered CAISR features on the dominant held-out site. On the official hidden test set, our current SleepFM submission achieved AUROC 0.500, again slightly below the sample baseline. While we already incorporated SleepFM tokens into our sequence models, we have not yet systematically analyzed the embedding space itself. A key next step is to investigate the embeddings more directly and pursue stronger data harmonization to reduce site bias and improve cross-site generalization.