As part of the George B. Moody PhysioNet Challenge 2026, our team, HKUSTGZ – ML4H, developed a multimodal framework to predict cognitive impairment from overnight polysomnography (PSG). This task is challenging because varying channel configurations, signal quality, and annotation availability across sites complicate cross-site generalization and create deployment-time modality mismatch. To address these heterogeneities, we mapped site-specific PSG channels to a common representation and segmented recordings into 30-s epochs resampled to 64 Hz. Epochs were encoded using a lightweight convolutional feature extractor, and whole-night temporal structure was modeled with a bidirectional gated recurrent unit (BiGRU). Recording-level representations—formed by combining attention pooling, mean pooling, and recurrent hidden states—were then integrated with demographic and annotation-derived covariates in a supervised prediction framework. We utilized a five-fold ensemble to improve stability. Because conventional random splits can obscure site-related distribution shifts, we additionally evaluated our method using leave-one-site-out (LOSO) analysis to rigorously test cross-site robustness. In five-fold cross-validation, our ensemble achieved a 0.621 mean AUROC. Under LOSO evaluation across the three training sites, it achieved a 0.580 macro AUROC. Although LOSO performance was lower and influenced by strong site size imbalances, it provided a critical stress test for cross-site transferability. On the hidden Challenge data, our submission achieved an unofficial score of 0.565. These findings emphasize that cognitive impairment prediction depends equally on robust PSG representation learning and the explicit handling of site shifts. For clinically meaningful sleep-based risk modeling, rigorous cross-site evaluation is as crucial as the model architecture itself.