We present an adaptive fusion framework developed by team CAU_KU for predicting future cognitive impairment from polysomnography (PSG) recordings. PSG captures diverse physiological signals during sleep and offers a valuable opportunity for early screening of neurological conditions. Our method builds upon a pretrained SleepFM backbone trained on large-scale physiological data to learn general multimodal representations across BAS, RESP, EKG, and EMG signals. While the pretrained model captures generic relationships among modalities, the learned fusion may not optimally reflect their relative importance for the downstream task of cognitive impairment prediction. To address this limitation, we propose an adaptive fusion strategy that adjusts modality contributions in a task-specific manner. We further introduce a lightweight gating module that produces sample-specific fusion weights, enabling the model to dynamically adapt to varying signal quality and patient-specific characteristics. This design allows the model to preserve general knowledge from pretraining while emphasizing informative modalities for each input. Using the official Challenge evaluation pipeline, team CAU_KU achieved an AUROC of 0.725 on the unofficial phase leaderboard. The proposed method improves robustness and flexibility by combining pretrained knowledge with task-specific and instance-level adaptation. These results demonstrate that adaptive fusion is an effective strategy for multi-modal physiological modeling.