Cardiovascular disease remains a leading cause of morbidity and mortality worldwide, driving demand for scalable tools that can extract rich cardiac information from widely available diagnostic tests. Cross-modal contrastive learning between electrocardiograms (ECGs) and cardiac magnetic resonance (CMR) imaging enables transfer of structural cardiac information into the accessible ECG modality. However, standard contrastive alignment objectives can impose overly rigid separation between physiologically similar patients, distorting the continuous structure that downstream clinical tasks rely on. We propose Phenotype-Aware Supervised Contrastive Learning (Pheno-SupCon), which instead uses a soft similarity structure derived from continuous cardiac phenotypes and categorical diagnoses to better preserve clinically meaningful relationships. We align ECG representations to a CMR target space constructed from seven clinical phenotypes from the UK Biobank (~33K paired subjects). Pheno-SupCon achieves statistically significant improvements over standard contrastive alignment in four of five regression targets, while maintaining comparable classification performance. Full finetuning experiments at reduced fractions of training data further demonstrate that phenotype-aware representations yield consistent gains in low-data regimes for structural phenotype regression.