Music is known to modulate autonomic nervous system activity but determining whether these responses are consistent and stimulus-specific across individuals remains challenging due to inter-subject variability in response and temporal alignment. We investigated whether short-term heart rate variability and respiratory signals encode reproducible, stimulus-driven structure across listeners exposed to controlled musical stimuli. Physiological responses from 110 participants listening to 30 classical piano excerpts were represented as 56-dimensional vectors derived from seven autonomic markers (HR, RMSSD, LF, HF, LF/HF, respiratory rate, respiratory amplitude), summarized using distributional statistics and normalized per participant to remove baseline differences. Music stimuli were encoded as 72-dimensional representations derived from nine acoustic features, including tempo, loudness, and spectral and onset descriptors, similarly summarized over time. As autonomic responses can be slow, we first evaluated stimulus–response distributional coupling within participants using Earth Mover's Distance (EMD). Responses were closer to their true stimulus than to random pairings (88.7% vs 50%, p < 0.001), indicating reliable stimulus specificity. We then trained a contrastive encoder (InfoNCE loss) to align physiological and acoustic representations across participants, achieving 0.906 accuracy on a match–mismatch task (chance 0.495 ± 0.021, p = 0.005). Stimulus features explained 14.1% of physiological variance (η² = 0.141), with within-stimulus similarity significantly exceeding across-stimulus similarity (cosine Δ = +0.141, p < 0.001). Participant discrimination remained at chance (0.50), confirming the model captures shared rather than individual-specific responses. Feature ablation identified respiratory amplitude, LF power, HF power and LF/HF ratio, as dominant contributors indicating autonomic engagement with stimuli. Some tracks characterised by higher tempo and onset density showed reduced HF and RMSSD and increased LF/HF ratios, suggesting parasympathetic withdrawal. These findings show that music induces weak but reproducible autonomic signatures driven by stimulus identity rather than individual physiology, supporting the use of controlled auditory stimuli for probing autonomic function.