This study investigates wearable estimation of heart rate recovery (HRR) to free-living physical activity in 529 patients with recurrent Major Depressive Disorder (MDD) using data from wrist-worn wearable devices (Fitbit) over a 2-year period. Depression is associated with autonomic nervous system dysregulation and increased cardiovascular risk. Our hypothesis was that HRR would be lower in patients with more severe depressive symptoms, and that surrogate physiological markers derived from wearables could complement clinical evaluations. Heart rate (HR) and step count data were continuously collected from the wearables. Depression severity was assessed biweekly using the Patient Health Questionnaire-8 (PHQ-8). Periods of physical activity were automatically detected from step count data using predefined criteria. To analyze HRR, we applied bivariate phase rectified signal averaging (BPRSA), estimating parameters characterizing HR response to physical exertion for each patient and PHQ-8 score. Univariate analyses did not show statistically significant differences in HRR across depression severity levels consistently, and a multivariate TabPFN model was able to classify patients with and without depressive symptoms with 53.82\% accuracy (AUC = 0.5876). Our results suggest that the relationship between HRR to free-living physical activity and depression severity is not straightforward, although further studies should investigate its potential for monitoring cardiovascular risk in these patients.