Frailty is associated with impaired cardiac autonomic nervous system (ANS) regulation, which controls heart rate and heart rate variability (HRV). Studies have shown a strong link between heart rate dynamics and frailty, particularly during physical activity. Frail individuals exhibit reduced heart rate increases during exertion and decreased recovery rates after activity, suggesting diminished cardiac autonomic responses. In this paper, frailty is assessed using HRV during physical activity. This study used a Physionet database entitled "Wearable-based signals during physical exercises from patients with frailty after open-heart surgery." A single lead ECG of patients after open-heart surgery was collected during multiple physical activities. Frailty was assessed using the Edmonton frail scale (EFS), where a score equal to and lower than 5 and greater than 5 were considered non-frail and frail, respectively. QRS peaks during a 6-minute walk test (6MWT) were detected using the Pan-Tompkins algorithm, followed by the extraction of 36 HRV features. Extracted features include statistical features (e.g., SDNN, pNN50), frequency features (e.g., low and high-frequency power), traditional Poincare features (e.g., SD1 and SD2), geometric Poincare features (e.g., local and global co-occurrence features), heart rate asymmetry (e.g., Guzik and Porta index). Multiple classifiers were trained for frailty assessment, including a decision tree and neural network for frailty assessment, and the best model was identified using 5-fold cross-validation. ECG for sixty-seven participants (29 non-frail and 38 frail) were analyzed. A Light Gradient-Boosting Machine (LightGBM) was the best-performing classifier, leading to an F1 score of 85.7% and an AUC of 0.83. Promising results suggest the potential of using HRV during physical activity and machine learning for frailty assessment. Furthermore, the results of this study should be evaluated on independent datasets to validate the generalizability and robustness of the proposed frailty assessment method.