With the rising incidence of cardiovascular diseases, efficient arrhythmia detection from ElectroCardioGram (ECG) signals is crucial. Traditional ML and DL methods for automatic ECG analysis have demonstrated promising results with high accuracy but often suffer from a lack of interpretability and challenges in correctly identifying similar beats and detecting sporadic abnormalities. To address these challenges, we propose a novel heartbeat arrhythmia classification model that combines XGBoost with SHAP (SHapley Additive exPlanations). Our model extracts features from single heartbeat, emphasizing the characteristics crucial for differentiating between similar heartbeat. By addressing data imbalance with SMOTE and selecting features via SHAP, we achieved 98.91% AUC in the AAMI 5-class classification. The contribution of our study is to incorporate interpretability into the heartbeat classification process, enabling clinicians to better understand model decision, which is relevant for exploring the precise diagnosis of these diseases.