In order to detect multi-class arrhythmias with high accuracy using multi-lead electrocardiogram (ECG) signals, we propose an arrhythmia classification method based on semantic segmentation. In our framework, ECG signals are firstly filtered and normalized, and divided into 30-second segments. Then, a convolutional neural network (CNN) with different dilation rates is designed to extract and integrate the multi-scale features of ECG signals. Particularly, we apply squeeze-and-excitation blocks to assign weights to features, and heartbeats are finally classified by Softmax function. Aiming at the problem of class-imbalance, the method of overlap between segments is futher adopted to increase the samples, and probability threshold values are set. We evaluate the performance of the proposed method on five public databases. The precision, sensitivity and F1 score for fusion of ventricular contraction and normal beat (F), supraventricular escape beat (AE) and ventricular escape beat (VE) are all over than 90%. The proposed method combines CNN and semantic segmentation could be helpful for automated ECG diagnosis in clinical practice.