The automated detection of murmurs in the heart sounds has played an important role in the diagnosis of cardiovascular diseases. This paper has proposed an ensemble deep learning framework, which unifies the 1D CNN and 2D CNN to extract discriminative features from time domain and frequency domain, to identify the presence of the abnormal heart murmurs in the phonocardiogram (PCG) signals. First, in consideration of different lengths of the heart sound recordings and the excessive sampling rate, a temporal sampling block is used to align the time durations and down-sample the raw PCG signals to a low sampling frequency. Second, two different branches of CNN are used to learn multiple levels of representations. One branch of the 1D CNN with different kernel sizes is used to extract the multi-scale time-domain features from the preprocessed PCG signals, while the other branch of 2D CNN is used to extract features from the time-frequency maps transformed from the PCG signals. In particular, a location-aware block is designed for multi-auscultation location recordings in each branch. Then the extracted features from the two CNN branches as well as features from some demographic and clinical information (like age, gender, and so on) are concatenated together as the final feature representation. At last, a multi-layer perceptron (MLP) is used as the classifier to identify whether there are heart murmurs in the PCG signals or not. By splitting the official public dataset into 80%-20% stratified training-test sets, the proposed method has achieved a 5-fold cross-validation score of 1556 using the Challenge burden score metric. Our team, BioInfo_ML_Yan, received a Challenge score of 2212.409 using the Challenge evaluation metric on the validation set so far.