Heart murmurs are abnormal sounds heard in heartbeats, caused by turbulent blood flow through the heart. The Physionet 2022 challenge targets automatic detection of murmur from audio recordings of the heart. The objective is to identify individuals with murmur from multiple recordings of multiple locations around the heart. Our participation involves a DNN that classifies murmur vs. non-murmur sounds and a hierarchical classifier that also takes advantage of the available demographic information that is provided for each subject. The DNN is based on the ResNet architecture and is first trained using a self-supervised approach with contrastive loss on publicly available datasets, in an attempt to push the network to extract the underlying structures present in phonocardiograms. We then train the final classification layers using the dataset provided by the challenge. We evaluate both the effectiveness of the trained model in two ways: (a) using standard metrics on audio windows extracted from the recordings of the subjects of the test set, and (b) using the challenge score, as defined by the challenge. Our team, “Listen2YourHeart”, achieved a challenge score of 853 and ranked 41st out of 167 teams in the unofficial phase.