This paper introduces our solution (ISIBrno-AIMT team) to the Physionet Challenge 2022. The main goal of the challenge was a classification of heart murmurs from phonocardiographic recordings into three mutually exclusive classes (i.e., present, unknown, and not present) and to determine whether the patient's overall status is Normal or Abnormal. We propose a deep learning method that classifies whether there is a heart murmur in the phonocardiographic recording and also provides heart sound segmentation. Furthermore, the expert feature classifier assesses whether the patient's status is normal or abnormal. Our approach achieved a validation challenge score of 0.709 in the murmur classification task and a score of 9657 in the patient outcome classification task. Our team was ranked as X out of Y teams in the official ranking.