Cardiac auscultation through phonocardiogram (PCG) is still the most commonly used approach for evaluating the mechanical functionality of the heart when diagnosing congenital heart disease. Despite of its time- and cost-effectiveness, it is still limited due to the extensive need for clinical expertise for interpretation. In this study, we propose the use of ensemble transformer-based neural networks to aid in the detection of heart murmur in PCG recordings and for the prediction of clinical outcomes of patients as part of George B.\ Moody PhysioNet 2022 Challenge. Our team, Care4MyHeart, developed an approach that transforms the raw PCG recordings into wavelet power features signals for the use within the proposed deep learning models. We have achieved a maximum accuracy of 0.855 and 0.761 for murmur detection in the training and hidden validation datasets, respectively. In addition, we had an overall clinical outcome cost of 9980 and 11490 for both datasets, respectively. Our team was ranked 12th/305 for murmur detection and 173rd/305 for clinical outcome predictions. We had the lowest clinical outcome cost on the validation set (9737, 41st/305) with a murmur detection score of 0.730 when reducing the number of features used to train the models.