In this work, we describe an end-to-end deep learning architecture for Heart Murmur Detection from Phonocardiogram(PCG) recordings as part of The George B. Moody PhysioNet Challenge 2022. Our team, "Team_IIITH” received a weighted accuracy score of 0.55 (ranked 28th out of 64 teams) and Challenge cost score of 12345 (ranked 43rd out of 63 teams) on the hidden validation set for our best entry. In our approach, the PCG recordings are first down-sampled to 1000 Hz before being passed through a Butterworth's low and high pass filter to remove baseline wanders and high-frequency noise present in the recordings. The PCG recordings are then broken down into 10-second segments and normalized to bring all trainable samples to the same size. To extract embeddings more efficiently, we built a custom 1-dimensional Residual Network (ResNet) where the 10-second inputs are passed through variable-sized kernel ResNets in parallel, before being concatenated and passed through the next ResNet layer to account for different length dependencies across the PCG signal. The output of this custom ResNet is then fed to a 2-layer feed-forward network for final classification. Cross-Entropy Loss with class weights was employed to account for class imbalance. Our approach obtained a 5-fold Cross-Validation weighted accuracy score of 0.71 and challenge cost score of 12067 on our local test setup.