Aim: Heart murmur detection can provide a preliminary diagnosis of heart disease, and has become increasingly important in assisting clinical diagnosis and treatment in recent years. The purpose of this study is to construct an automatic detection system for heart murmur. Methods: We constructed a learnable filter-based transformer architecture. The learnable filter was embedded between the embedding layer and the encoder layer of the transformer, and its parameters are optimized by the gradient descent algorithm, thereby adaptively reducing noise. Then, the transformer encoder module captures the long-term dependencies of the PCG signal, allowing the network to learn more effective features from the input signal. Results: Our (Bear_FH) method was evaluated using the challenge validation dataset, and we obtained a challenge metric score of 2129 in the unofficial stage. Conclusion: The results show that the method performs well in detecting heart murmur from heart sound recordings.