Atrial fibrillation (AF) is a common cardiac arrhythmia with serious health implications. Passive monitoring using photoplethysmography (PPG) is desirable for long-term detection of AF. Deep neural networks (DNNs) show promise, but it requires massive training data with clean label, which is hard to obtain. To address the challenge, in this study, a large-scale dataset is created using PPG signals from hospital monitors, and each PPG signal is automatically annotated by concurrent alarms. However, the labels of collected PPG can be noisy because of the existence of false alarms. Then a novel loss function, cluster membership consistency (CMC) loss, is introduced to handle label noise caused by inaccurate PPG labels. The proposed approach shows superior performance in handling label noise and poor-quality signals. This novel approach is shown to be effective in further improving the model performance. It achieves superior or comparable results when evaluated against five different state of the art robust learning algorithms for noisy labels, while at the same time maintains computational efficiency advantage.