Photoplethysmography (PPG)-based atrial fibrillation (AF) detection has shown promising performance in wearable applications. However, most studies still use a binary AF versus non-AF setting, where sinus rhythm and different non-AF arrhythmias are grouped into a single non-AF class. This simplified setup may obscure important sources of misclassification, especially in elderly hospitalized patients with cardiovascular comorbidities. This study investigates how label granularity affects PPG-based AF detection performance and which clinically relevant non-AF rhythm groups are more likely to be misclassified as AF. We analyzed long-term clinical wrist-PPG recordings from 49 elderly hospitalized patients, segmented into non-overlapping 30-s windows and organized into hierarchical 2-, 4-, and 6-class rhythm-label settings. The same data split and training framework were used across label settings. On the full test set, AF AUC decreased only modestly from 0.821 to 0.807 and 0.799 as label granularity increased, whereas Macro-F1 fell from 0.700 to 0.350 and 0.240. AF false positives were concentrated in specific non-AF subgroups, with ectopic-like rhythms and the Couplet-Geminy family showing the highest within-group false-positive rates. These findings indicate substantial variation in AF false-positive susceptibility across non-AF rhythm groups, which is not evident from binary AF versus non-AF evaluation alone.