Accurately assessing the severity of pulmonary edema is critical for making treatment decisions in congestive heart failure patients; however, the current scale for quantifying pulmonary edema based on chest radiographs does not have well-characterized severity levels, with substantial inter-radiologist disagreement. While the severity score documented in the radiology report may not be accurate, there are other pieces of information that may be more reliable: specifically, the status comparisons that radiologists make between the current radiograph and the previous radiograph in the series. In this study, we investigate whether those comparisons can provide accurate characterizations of pulmonary edema progression. We propose a rules-based natural language processing approach to assess the change in a patient's pulmonary edema status (e.g. better, worse, no change) by performing pairwise comparisons of consecutive radiology reports taken over the course of their hospital stay. We apply regular expressions and heuristics based on clinical knowledge to assign comparison labels, then use the labels to derive comparisons between pairs of consecutive reports. Evaluated against ground-truth labels from expert radiologists, our labeler extracts comparisons describing the progression of pulmonary edema with 0.875 precision and 0.891 recall. We also demonstrate the potential utility of comparison labels in providing additional fine-grained information over noisier labels produced by models that directly estimate severity level. Our hope is that the results presented here can assist researchers in developing more accurately labeled datasets for modeling, better inform radiologists on the characteristics of different pulmonary edema severity levels, and improve treatment decision-making for pulmonary edema patients.