Convergent Validity of 2 Widely Used Methodologies for Calculating the Hospital Standardized Mortality Ratio in Flanders, Belgium

Abstract

Objectives: To assess their construct validity, we compared results from 2 models used for estimating hospital standardized mortality ratios (HSMRs) in Belgium. The method of the Flemish Hospital Network (FHN) is based on a logistic regression for each of the 64 All Patient Refined Diagnosis-Related Groups that explain 80% of mortality and uses the Elixhauser score to correct for comorbidities. (H)SMRs published on the 3M-Benchmark-Portal are calculated by a simpler indirect standardization for All Patient Refined Diagnosis-Related Groups and risk of mortality (ROM) at discharge.

Methods: We used administrative data from all eligible hospital admissions in 22 Flemish hospitals between 2016 and 2019 (FHN, n = 682,935; 3M, n = 2,122,305). We evaluated model discrimination and accuracy and assessed agreement in estimated HSMRs between methods.

Results: The Spearman correlation between HSMRs generated by the FHN model and the standard 3M model was 0.79. Although 2 of 22 hospitals showed opposite classification results, that is, an HSMR significantly <1 according to the FHN method but significantly >1 according to the 3M model, classification agreement between methods was significant (agreement for 59.1% of hospitals, κ = 0.45). The 3M model ( c statistic = 0.96, adjusted Brier score = 26%) outperformed the FHN model (0.87, 17%). However, using ROM at admission instead of at discharge in the 3M model significantly reduced model performance ( c statistic = 0.94, adjusted Brier score = 21%), but yielded similar HSMR estimates and eliminated part of the discrepancy with FHN results.

Conclusions: Results of both models agreed relatively well, supporting convergent validity. Whereas the FHN method only adjusts for disease severity at admission, the ROM indicator of the 3M model includes diagnoses not present on admission. Although diagnosis codes generated by complications during hospitalization have the tendency to increase the predictive performance of a model, these should not be included in risk adjustment procedures.