Inter-rater reliability of assessments regarding the quality of drug treatment, and drug-related hospital admissions

Abstract

Aims: To investigate inter-rater agreement on the quality of drug treatment, and the relationship between the drug treatment and hospital admission.

Methods: Three specialist physicians and two resident physicians determined, independently and in consensus, the quality of drug treatment from an overall medical perspective, and its association with admission, in 30 randomly selected patients (50% female, median age 72 years) admitted to Sahlgrenska University Hospital, Sweden, in April 2018. The inter-rater agreement was evaluated with Gwet's agreement coefficient (AC₁ ).

Results: In all, 200 (95%) out of 210 drugs at admission and 238 (97%) out of 245 drugs at discharge were assessed as reasonable drug treatment by all assessors. Conversely, none of the drugs at admission, and two at discharge, were assessed as unreasonable drug treatment by all assessors (AC₁ : 0.88 and 0.94 [all], 0.86 and 0.95 [specialists], 0.92 and 0.92 [residents], respectively). The assessments regarding the association between the drug treatment and the hospital admission (not related or main/contributory reason) were consistent between the assessors for 16 out of 30 patients (AC₁ : 0.67 [all], 0.74 [specialists], 0.54 [residents]). In none of the three cases where the hospital admission was considered possibly attributable to a prescribing error did the assessors make consistent assessments.

Conclusions: As the inter-rater agreement ranged between weak and almost perfect, the reliability of assessments of drug treatment quality, as well as adverse consequences, appears to be a methodological concern. To yield acceptably reliable results regarding both drug treatment aspects at issue, specialist physicians should be involved.

Keywords: adverse drug reaction; inter-rater agreement; pharmacotherapy; prescribing quality.