Human versus machine editing of electronic prescription directions

Abstract

Background: Pharmacy staff are responsible for editing poor-quality and difficult-to-read electronic prescription (e-prescription) directions. Machine translation (MT) models are capable of translating free text from 1 sequence into another. However, the quality of MTs of e-prescriptions into pharmacy label directions is unknown.

Objective: To determine the types and frequencies of e-prescription direction component errors made by an MT model, pharmacy staff, and prescribers.

Methods: A prospective evaluation was conducted on a random sample of 300 patient directions in a test set of e-prescriptions from a mail-order pharmacy. Each row included directions produced by (1) prescribers on e-prescriptions, (2) pharmacy staff on prescription labels, and (3) an open neural MT model. Annotators labeled direction sets for missing direction components, use of abbreviations and medical jargon, and incorrect information (e.g., changing the number of tablets to be taken). The longest common subsequence (LCS) compared the amount of pharmacy staff editing with and without MT.

Results: Out of 279 direction sets labeled, the MT model directions contained no quality issues in 196 (70.3%) samples compared with 187 (67.0%) and 83 (29.8%) samples for pharmacy staff directions and prescriber directions, respectively. The MT model directions contained more incorrect components (n = 23). Median LCS was greater without MT (30.0 vs. 18.5, P < 0.01, Wilcoxon signed-rank test), indicating more editing was needed.

Conclusion: MT could be used to improve the quality of e-prescription directions; however, MT makes high-risk mistakes such as incorrectly predicting the tapering regimen for prednisone. The use of semiautomated MT, where pharmacy staff can review model predictions to detect and resolve quality issues, should be considered to improve safety and decrease total work time compared with current practice. MT has strengths and weaknesses for improving the editing process of the patient directions compared with pharmacy staff alone.