At present no adequate annotation guidelines exists for incident report learning. This study aims at utilizing multiple quantitative and qualitative evidence to validate annotation guidelines for incident reporting of medication errors. Through multiple approaches via annotator training, annotation performance evaluation, exit surveys, and user and expert interviews, a mixed methods explanatory sequential design was utilized to collect 2-stage evidence for validation. We recruited two patient safety experts to participate in piloting, three annotators to receive annotation training and provide user feedback, and two incident report system designers to offer expert comments. Regarding the annotation performance evaluation, the overall accuracy reached 97% and 90% for named entity identification and attribute identification respectively. Participants provided invaluable comments and opinions towards improving the annotation methods. The mixed methods approach created a significant evidential basis for the use of annotation guidelines for incident report of medication errors. Further expansion of the guidelines and external validity present options for future research.
Keywords: Data Annotation; Hospital Incident Reporting; Medication Errors.