Introduction: The paediatric series of direct observation of procedural skills (DOPS) were introduced into the UK national endoscopy training curriculum in 2016, but lack validity evidence. We aimed to present validity evidence for paediatric colonoscopy DOPS and study competency development in a national trainee cohort.
Methods: This prospective UK-wide study analysed formative paediatric colonoscopy DOPS which were submitted to the e-Portfolio between 2016 and 2018. Item, domain, and average DOPS scores were correlated with the overall DOPS rating to evidence internal structure validity. Overall DOPS ratings were compared over lifetime procedure count to demonstrate learning curves (discriminant validity). Consequential validity was founded on receiver operating characteristic curve analyses.
Results: A total of 203 DOPS assessments were completed for 29 trainees from 11 UK training centres. Internal structure validity was provided through item-total correlation analyses. DOPS scores positively correlated with trainee seniority (P < 0.001) and lifetime procedure count (P < 0.001). Competency acquisition followed the order of: "preprocedure," "postprocedure," "endoscopic nontechnical skills," "management," "procedure" domains, followed by overall DOPS competency, which was achieved in 81% of the cohort after 125 to 149 procedures. Mean DOPS scores could be used to predict overall procedure competence (area under receiver operating characteristic curve 0.969, P < 0.001), with a mean score of 3.9 demonstrating optimal sensitivity (93.5%) and specificity (87.6%).
Conclusions: This study provides validity evidence supporting the use of paediatric colonoscopy DOPS as an in-training competence assessment tool. DOPS may also be used to measure competency development and benchmark performance during training, which may be of value to trainees, trainers, and training programmes.