Background: A study by de Oliveira Filho et al. reported a validated set of 9 questions by which Brazilian anesthesia residents assessed faculty supervision in the operating room. The aim of this study was to use this question set to determine whether faculty operating room supervision scores were associated with residents' year of clinical anesthesia training and/or number of specific resident-faculty interactions. We also characterized associations between faculty operating room supervision scores and resident assessments of: (1) faculty supervision in settings other than operating rooms, (2) faculty clinical ability (family choice), and (3) faculty teaching effectiveness. Finally, we characterized the psychometric properties of the de Oliveira Filho etal. question set in an United States anesthesia residency program.
Methods: All 39 residents in the Department of Anesthesia of the University of Iowa in their first (n = 14), second (n = 13), or third (n = 12) year of clinical anesthesia training evaluated the supervision provided by all anesthesia faculty who staffed in at least 1 of 3 clinical settings (operating room [n = 49], surgical intensive care unit [n = 10], pain clinic [n = 6]). For all resident-faculty pairs, departmental billing data were used to quantitate the number of resident-faculty interactions and the interval between the last interaction and the assessment. A generalizability study was performed to determine the minimum number of resident evaluations needed for high reliability and dependability.
Results: There were no significant associations between faculty mean operating room supervision scores and: (1) resident-faculty patient encounters (Kendall τb = 0.01; 95% confidence interval [CI], -0.02 to +0.04; P = 0.71), (2) resident-faculty days of interaction (τb = -0.01; 95% CI, -0.05 to +0.02; P = 0.46), and (3) days since last resident-faculty interaction (τb = 0.01; 95% CI, -0.02 to 0.05; P = 0.49). Supervision scores for the operating room and surgical intensive care unit were highly correlated (τb = 0.71; 95% CI, 0.63 to 0.78; P < 0.0001). Supervision scores for the operating room also were highly correlated with family choice scores (τb = 0.77; 95% CI, 0.70 to 0.84; P < 0.0001) and teaching scores (τb = 0.87; 95% CI, 0.82 to 0.92; P < 0.0001). High reliability and dependability (both G- and ϕ-coefficients > 0.80) occurred when individual faculty anesthesiologists received assessments from 15 or more different residents.
Conclusion: Supervision scores provided by all residents can be given equal weight when calculating an individual faculty anesthesiologist's mean supervision score. Assessments of supervision, teaching, and quality of clinical care are highly correlated. When the de Oliveira Filho et al. question set is used in a United States anesthesia residency program, supervision scores are highly reliable and dependable when at least 15 residents assess each faculty.