Objective: An Objective Structured Clinical Exam was designed to assess physician's ability to discuss end-of-life (EOL) and disclose iatrogenic complications (DOC) with family members of intensive care unit patients. The study explores reliability and validity based on scores from contrasting rater groups (clinicians, SPs, and examinees).
Methods: Two 20-minute stations were administered to 17 surgical residents and 2 critical fellows at a university-based training program. The exam was conducted, videotaped, and scored in a standardized setting by 8 clinical raters (MD and RN) and 8 standardized families using separate rating tools (EOL and DOC). Examinees assessed themselves using the same tools. We analyzed the internal consistency, inter-rater agreement, and discriminant validity of both cases using data from each rater group. Cross-rater group comparisons were also made.
Results: The internal consistency reliability correlations were above 0.90 regardless of case or rater group. Within rater groups, raters were within 1 point of agreement (5-pt and 6-pt scales) on 81% of the DOC and between 74% and 79% of the EOL items. Family raters were more favorable than clinical raters in scoring DOC, but not EOL cases. Large raw differences in performance by training level favored more experienced trainees (3rd year residents and fellows). These differences were statistically significant when based on residents own self-ratings, but not when they were based on clinical or family ratings.
Discussion: The Family Conference Objective Structured Clinical Exam is a reliable exam with high content validity. It seems unique in the literature for assessing surgical trainees' ability to discuss "bad news" with family members in intensive care.