Quality assessment is the focus of many health care initiatives. Yet it is not well understood how the type of information used in decision support tools to enable judgments of quality based on data impacts the accuracy, consistency and reliability of judgments made by physicians. Comparative pooled information could allow physicians to judge the quality of their practice by making comparisons to other practices or other specific populations of patients. In this study, resident physicians were provided with varying types of information derived from pooled patient data sets: quality component measures at the individual and group level, a qualitative interpretation of the quality measures using percentile rank, and an aggregate composite quality score. 32 participants viewed thirty quality profiles consisting of information applicable to the practice of thirty de-identified resident physicians. Those provided with quality component measures and a qualitative interpretation of the quality measures (rankings) judged quality of care more similarly to experts and were more internally consistent compared to participants who were provided with quality component measures alone. Reliability between participants was significantly less for those who were provided with a composite quality score compared to those who were not.
Keywords: Quality assessment; decision support; judgment analysis; quality improvement.