Limited research on the reliability of cognitive case formulation suggests cognitive therapists can agree about clients' presenting problems but show poor agreement about the inferential aspects of formulation. There has been no research examining the quality of practitioners' case formulations. This study assessed whether participants with different levels of experience could produce reliable cognitive formulations using a systematic cognitive therapy case formulation method: the J. Beck Case Conceptualization Diagram. As part of continuing education workshops on cognitive case formulation, 115 mental health practitioners were given the same case description and asked to provide case formulations. Inter-rater agreement and agreement with a "benchmark" formulation provided by J. Beck were measured. The results showed that participants were able to agree with each other and with the benchmark on most descriptive aspects of the formulation but rates of agreement decreased for aspects of the formulation requiring greater levels of theory-driven inference. Based on definitions and measurements of the quality of cognitive formulations derived in this study, the quality of formulations ranged from very poor to good, with only 44% rated as being at least good enough. Both reliability and quality of case formulations were associated with levels of clinical experience and accreditation status. Implications for training and supervision are discussed.