Persistent racial and ethnic disparities in access and utilization of behavioral health services have highlighted the need for cultural competence among providers. In response, many agencies are now implementing education and training programs to ensure that behavioral health professionals improve their skills when serving diverse ethnic, racial, and cultural populations. The evaluation of these trainings is vital to ensure that they both improve the cultural competence of providers and promote recovery among persons with severe mental illnesses. This paper discusses the philosophical and practical issues related to measuring cultural competence, based on the evaluation of statewide cultural competence trainings for behavioral health professionals. The evaluation process illustrates the challenges of operationalizing cultural competence, balancing the needs of program implementers and evaluators, and developing a robust and feasible evaluation design, which assesses outcomes both for persons in recovery and providers.