There is a growing consensus favouring the development, advancement, and implementation of a competency-based approach for psychology training and supervision. There is wide recognition that skills, attitude-values, and relationship competencies are as critical to a psychologist's competence as are knowledge capabilities, and that these key competencies are best measured during placements, leaving the clinical supervisor in an unparalleled position of advantage to provide formative and summative evaluations on the supervisee's progression towards competence. Paradoxically, a compelling body of literature from across disciplines indicates that supervisor ratings of broad domains of competence are systematically compromised by biases, including leniency error and halo effect. The current paper highlights key issues affecting summative competency evaluations by supervisors: what competencies should be evaluated, who should conduct the evaluation, how (tools) and when evaluations should be conducted, and process variables that affect evaluation. The article concludes by providing research recommendations to underpin and promote future progress and by offering practice recommendations to facilitate a more credible and meaningful evaluation of competence and competencies.