We describe the design of an automated assessment and training tool for psychotherapists to illustrate challenges with creating interactive machine learning (ML) systems, particularly in contexts where human life, livelihood, and wellbeing are at stake. We explore how existing theories of interaction design and machine learning apply to the psychotherapy context, and identify "contestability" as a new principle for designing systems that evaluate human behavior. Finally, we offer several strategies for making ML systems more accountable to human actors.
Keywords: Applications and Expert Systems; J.4 Applications; H.5.m. Information interfaces and presentation (e.g., HCI); Machine learning; Miscellaneous; I.2.1. Artificial Intelligence; Psychology; Design; Social and Behavioral Sciences; interaction design; mental health; psychotherapy.