Correctly recognizing emotions is an essential skill to manage interpersonal relationships in everyday life. Facial expression represents the most powerful mean to convey important information on emotional and cognitive states during interactions with others. In this paper, we analyze physiological responses triggered by an emotion recognition test, which requires the processing of facial cues. In particular, we evaluate the modulation of several Heart Rate Variability indices, collected during the Reading the Mind in the Eyes Test, accounting for test difficulty (derived from a Rasch analysis), test performances, demographic and psychological characteristics of the participants. The main idea is that emotion recognition is associated with the Autonomic Nervous System and, as a consequence, with the Heart Rate Variability. The principal goal of our study was to explore the complexity of the collected measures and their possible interactions by applying a class of flexible models, i.e., the latent class mixed models. Actually, this modelling strategy allows for the identification of clusters of subjects characterized by similar longitudinal trajectories. Both univariate and multivariate latent class mixed models were used. In fact, while the interpretation of the Heart Rate Variability indices is very difficult when considered individually, a joint evaluation provides a better description of the Autonomic Nervous System state.