Objective: Autism Spectrum Disorders (ASD) are characterized by impairments in joint attention (JA) comprising two components: responding to JA (RJA) and initiating JA (IJA). RJA and IJA are considered two interrelated aspects of JA, related to different stages of infant development. While recent technologies have been used to characterize RJA emerging in earlier childhood, only a limited number of studies have attempted to explore IJA, which progressively becomes evident as a hallmark of ASD. This study aims to achieve the social recognition of both RJA and IJA by vision-based human behavior perception through a multi-modal framework automatically and comprehensively.
Methods: The first three layers of this framework leverage localization, feature extraction, and activity recognition. On this basis, three critical activities in JA are recognized: attention estimation, spontaneous pointing, and showing actions. Then different behaviors are linked through the fourth layer, semantic interpretation, to model the JA event. The proposed framework is evaluated on experiments of four groups: 7 children with ASD, 5 children with mental retardation (MR), 5 children with developmental language disorder (DLD), and 3 typically developed children (TD).
Results: Experimental results compared with human codings demonstrate recognition reliability with an intra-class coefficient of 0.959. In addition, statistical analysis suggests significant group difference and correlations.
Conclusions: The multi-modal human behavior perception-based framework is a feasible solution for the recognition of joint attention in unconstrained environments.
Significance: Thus the proposed approach has the potential to improve the clinical diagnosis of autism by offering quantitative monitoring and statistical analysis.