In this paper, we propose a novel multi-stream video classifier for infant needs detection. The proposed system is an ensemble-based system that combines several machine learning to improve the overall result of the state-of-the-art algorithms. It is a multi-stream in the sense that it combines the output predictions of both audio and images of infants from every single classifier employed in the system for a unified result. This produces better performance and results compared to the previous other research techniques, which relied on only one of these modalities. For training and testing the proposed system, from the Dunstan Baby Language video collection, we built three separate datasets for videos, images, and sounds encompassing the five primary infant needs that require predicting. These are: hunger, have wind, uncomfortable (require diaper change), wants to burp or tired, with a total of 3348 samples. We used four different ensemble algorithms for the best reachable performance. The proposed algorithm improves the overall accuracies of each single classifier from a low of 51% to a high of 99%. The proposed method also improves the accuracy of the classification process by about 9% compared to the state-of-the-art approaches, which was 90%.
Keywords: 68T05; 68T07; 68T10; 68U10; Deep learning; Dunstan baby language; Ensemble classifier; Infant needs; Machine learning; Video classifier.
© 2023 The Authors.