Serious games are receiving increasing attention in the field of cultural heritage (CH) applications. A special field of CH and education is intangible cultural heritage and particularly dance. Machine learning (ML) tools are necessary elements for the success of a serious game platform since they introduce intelligence in processing and analysis of users' interactivity. ML provides intelligent scoring and monitoring capabilities of the user's progress in a serious game platform. In this article, we introduce a deep learning model for motion primitive classification. The model combines a convolutional processing layer with a bidirectional analysis module. This way, RGB information is efficiently handled by the hierarchies of convolutions, while the bidirectional properties of a long short term memory (LSTM) model are retained. The resulting convolutionally enhanced bidirectional LSTM (CEBi-LSTM) architecture is less sensitive to skeleton errors, occurring using low-cost sensors, such as Kinect, while simultaneously handling the high amount of detail when using RGB visual information.