Context awareness based Sketch-DeepNet architecture for hand-drawn sketches classification and recognition in AIoT

Safdar Ali; Nouraiz Aslam; DoHyeun Kim; Asad Abbas; Sania Tufail; Beenish Azhar

doi:10.7717/peerj-cs.1186

Context awareness based Sketch-DeepNet architecture for hand-drawn sketches classification and recognition in AIoT

PeerJ Comput Sci. 2023 Apr 27:9:e1186. doi: 10.7717/peerj-cs.1186. eCollection 2023.

Authors

Safdar Ali¹, Nouraiz Aslam¹, DoHyeun Kim², Asad Abbas³, Sania Tufail¹, Beenish Azhar¹

Affiliations

¹ Department of Software Engineering, University of Lahore, Lahore, Punjab, Pakistan.
² Department of Computer Engineering, Jeju National University, Jeju, Jeju, South Korea.
³ Department of Computer Science, University of Central Punjab, Lahore, Punjab, Pakistan.

Abstract

A sketch is a black-and-white, 2-D graphical representation of an object and contains fewer visual details as compared to a colored image. Despite fewer details, humans can recognize a sketch and its context very efficiently and consistently across languages, cultures, and age groups, but it is a difficult task for computers to recognize such low-detail sketches and get context out of them. With the tremendous increase in popularity of IoT devices such as smartphones and smart cameras, etc., it has become more critical to recognize free hand-drawn sketches in computer vision and human-computer interaction in order to build a successful artificial intelligence of things (AIoT) system that can first recognize the sketches and then understand the context of multiple drawings. Earlier models which addressed this problem are scale-invariant feature transform (SIFT) and bag-of-words (BoW). Both SIFT and BoW used hand-crafted features and scale-invariant algorithms to address this issue. But these models are complex and time-consuming due to the manual process of features setup. The deep neural networks (DNNs) performed well with object recognition on many large-scale datasets such as ImageNet and CIFAR-10. However, the DDN approach cannot be carried out for hand-drawn sketches problems. The reason is that the data source is images, and all sketches in the images are, for example, 'birds' instead of their specific category (e.g., 'sparrow'). Some deep learning approaches for sketch recognition problems exist in the literature, but the results are not promising because there is still room for improvement. This article proposed a convolutional neural network (CNN) architecture called Sketch-DeepNet for the sketch recognition task. The proposed Sketch-DeepNet architecture used the TU-Berlin dataset for classification. The experimental results show that the proposed method beats the performance of the state-of-the-art sketch classification methods. The proposed model achieved 95.05% accuracy as compared to existing models DeformNet (62.6%), Sketch-DNN (72.2%), Sketch-a-Net (77.95%), SketchNet (80.42%), Thinning-DNN (74.3%), CNN-PCA-SVM (72.5%), Hybrid-CNN (84.42%), and human recognition accuracy of 73% on the TU-Berlin dataset.

Keywords: Convolutional neural networks (CNNs); Deep neural networks (DNNs); Sketch recognition; TU-Berlin.

Grants and funding

This work was supported by the Institute for Information & Communications Technology Promotion (IITP) (NO. 2022-0-00980, Cooperative Intelligence Framework of Scene Perception for Autonomous IoT Device) and the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (2021-0-00188, Open source development and standardization for AI enabled IoT platforms and interworking). Any correspondence related to this article should be addressed to Dohyeun Kim. The funders had a role in the study design, the decision to publish, and the preparation of the manuscript. The funders had no role in data collection and analysis.