Challenging deep learning models with image distortion based on the abutting grating illusion

Jinyu Fan; Yi Zeng

doi:10.1016/j.patter.2023.100695

Challenging deep learning models with image distortion based on the abutting grating illusion

Patterns (N Y). 2023 Feb 28;4(3):100695. doi: 10.1016/j.patter.2023.100695. eCollection 2023 Mar 10.

Authors

Jinyu Fan¹, Yi Zeng^{1

2

3

4

5}

Affiliations

¹ Brain-inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China.
² National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China.
³ School of Future Technology, University of Chinese Academy of Sciences, Beijing 100049, China.
⁴ School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China.
⁵ Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai 200031, China.

Abstract

Even state-of-the-art deep learning models lack fundamental abilities compared with humans. While many image distortions have been proposed to compare deep learning with humans, they depend on mathematical transformations instead of human cognitive functions. Here, we propose an image distortion based on the abutting grating illusion, which is a phenomenon discovered in humans and animals. The distortion generates illusory contour perception using line gratings abutting each other. We applied the method to MNIST, high-resolution MNIST, and "16-class-ImageNet" silhouettes. Many models, including models trained from scratch and 109 models pretrained with ImageNet or various data augmentation techniques, were tested. Our results show that abutting grating distortion is challenging even for state-of-the-art deep learning models. We discovered that DeepAugment models outperformed other pretrained models. Visualization of early layers indicates that better-performing models exhibit the endstopping property, which is consistent with neuroscience discoveries. Twenty-four human subjects classified distorted samples to validate the distortion.

Keywords: DeepAugment; abutting grating illusion; deep learning; endstopping; illusory contour; image distortion; robustness.