Transformer Based Binocular Disparity Prediction with Occlusion Predict and Novel Full Connection Layers

Yi Liu; Xintao Xu; Bajian Xiang; Gang Chen; Guoliang Gong; Huaxiang Lu

doi:10.3390/s22197577

Transformer Based Binocular Disparity Prediction with Occlusion Predict and Novel Full Connection Layers

Sensors (Basel). 2022 Oct 6;22(19):7577. doi: 10.3390/s22197577.

Authors

Yi Liu^{1

2}, Xintao Xu^{1

3}, Bajian Xiang^{1

2}, Gang Chen^{1

2}, Guoliang Gong^{1

2}, Huaxiang Lu^{1

2

4

5

6}

Affiliations

¹ Institute of Semiconductors, Chinese Academy of Sciences, Beijing 100083, China.
² University of Chinese Academy of Sciences, Beijing 100089, China.
³ School of Microelectronics, University of Science and Technology of China, Hefei 230026, China.
⁴ Materials and Optoelectronics Research Center, University of Chinese Academy of Sciences, Beijing 200031, China.
⁵ College of Microelectronics, University of Chinese Academy of Sciences, Beijing 100049, China.
⁶ Semiconductor Neural Network Intelligent Perception and Computing Technology Beijing Key Laboratory, Beijing 100083, China.

Abstract

The depth estimation algorithm based on the convolutional neural network has many limitations and defects by constructing matching cost volume to calculate the disparity: using a limited disparity range, the authentic disparity beyond the predetermined range can not be acquired; Besides, the matching process lacks constraints on occlusion and matching uniqueness; Also, as a local feature extractor, a convolutional neural network lacks the ability of global context information perception. Aiming at the problems in the matching method of constructing matching cost volume, we propose a disparity prediction algorithm based on Transformer, which specifically comprises the Swin-SPP module for feature extraction based on Swin Transformer, Transformer disparity matching network based on self-attention and cross-attention mechanism, and occlusion prediction sub-network. In addition, we propose a double skip connection fully connected layer to solve the problems of gradient vanishing and explosion during the training process for the Transformer model, thus further enhancing inference accuracy. The proposed model in this paper achieved an EPE (Absolute error) of 0.57 and 0.61, and a 3PE (Percentage error greater than 3 px) of 1.74% and 1.56% on KITTI 2012 and KITTI 2015 datasets, respectively, with an inference time of 0.46 s and parameters as low as only 2.6 M, showing great advantages compared with other algorithms in various evaluation metrics.

Keywords: attention; binocular disparity; transformer.

MeSH terms

Algorithms
Neural Networks, Computer*
Vision Disparity*

Grants and funding

This work was supported by the National Natural Science Foundation of China U19A2080, the National Natural Science Foundation of China U1936106, the CAS Strategic Leading Science and Technology Project XDA27040303, XDA18040400, and XDB44000000, and the High Technology Project 31513070501 and 1916312ZD00902201.