CT-Loc: Cross-domain visual localization with a channel-wise transformer

Abstract

We tackle the cross-domain visual localization problem of estimating camera position and orientation from real images without three-dimensional (3D) spatial mapping or modeling. Recent studies have shown suboptimal performance in this task owing to the photometric and geometric differences between synthetic and real images. In this study, we present a deep learning approach that uses a channel-wise transformer localization (CT-Loc) framework. Inspired by the human behavior of looking for structural landmarks to estimate one's location, CT-Loc encodes the most salient features of task-relevant objects in target scenes. To evaluate the efficacy of the proposed method in a real-world application, we built a complex and large-scale dataset of the interior of the mechanical room during operations and conducted extensive performance comparisons with the publicly available state-of-the-art University of Melbourne Corridor and Virtual KITTI 2 datasets. Compared with the otherwise best-performing BIM-PoseNet indoor camera localization model, our method significantly reduces position and orientation errors through the application of attention weights and saliency maps while also learning only the visual structural patterns (e.g., floors and doors) that are most relevant to localization tasks. Our model successfully ignores uninformative objects. This approach yields higher-level robust camera-pose regression localization results without requiring prebuilt maps. The code is available at https://github.com/kdaeho27/CT-Loc.

Keywords: 3D model; Cross-domain; Deep learning; Transformer; Visual localization.