Non-line-of-Sight Imaging via Neural Transient Fields

IEEE Trans Pattern Anal Mach Intell. 2021 Jul;43(7):2257-2268. doi: 10.1109/TPAMI.2021.3076062. Epub 2021 Jun 8.

Abstract

We present a neural modeling framework for non-line-of-sight (NLOS) imaging. Previous solutions have sought to explicitly recover the 3D geometry (e.g., as point clouds) or voxel density (e.g., within a pre-defined volume) of the hidden scene. In contrast, inspired by the recent Neural Radiance Field (NeRF) approach, we use a multi-layer perceptron (MLP) to represent the neural transient field or NeTF. However, NeTF measures the transient over spherical wavefronts rather than the radiance along lines. We therefore formulate a spherical volume NeTF reconstruction pipeline, applicable to both confocal and non-confocal setups. Compared with NeRF, NeTF samples a much sparser set of viewpoints (scanning spots) and the sampling is highly uneven. We thus introduce a Monte Carlo technique to improve the robustness in the reconstruction. Experiments on synthetic and real datasets demonstrate NeTF achieves state-of-the-art performance and can provide reliable reconstructions even under semi-occlusions and on non-Lambertian materials.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Algorithms*
  • Imaging, Three-Dimensional
  • Monte Carlo Method
  • Neural Networks, Computer*