Refined UNet v3: Efficient end-to-end patch-wise network for cloud and shadow segmentation with multi-channel spectral features

Semantic segmentation is one of the essential prerequisites for computer vision tasks, but edge-precise segmentation stays challenging due to the potential lack of a proper model indicating the low-level relation between pixels. We have presented Refined UNet v2, a concatenation of a network backbone and a subsequent embedded conditional random field (CRF) layer, which coarsely performs pixel-wise classification and refines edges of segmentation regions in a one-stage way. However, the CRF layer of v2 employs a gray-scale global observation (image) to construct contrast-sensitive bilateral features, which is not able to achieve the desired performance on ambiguous edges. In addition, the naïve depth-wise Gaussian filter cannot always compute efficiently, especially for a longer-range message-passing step. To address the aforementioned issues, we upgrade the bilateral message-passing kernel and the efficient implementation of Gaussian filtering in the CRF layer in this paper, referred to as Refined UNet v3, which is able to effectively capture ambiguous edges and accelerate the message-passing procedure. Specifically, the inherited UNet is employed to coarsely locate cloud and shadow regions and the embedded CRF layer refines the edges of the forthcoming segmentation proposals. The multi-channel guided Gaussian filter is applied to the bilateral message-passing step, which improves detecting ambiguous edges that are hard for the gray-scale counterpart to identify, and fast Fourier transform-based (FFT-based) Gaussian filtering facilitates an efficient and potentially range-agnostic implementation. Furthermore, Refined UNet v3 is able to be extended to segmentation on multi-spectral datasets, and the corresponding refinement examination confirms the development of shadow retrieval. Experiments and corresponding results demonstrate that the proposed update can outperform its counterpart in terms of the detection of vague edges, shadow retrieval, and isolated redundant regions, and it is practically efficient in our TensorFlow implementation. The demo source code is available at https://github.com/92xianshen/refined-unet-v3.