Three-dimensional (3D) metrology has received a lot of attention from academic and industrial communities due to its broad applications, such as 3D contents, 3D printing, and autonomous driving. The all-optical depth coloring (AODC) camera has some benefits in computation load since it extracts depth information of an object fully optically. The AODC camera represents the depth of the object as a variation of wavelength, and spectroscopy is generally required to measure the wavelength. However, in the AODC camera, the color vector in RGB color space is convertible inversely into the wavelength after projection on the normalized rgb plane because the detected spectrum through the gating part has a narrow bandwidth as a result of the width of the slit in the projection part. In this paper, we propose an inverse conversion algorithm from RGB color to depth without spectroscopy. Experimental results are presented to confirm its feasibility. Also, some practical limitations are discussed, resulting from the nonlinearity of the response of the image sensor and the widths of the slits in the projection part and the gating part.