Diffuse polarization-based 3D imaging has flourished with the ability to obtain the 3D shapes of objects without multiple detectors, active mode lighting, or complex mechanical structures, which are major drawbacks of other methods for 3D imaging in natural scenes. However, traditional polarization-based 3D imaging technology introduces color distortion when reconstructing the surface of multi-colored targets. We propose a polarization-based 3D imaging model to recover the 3D geometry of multi-colored Lambertian objects. In particular, chromaticity-based color removal theory is used to restore the intrinsic intensity, which is modulated only by the target shape, and we apply the recovered intrinsic intensity to address the orientation uncertainty of target normals due to azimuth ambiguity. Finally, we integrate the corrected normals to reconstruct high-precision 3D shapes. Experimental results demonstrate that the proposed model has the ability to reconstruct multi-colored Lambertian objects exhibiting non-uniform reflectance from single views under natural light conditions.