I propose a holography technique that detects both three-dimensional (3D) and polarization information on incoherent light without any polarization filters. Two polarization-sensitive phase-only spatial light modulators are used to generate self-interference holograms with different polarization directions. 3D images for different polarization directions are retrieved from the recorded polarization-multiplexed holograms, exploiting the developed phase-shifting interferometry and numerical refocusing. Moreover, light-use efficiency is improved in general incoherent holography because there is no polarization filter. Its validity is experimentally demonstrated.