We propose and experimentally demonstrate a method of polarization-sensitive quantitative phase imaging using two photodetectors and a digital micromirror device. Instead of recording wide-field interference patterns, finding the modulation patterns maximizing focused intensities in terms of the polarization states enables polarization-dependent quantitative phase imaging without the need for a reference beam and an image sensor. The feasibility of the present method is experimentally validated by reconstructing Jones matrices of several samples including a polystyrene microsphere, a maize starch granule, and a mouse retinal nerve fiber layer. Since the present method is simple and sufficiently general, we expect that it may offer solutions for quantitative phase imaging of birefringent materials.