In this Letter, we describe a single-pixel polarization-sensitive imaging technique, capable of generating the birefringence map of a thin specimen by using single-pixel detectors. Spatially modulated light is circularly polarized to illuminate the specimen. The transmitted light through the specimen is then focused via a lens and measured by position-sensitive detectors in two orthogonal polarization channels. The measurement of the irradiance and centroid position of the optical focus and subsequent computations enable the production of polarization-dependent wavefront maps, which can then be utilized to reconstruct sample birefringence information. We demonstrate the feasibility of our method by measuring distribution of optic-axis orientation and phase retardation of various birefringent samples.