This letter describes ptychography in the context of polarized light probing anisotropic specimen (i.e., showing properties of birefringence and/or diattenuation). We established an optimization strategy using a vectorial formalism. We propose a measurement scheme using a set of linearly polarized probes and linear polarization analyzers, which allows the retrieval of the full anisotropy map of the specimen.