Estimating the polarization properties of objects from polarization images is still an important but extremely undefined problem. Currently, there are two types of methods to probe the polarization properties of complex materials: one is about the equipment acquisition, which makes the collection of polarization information unsatisfactory due to the cumbersome equipment and intensive sampling, and the other is to use polarized imaging model for probing. Therefore, the accuracy of the polarized imaging model will be crucial. From an imaging perspective, we propose an end-to-end learning method that can predict accurate, physically based model parameters of polarimetric BRDF from a limited number of captured photographs of the object. In this work, we first design a novel pBRDF model as a powerful prior knowledge. This hybrid pBRDF model completely defines specular reflection, body scattering and directional diffuse reflection in imaging. Then, an end-to-end inverse rendering is performed to connect the multi-view measurements of the object with the geometry and pBRDF parameter estimation, and a reflectance gradient consistency loss is introduced to iteratively estimate the per-pixel normal, roughness, and polarimetric reflectance. Real-world measurement and rendering experiments show that the results obtained by applying our method are in strong agreement with ground truth, validating that we can reproduce the polarization properties of real-world objects using the estimated polarimetric reflectance.