Current non-Lambertian photometric stereo methods generally require a large number of images to ensure accurate surface normal estimation. To achieve accurate surface normal recovery under a sparse set of lights, this paper proposes a non-Lambertian photometric stereo network based on a derived inverse reflectance model with collocated light. The model is deduced using monotonicity of isotropic reflectance and the univariate property of collocated light to decouple the surface normal from the reflectance function. Thus, the surface normal can be estimated by three steps, i.e., model fitting, shadow rejection, and normal estimation. We leverage a supervised deep learning technique to enhance the shadow rejection ability and the flexibility of the inverse reflectance model. Shadows are handled through max-pooling. Information from a neighborhood image patch is utilized to improve the flexibility to various reflectances. Experiments using both synthetic and real images demonstrate that the proposed method achieves state-of-the-art accuracy in surface normal estimation.