Photometric stereo (PS) estimates the surface normals of an object by utilizing multiple images captured under different light conditions. To obtain accurate surface normals, a large number of input images is often required. Therefore, a huge effort is needed to capture images and calibrate light directions together with a heavy computational cost. Therefore, in this paper, we propose a robust photometric stereo method even when the number of input images is very small. To this end, we design a feature translation module (FTM) that enriches features having scarce information. In particular, we insert FTMs between the layers of the baseline backbone PS network. Then, activations of each FTM are supervised by distillation loss. For computing distillation loss, we utilize a teacher PS network trained by taking lots of images as inputs. As a result, our PS network requires very few input images but produces a similar quality of output surface normals with the teacher PS network. The proposed method is applicable to both calibrated and uncalibrated PS. We show the effectiveness of the proposed method not only when the number of input images is small but also in various input conditions.