We report on a noncontact photoacoustic imaging system utilizing an all-fiber-optic heterodyne interferometer as an acoustic wave detector. The acoustic wave generated by a short laser pulse via the photoacoustic effect and arriving at the sample surface could be detected with the fiber-optic heterodyne interferometer without physical contact or using an impedance matching medium. A phantom experiment was conducted to evaluate the proposed system, and the initial acoustic pressure distribution was calculated using a Fourier-based reconstruction algorithm. It is expected that the all-fiber-optic configuration of the proposed system can be applied as a minimally invasive diagnostic tool.