We introduce a method to extract the photoacoustic (PA) signal from the phase time evolution of an optical coherence tomography (OCT) swept source spectral sweep. This all-optical detection is achieved in a noncontact fashion directly on the sample surface by using its specular reflection. High-speed measurement and referencing allow for close to shot noise limited phase-sensitive detection. It offers a simple way to perform OCT and PA imaging by sharing the same system components.