We report on an all-optical ultrasonic detecting method based on differential interference. A linearly polarized probe beam is split into two closely separated ones with orthogonal polarization. After interacting with propagating ultrasonic waves in a coupling media, the split beams are recombined into one beam, with its polarization being changed into an elliptical one by the elastic-optical effect. The recombined beam is filtered by an analyzer and detected by a photodetector. The bandwidth and noise-equivalent pressure (NEP) of the acoustic detector are determined to be 107.4 MHz and 2.18 kPa, respectively. We also demonstrate its feasibility for photoacoustic microscopy (PAM) using agar-embedded phantoms.