Combined ultrasound and photoacoustic imaging has attracted significant interests for intravascular imaging such as atheromatous plaque detection, with ultrasound imaging providing spatial location and morphology and photoacoustic imaging highlighting molecular composition of the plaque. Conventional ultrasound imaging systems utilize piezoelectric ultrasound transducers, which suffer from limited frequency bandwidths and reduced sensitivity with miniature transducer elements. Recent advances on optical methods for both ultrasound generation and detection have shown great promise, as they provide efficient and ultrabroadband ultrasound generation and sensitive and ultrabroadband ultrasound detection. As such, all-optical ultrasound imaging has a great potential to become a next generation ultrasound imaging method. In this paper, we review recent developments on optical ultrasound transmitters, detectors, and all-optical ultrasound imaging systems, with a particular focus on fiber-based probes for intravascular imaging. We further discuss our thoughts on future directions on developing combined all-optical photoacoustic and ultrasound imaging systems for intravascular imaging.