The phase differences between the transverse modes of an optical fiber can be altered all-optically by intermodal cross-phase modulation. In this Letter, we experimentally demonstrate this effect with ultrashort laser pulses. An ultrashort probe pulse, guided in both modes of a two-mode fiber, is co-propagating and temporally overlapping with an ultrashort control pulse, guided in the fundamental mode only and centered at a separate wavelength. The use of ultrashort pulses allows for a notable phase shift at a 33-fold reduced control pulse energy and a 173-fold reduced fiber length, compared to previous experiments. A total phase shift of 0.285π between the two probe modes was achieved at a 9 nJ control pulse energy in a 19 cm long two-mode graded-index fiber. Additionally, the capability of this scheme to switch ultrashort pulses in an all-optical manner was investigated. A modulation depth of 50% was achieved, limited by temporal nonlinear effects.