We have measured the intrinsic exciton dephasing in high-quality zinc blende CdSe/CdS colloidal quantum dots in the temperature range from 5 to 170 K using a sensitive three-beam photon echo technique in heterodyne detection, which is not affected by spectral diffusion. Pure dephasing via acoustic phonons dominates the initial dynamics, followed by an exponential zero-phonon line dephasing. From the temperature dependence of the zero-phonon line dephasing, the exciton lifetime, and the exciton thermalization within its fine structure, we show that the zero-phonon line dephasing of the lowest bright state originates from the phonon-assisted spin-flip to dark exciton states. Importantly, we can control the dephasing by tailoring the exciton fine structure through its dependence on the dot core size and shell thickness, as expected from the spin-flip mechanism. By reducing the electron-hole exchange interaction with increasing core size and delocalization of the electron wave function in the quasi-type-II core/shell band alignment, we find the longest zero-phonon line dephasing time of ∼110 ps at 5 K in dots with the largest core diameter (5.7 nm) and the thickest CdSe shell (9 monolayers) in the series studied.