Optical modulation of high harmonic generation (HHG) is of fundamental interest in science and technology, which can facilitate understanding of HHG generation mechanisms and expand the potential optoelectronic applications. However, the current established works have neither shown the advanced modulation performance nor provided a deep understanding of modulation mechanisms. In this work, taking wurtzite zinc oxide (ZnO) single crystal as a prototype, we have demonstrated an all-optical intensity modulation of high-order HHG with a response time of less than 0.2 ps and a depth of more than 95%, based on the pump-probe configuration with two different pumping wavelengths. Besides the achieved excellent modulation performance, we have also revealed that the modulation dynamics in ZnO single crystal highly depend on the excitation conditions. Specifically, the modulation dynamics with the near-bandgap or above-bandgap excitation are attributed to the non-equilibrium interband carrier relaxations, while for mid-gap excitation, the modulation dynamics are dominated by the nonlinear frequency mixing process. This work may enhance the current understanding of the HHG modulation mechanism and enlighten novel device designs.