Quasi-continuous wave (CW) laser output with high peak power and high energy is preferred in some industrial applications. Due to the non-linear effects and transverse mode instability, such high-peak-power laser output is difficult to achieve via monolithic fiber lasers in CW mode. For diode-pumped monolithic fiber lasers, by applying overshoot pulse modulation to the pumping diodes, we obtain a pulse laser output with a peak power much higher than that in CW mode. In this paper, it has been theoretically studied that stable pulses with the width of µs level can be generated with quasi-CW operation without distortion according to our simulation. We also experimentally demonstrate a bi-directional pumped quasi-CW monolithic fiber laser operating in CW and pulse modes. In quasi-CW mode, by applying overshoot pulse modulation to the diodes, with a frequency of 1 kHz and a pulse width of 100 µs, the peak power of the output laser reached 9713 W with an average power of 898 W and M2 of 2.4 and 2.3 in the two orthogonal directions, respectively. To the best of our knowledge, this is the very first quasi-CW fiber laser of a 10-kW level with the M2 level of 2.