The microstructures on a diamond surface have attracted extensive attention in microelectronics, ultra-precision machining tools, and optical elements, etc. In this work, microgrooves were fabricated on a single-crystal diamond surface using ultraviolet nanosecond or infrared picosecond laser pulses. The surface and internal morphologies of the microgrooves were characterized. The chemical composition and phase transition of the diamond after laser irradiation were analyzed. Furthermore, the ablation threshold, ablation rate, and material removal rate of the diamond processed by nanosecond or picosecond lasers were also calculated. In addition, the temperature distributions of the diamond ablated by nanosecond or picosecond lasers were simulated. Finally, the material removal mechanisms of a single-crystal diamond processed by nanosecond or picosecond lasers were revealed. This work is expected helpful to provide a guidance for the laser fabrication of microstructures on diamond.