Spin ensemble based hybrid quantum systems suffer from a significant degree of decoherence resulting from the inhomogeneous broadening of the spin transition frequencies in the ensemble. We demonstrate that this strongly restrictive drawback can be overcome simply by burning two narrow spectral holes in the spin spectral density at judiciously chosen frequencies. Using this procedure we find an increase of the coherence time by more than an order of magnitude as compared to the case without hole burning. Our findings pave the way for the practical use of these hybrid quantum systems for the processing of quantum information.