Extensive efforts have been expended in developing hybrid quantum systems to overcome the short coherence time of superconducting circuits by introducing the naturally long-lived spin degree of freedom. Among all the possible materials, single-crystal yttrium iron garnet has shown up recently as a promising candidate for hybrid systems, and various highly coherent interactions, including strong and even ultrastrong coupling, have been demonstrated. One distinct advantage in these systems is that spins form well-defined magnon modes, which allows flexible and precise tuning. Here we demonstrate that by dissipation engineering, a non-Markovian interaction dynamics between the magnon and the microwave cavity photon can be achieved. Such a process enables us to build a magnon gradient memory to store information in the magnon dark modes, which decouple from the microwave cavity and thus preserve a long lifetime. Our findings provide a promising approach for developing long-lifetime, multimode quantum memories.