Nonreciprocity has always been a subject of interest and plays a key role in a variety of applications like signal processing and noise isolation. In this work, we propose a simple and feasible scheme to implement nonreciprocal microwave transmission in a high-quality-factor superconducting cavity with ferrimagnetic materials. We derive necessary requirements to create nonreciprocity in our system where a magnon mode and two microwave modes are coupled to each other, highlighting the adjustability of a static magnetic field controlled nonreciprocal transmission based on quantum interference between different transmission paths, which breaks time-reversal symmetry of the three-mode cavity magnonics system. The high light isolation adjusted within a range of different magnetic fields can be obtained by modulating the photon-magnon coupling strength. Due to the simplicity of the device and the system tunability, our results may facilitate potential applications for light magnetic sensing and coherent information processing.