An open quantum system operated at the spectral singularities where dimensionality reduces, known as exceptional points (EPs), demonstrates distinguishing behavior from the Hermitian counterpart. Here, we present an analytical description of local density of states (LDOS) for microcavity featuring chiral EPs, and unveil the anomalous spontaneous emission dynamics from a quantum emitter (QE) due to the non-Lorentzian response of EPs. Specifically, we reveal that a squared Lorentzian term of LDOS contributed by chiral EPs can destructively interfere with the linear Lorentzian profile, resulting in the null Purcell enhancement to a QE with special transition frequency, which we call EP induced transparency. While for the case of constructive interference, the squared Lorentzian term can narrow the linewidth of Rabi splitting even below that of bare components, and thus significantly suppresses the decay of Rabi oscillation. Interestingly, we further find that an open microcavity with chiral EPs supports atom-photon bound states for population trapping and decay suppression in long-time dynamics. As applications, we demonstrate the advantages of microcavity operated at chiral EPs in achieving high-fidelity entanglement generation and high-efficiency single-photon generation. Our work unveils the exotic cavity quantum electrodynamics unique to chiral EPs, which opens the door for controlling light-matter interaction at the quantum level through non-Hermiticity, and holds great potential in building high-performance quantum-optics devices.