Pb-free double perovskites, such as Cs2AgBiBr6, are alternatives to lead halide perovskites for photovoltaic applications due to superior stability, low toxicity, and promising optoelectronic properties. However, their performance is subpar. We combine nonadiabatic molecular dynamics and real-time time-dependent density-functional theory to show that the negatively charged Br vacancy in Cs2AgBiBr6 creates an extremely detrimental donor-yielded (DY) center, which is a typical defect in six-coordinated semiconductors. Ag+ and Bi3+ form a bond by attraction through the anisotropic vacancy charge, generating a midgap state that traps holes within tens of picoseconds. Substituting Ag with indium by doping produces a weak and long In-Bi bond, lifting the defect energy level to the conduction band. Hole trapping slows down by an order or magnitude, and trap-assisted charge recombination decreases 4-fold. The simulations bring atomistic insights into defects of Pb-free double perovskites and provide a defect mitigation strategy for rational design of high-performance optoelectronic devices.