Materials that luminesce after excitation with ionizing radiation are extensively applied in physics, medicine, security, and industry. Lanthanide dopants are known to trigger crystal scintillation through their fast d-f emissions; the same is true for other important applications as lasers or phosphors for lighting. However, this ability can be seriously compromised by unwanted anomalous emissions often found with the most common lanthanide activators. We report high-resolution X-ray-excited optical (IR to UV) luminescence spectra of CaF2:Yb and SrF2:Yb samples excited at 8949 eV and 80 K. Ionizing radiation excites the known anomalous emission of ytterbium in the CaF2 host but not in the SrF2 host. Wave function-based ab initio calculations of host-to-dopant electron transfer and Yb2+/Yb3+ intervalence charge transfer explain the difference. The model also explains the lack of anomalous emission in Yb-doped SrF2 excited by VUV radiation.