In this paper, we report the existence of intervalence charge transfer (IVCT) luminescence in Yb-doped fluorite-type crystals associated with Yb(2+)-Yb(3+) mixed valence pairs. By means of embedded cluster, wave function theory ab initio calculations, we show that the widely studied, very broad band, anomalous emission of Yb(2+)-doped CaF2 and SrF2, usually associated with impurity-trapped excitons, is, rather, an IVCT luminescence associated with Yb(2+)-Yb(3+) mixed valence pairs. The IVCT luminescence is very efficiently excited by a two-photon upconversion mechanism where each photon provokes the same strong 4f(14)-1A1g→ 4f(13)((2)F7/2)5deg-1T1u absorption in the Yb(2+) part of the pair: the first one, from the pair ground state; the second one, from an excited state of the pair whose Yb(3+) moiety is in the higher 4f(13)((2)F5/2) multiplet. The Yb(2+)-Yb(3+) → Yb(3+)-Yb(2+) IVCT emission consists of an Yb(2+) 5deg → Yb(3+) 4f7/2 charge transfer accompanied by a 4f7/2 → 4f5/2 deexcitation within the Yb(2+) 4f(13) subshell: [(2)F5/25deg,(2)F7/2] → [(2)F7/2,4f(14)]. The IVCT vertical transition leaves the oxidized and reduced moieties of the pair after electron transfer very far from their equilibrium structures; this explains the unexpectedly large band width of the emission band and its low peak energy, because the large reorganization energies are subtracted from the normal emission. The IVCT energy diagrams resulting from the quantum mechanical calculations explain the different luminescent properties of Yb-doped CaF2, SrF2, BaF2, and SrCl2: the presence of IVCT luminescence in Yb-doped CaF2 and SrF2; its coexistence with regular 5d-4f emission in SrF2; its absence in BaF2 and SrCl2; the quenching of all emissions in BaF2; and the presence of additional 5d-4f emissions in SrCl2 which are absent in SrF2. They also allow to interpret and reproduce recent experiments on transient photoluminescence enhancement in Yb(2+)-doped CaF2 and SrF2, the appearance of Yb(2+) 4f-5d absorption bands in the excitation spectra of the IR Yb(3+) emission in partly reduced CaF2:Yb(3+) samples, and to identify the broadband observed in the excitation spectrum of the so far called anomalous emission of SrF2:Yb(2+) as an IVCT absorption, which corresponds to an Yb(2+) 4f5/2 → Yb(3+) 4f7/2 electron transfer.