Density functional theory can be generalized to mixtures of ground and excited states, for the purpose of determining energies of excitations using low-cost density functional approximations. Adapting approximations originally developed for ground states to work in the new setting would fast-forward progress enormously. But, previous attempts have stumbled on daunting fundamental issues. Here we show that these issues can be prevented from the outset, by using a fluctuation dissipation theorem (FDT) to dictate key functionals. We thereby show that existing exchange energy approximations are readily adapted to excited states, when combined with a rigorous exact Hartree term that is different in form from its ground state counterpart, and counterparts based on ensemble Ansatzë. Applying the FDT to correlation energies also provides insights into ground statelike and ensemble-only correlations. We thus provide a comprehensive and versatile framework for ensemble density functional approximations.