A flexible self-consistent field method, called target state optimization (TSO), is presented for exploring electronic excited configurations and localized diabatic states. The key idea is to partition molecular orbitals into different subspaces according to the excitation or localization pattern for a target state. Because of the orbital-subspace constraint, orbitals belonging to different subspaces do not mix. Furthermore, the determinant wave function for such excited or diabatic configurations can be variationally optimized as a ground state procedure, unlike conventional ΔSCF methods, without the possibility of collapsing back to the ground state or other lower-energy configurations. The TSO method can be applied both in Hartree-Fock theory and in Kohn-Sham density functional theory (DFT). The density projection procedure and the working equations for implementing the TSO method are described along with several illustrative applications. For valence excited states of organic compounds, it was found that the computed excitation energies from TSO-DFT and time-dependent density functional theory (TD-DFT) are of similar quality with average errors of 0.5 and 0.4 eV, respectively. For core excitation, doubly excited states and charge-transfer states, the performance of TSO-DFT is clearly superior to that from conventional TD-DFT calculations. It is shown that variationally optimized charge-localized diabatic states can be defined using TSO-DFT in energy decomposition analysis to gain both qualitative and quantitative insights on intermolecular interactions. Alternatively, the variational diabatic states may be used in molecular dynamics simulation of charge transfer processes. The TSO method can also be used to define basis states in multistate density functional theory for excited states through nonorthogonal state interaction calculations. The software implementing TSO-DFT can be accessed from the authors.