A combined solvent system composed of an acidic and a neutral extractant is demonstrated as an effective system for the mutual separation of lanthanides and actinides from nitric acid solutions. The geometry and stability of various possible complexes formed under extraction and stripping conditions in a combined solvent system composed of N,N-dioctyl hydroxyacetamide (DOHyA) and bis(2-ethylhexyl)phosphoric acid (HDEHP) in the n-dodecane medium were studied both experimentally and theoretically. Experimental observations of the distribution ratios of Am(III) and Eu(III) in the combined solvent system revealed synergistic extraction of trivalent metal ions at all nitric acid concentrations. Density functional theory (DFT) calculations were employed to understand the geometric and electronic properties of the ligands and their corresponding complexes with Am(III) and Eu(III). The calculated results indicate that the feasibility and behaviour of complex formation in the combined solvent system using different methods are based on the energetic aspects of the formation reactions. The study also reveals the participation of the neutral and acid extractants in the combined solvent system facilitating the separation of Eu(III) and Am(III) from high-level liquid waste.