The energies of valley-orbit states in silicon quantum dots are determined by an as yet poorly understood interplay between interface roughness, orbital confinement, and electron interactions. Here, we report measurements of one- and two-electron valley-orbit state energies as the dot potential is modified by changing gate voltages, and we calculate these same energies using full configuration interaction calculations. The results enable an understanding of the interplay between the physical contributions and enable a new probe of the quantum well interface.