We carry out nonperturbative calculations of the single-particle excitation spectrum in strongly interacting neutron matter. These are microscopic quantum Monte Carlo computations of many-neutron energies at different densities as well as several distinct excited states. As input, we employ both phenomenological and chiral two- and three-nucleon interactions. We use the single-particle spectrum to extract the effective mass in neutron matter. With a view to systematizing the error involved in this extraction, we carefully assess the impact of finite-size effects on the quasiparticle dispersion relation. We find an effective-mass ratio that drops from 1 as the density is increased. We conclude by connecting our results with the physics of ultracold gases as well as with energy-density functional theories of nuclei and neutron-star matter.