We derive ab initio optical potentials from self-consistent Green's function theory and compute the elastic scattering of neutrons off oxygen and calcium isotopes. The comparison with scattering data is satisfactory at low scattering energies. The method is benchmarked against the no-core shell model with continuum calculations, showing that virtual excitations of the target are crucial to predict proper fragmentation and absorption at higher energies. This is a significant step toward deriving optical potentials for medium mass nuclei and complex many-body systems in general.