Most known trans-Neptunian objects (TNOs) gravitationally scattering off the giant planets have orbital inclinations consistent with an origin from the classical Kuiper belt, but a small fraction of these "scattering TNOs" have inclinations that are far too large (i > 45°) for this origin. These scattering outliers have previously been proposed to be interlopers from the Oort cloud or evidence of an undiscovered planet. Here we test these hypotheses using N-body simulations and the 69 centaurs and scattering TNOs detected in the Outer Solar Systems Origins Survey and its predecessors. We confirm that observed scattering objects cannot solely originate from the classical Kuiper belt, and we show that both the Oort cloud and a distant planet generate observable highly inclined scatterers. Although the number of highly inclined scatterers from the Oort Cloud is ~3 times less than observed, Oort cloud enrichment from the Sun's galactic migration or birth cluster could resolve this. Meanwhile, a distant, low-eccentricity 5 M⊕ planet replicates the observed fraction of highly inclined scatterers, but the overall inclination distribution is more excited than observed. Furthermore, the distant planet generates a longitudinal asymmetry among detached TNOs that is less extreme than often presumed, and its direction reverses across the perihelion range spanned by known TNOs. More complete models that explore the dynamical origins of the planet are necessary to further study these features. With observational biases well-characterized, our work shows that the orbital distribution of detected scattering bodies is a powerful constraint on the unobserved distant solar system.