Non-Markovian processes may arise in physics due to memory effects of environmental degrees of freedom. For quantum non-Markovianity, it is an ongoing debate to clarify whether such memory effects have a verifiable quantum origin, or whether they might equally be modeled by a classical memory. In this contribution, we propose a criterion to test locally for a truly quantum memory. The approach is agnostic with respect to the environment, as it solely depends on the local dynamics of the system of interest. Experimental realizations are particularly easy, as only single-time measurements on the system itself have to be performed. We study memory in a variety of physically motivated examples, both for a time-discrete case, and for time-continuous dynamics. For the latter, we are able to provide an interesting class of non-Markovian master equations with classical memory that allows for a physically measurable quantum trajectory representation.