We consider communication scenarios where one party sends quantum states of known dimensionality D, prepared with an untrusted apparatus, to another, distant party, who probes them with uncharacterized measurement devices. We prove that, for any ensemble of reference pure quantum states, there exists one such prepare-and-measure scenario and a linear functional W on its observed measurement probabilities, such that W can only be maximized if the preparations coincide with the reference states, modulo a unitary or an antiunitary transformation. In other words, prepare-and-measure scenarios allow one to "self-test" arbitrary ensembles of pure quantum states. Arbitrary extreme D-dimensional quantum measurements, or sets thereof, can be similarly self-tested. Our results rely on a robust generalization of Wigner's theorem, a well-known result in particle physics that characterizes physical symmetries.