Stimuli-responsive polymers are an important class of materials with many applications in nanotechnology and drug delivery. The most prominent one is poly-N-isopropylacrylamide (PNIPAm). The characterization of the kinetics of its change after a temperature jump is still a lively research topic, especially at nanometer-length scales where it is not possible to rely on conventional microscopic techniques. Here, we measured in real time the collapse of a PNIPAm shell on silica nanoparticles with megahertz x-ray photon correlation spectroscopy at the European XFEL. We characterize the changes of the particles diffusion constant as a function of time and consequently local temperature on sub-microsecond timescales. We developed a phenomenological model to describe the observed data and extract the characteristic times associated to the swelling and collapse processes. Different from previous studies tracking the turbidity of PNIPAm dispersions and using laser heating, we find collapse times below microsecond timescales and two to three orders of magnitude slower swelling times.