The swelling and collapsing of thermo-responsive poly(N-isopropylacrylamide)-based polymer (pNIPAAm) networks are investigated in order to reveal the dependency on their kinetics and maximum possible actuation speed. The pNIPAAm-based network was attached as thin hydrogel film to lithographically prepared gold nanoparticle arrays to exploit their localized surface plasmon resonance (LSPR) for rapid local heating. The same substrate also served for LSPR-based monitoring of the reversible collapsing and swelling of the pNIPAAm network through its pronounced refractive index changes. The obtained data reveal signatures of multiple phases during the volume transition, which are driven by the diffusion of water molecules into and out of the network structure and by polymer chain re-arrangement. For the micrometer-thick hydrogel film in the swollen state, the layer can respond as fast as several milliseconds depending on the strength of the heating optical pulse and on the tuning of the ambient temperature with respect to the lower critical solution temperature of the polymer. Distinct differences in the time constants of swelling and collapse are observed and attributed to the dependence of the cooperative diffusion coefficient of polymer chains on polymer volume fraction. The reported results may provide guidelines for novel miniature actuator designs and micromachines that take advantages of the non-reciprocal temperature-induced volume transitions in thermo-responsive hydrogel materials.