Vanadium oxide is a key sensing material for bolometric photodetection, thanks to its strong temperature dependence of resistivity close to room temperature. Here we demonstrate the photodetection of a stoichiometric vanadium dioxide thin film integrated with silver nanorods. The nanorods convert light into heat, consequently suppressing the resistivity of vanadium dioxide via localised surface plasmon resonance. Incorporation of this thermo-plasmonic effect into bolometric photodetection allows for wavelength and polarisation sensitivity. This work opens the path to a broad family of photodetection functionalities for vanadium dioxide-based microbolometers.