Polymeric microgels are very interesting systems to study polymer-solvent interactions since they react to changes in the solvent properties by swelling or deswelling to reach a final equilibrium state of minimal free energy. Accordingly, factors such as pH, temperature, or salt concentration can induce size changes. While the volume phase transition undergone by ionic microgels with addition of salt is reasonably well understood in terms of Donnan effects, the origin of these transitions in neutral microgels is devoid of such understanding. In this paper, the effects of electrolytes on the swelling behavior of neutral thermosensitive microgels are interpreted within the Flory-Rehner theory for the swelling of neutral gels, by considering that ionic concentration affects the entropic contribution to the Flory solvency chi parameter; this manifests macroscopically as particle deswelling. The entropic changes depend on the ability of the electrolyte to disrupt the neighboring water structure and result in increasing values of chi with salt concentration. In this sense, we find a clear correlation between the deswelling ability of the ions and their chaotropic (structure breaking) or kosmotropic (structure making) nature as classified in the Hofmeister series.