The size-dependence of surface plasmon resonances (SPRs) is poorly understood in the small particle limit due to complex physical/chemical effects and uncertainties in experimental samples. In this article, we report an approach for synthesizing an ideal class of colloidal Ag nanoparticles with highly uniform morphologies and narrow size distributions. Optical measurements and theoretical analyses for particle diameters in the d approximately 2-20 nm range are presented. The SPR absorption band exhibits an exceptional behavior: As size decreases from d approximately 20 nm it blue-shifts but then turns over near d approximately 12 nm and strongly red-shifts. A multilayer Mie theory model agrees well with the observations, indicating that lowered electron conductivity in the outermost atomic layer, due to chemical interactions, is the cause of the red-shift. We corroborate this picture by experimentally demonstrating precise chemical control of the SPR peak positions via ligand exchange.