Lithography provides the precision to pattern large arrays of metallic nanostructures with varying geometries, enabling systematic studies and discoveries of new phenomena in plasmonics. However, surface plasmon resonances experience more damping in lithographically-defined structures than in chemically-synthesized nanoparticles of comparable geometries. Grain boundaries, surface roughness, substrate effects, and adhesion layers have been reported as causes of plasmon damping, but it is difficult to isolate these effects. Using monochromated electron energy-loss spectroscopy (EELS) and numerical analysis, we demonstrate an experimental technique that allows the study of these effects individually, to significantly reduce the plasmon damping in lithographically-defined structures. We introduce a method of encapsulated annealing that preserves the shape of polycrystalline gold nanostructures, while their grain-boundary density is reduced. We demonstrate enhanced Q-factors in lithographically-defined nanostructures, with intrinsic damping that matches the theoretical Drude damping limit.