We have measured the low-frequency time instability known as charge offset drift of Si/SiO2 single electron devices (SEDs) with and without an overall poly-Si top gate. We find that SEDs with a poly-Si top gate have significantly less charge offset drift, exhibiting fewer isolated jumps and a factor of two reduction in fluctuations about a stable mean value. The observed reduction can be accounted for by the electrostatic reduction in the mutual capacitance Cm between defects and the quantum dot, and increase in the total defect capacitance Cd due to the top gate. These results depart from the prominent interpretation that the level of charge offset drift in SEDs is determined by the intrinsic material properties, forcing consideration of the device design as well. We expect these results to be of importance in developing SEDs for applications from quantum information to metrology or wherever charge noise or integrability of devices is a challenge.