We use a confocal microscope to study the aging of a bidisperse colloidal glass near rough and smooth boundaries. Near smooth boundaries, the particles form layers, and particle motion is dramatically slower near the boundary as compared to the bulk. Near rough boundaries, the layers nearly vanish, and particle motion is nearly identical to that of the bulk. The gradient in dynamics near the boundaries is demonstrated to be a function of the gradient in structure for both types of boundaries. Our observations show that wall-induced layer structures strongly influence aging.