Convolved images are often used to simulate the effect of ocular aberrations on image quality, where the retinal image is simulated by convolving the stimulus with the point spread function derived from the subject's aberrations. However, some studies have shown that convolved images are perceived far more degraded than the same image blurred with optical defocus. We hypothesized that the positive interactions between the monochromatic and chromatic aberrations in the eye are lost in the convolution process. To test this hypothesis, we evaluated optical and visual quality with natural optics and with convolved images (on-bench, computer simulations, and visual acuity [VA] in subjects) using a polychromatic adaptive optics system with monochromatic (555 nm) and polychromatic light (WL) illumination. The subject's aberrations were measured using a Hartmann Shack system and were used to convolve the visual stimuli, using Fourier optics. The convolved images were seen through corrected optics. VA with convolved stimuli was lower than VA through natural aberrations, particularly in WL (by 26% in WL). Our results suggest that the systematic decrease in visual performance with visual acuity and retinal image quality by simulation with convolved stimuli appears to be primarily associated with a lack of favorable interaction between chromatic and monochromatic aberrations in the eye.