Visual perception is mediated by unique contributions of the numerous brain regions that constitute the visual system. We performed simultaneous recordings of local field potentials (LFPs) and single unit activity (SUA) in areas V4 and lateral prefrontal cortex to characterize their contribution to visual processing. Here, we trained monkeys to identify natural images at different degradation levels in a visual recognition task. We parametrically varied color and structural information of natural images while the animals were performing the task. We show that the visual-evoked potential (VEP) of the LFP in V4 is highly sensitive to color, whereas the VEP in prefrontal cortex predominantly depends on image structure. When examining the relationship between VEP and SUA, we found that stimulus sensitivity for SUA was well predicted by the VEP in PF cortex but not in V4. Our results first reveal a functional specialization in both areas at the level of the LFP and further suggest that the degree to which mesoscopic signals, such as the VEP, are representative of the underlying SUA neural processing may be brain region specific within the context of visual recognition.