It is generally accepted that the N170 component of an event-related potential (ERP) reflects the structural encoding of faces and is specialized for face processing. Recent neuroimaging and ERP studies have demonstrated that spatial frequency is a crucial factor for face recognition. To clarify which early ERP components reflect either coarse (low spatial frequency, LSF) or fine (high spatial frequency, HSF) processing of faces, we recorded ERPs induced by manipulated face stimuli. By filtering the original grayscale faces (broadband spatial frequency) spatially, we created LSF and HSF face stimuli. Next, we created physically equiluminant (PEL) face stimuli to eliminate the effects of lower order information, such as luminance and contrast. The P1 amplitude at the occipital region was augmented by LSF faces, while the N170 amplitude increased for HSF faces. The occipital P1 amplitude for PEL faces was relatively unaffected compared with that for PEL houses. In addition, the occipital N2 for PEL faces was spatiotemporally separable from N170 in a time-window between P1 and N170. These results indicate that P1 reflects coarse processing of faces, and that the face robustness further assures face-specific processing in the early component. Moreover, N2 reflects the early contrast processing of faces whereas N170 analyzes the fine facial features. Our findings suggest the presence of spatial frequency-and-contrast detectors for face processing.