The initial timing of face-specific effects in event-related potentials (ERPs) is a point of contention in face-processing research. The occasional reports of a larger P100 to face stimuli compared to other image categories is often attributed to differences in low-level stimulus characteristics. Separating the P100 from the classic N170 effect has not been done except by adjusting stimuli to control for low-level stimulus characteristics, which yields robust face effects only after 130 ms. In the present study we use a stimulus set with minimal controls for low-level characteristics. This produces significantly larger (p < 0.01) P100 and N170 amplitudes for images of faces compared to houses in a group effect. However, with independent component analysis (ICA), we demonstrate that (a) the P100 scalp effect stems from a neural network that is indeed independent of that producing the N170 effect, despite the N170 component being active at the time of the P100; (b) compared to the N170 effect, the P100 effect is less reliable even when it is present because of intersubject variability; and (c) some individuals show a component with a larger response to houses over faces at the time of the P100 that is undetectable at the scalp because the activation of larger spatiotemporally overlapping activity cancels its field projection. Thus, with ICA, we are able to account for the general finding in the literature of a consistent N170 face effect and a less reliable P100 face effect at the level of anatomically independent electrocortical processes.