In order to tap the electrophysiological correlates of the perceptual and structural encoding stages of face processing, we investigated how inversion and familiarity affect the face-specific event-related potentials (ERPs) components. ERPs were recorded while participants performed a familiarity judgment task with upright and inverted photographs of famous and unknown faces. The early P100 component was found to be sensitive to facial configuration that is disrupted by face inversion. Noteworthy, in addition to the ongoing effect of orientation, an effect of familiarity, although limited to upright faces, emerged at the processing stage indexed by N170. Later on, as witnessed by the P250 component, the familiarity effect was generalized to both upright and inverted faces with a larger amplitude for inverted famous faces. All in all, the present results suggest that the face structural encoding stage is cognitively permeable by higher-order factors such as familiarity, especially when familiarity is crucial for mastering the task. From a more general viewpoint, these results indicate that face processing is subserved by multiple mechanisms in which structural (i.e. orientation) and semantic (i.e. familiarity) factors begin to interact at early processing stages with different time courses. The electrophysiological correlates of these mechanisms are documented by the differential involvement of the major ERP components in the "familiarity check". With upright faces familiarity affects the N170 component, while with inverted faces it affects later components, in keeping with a prolonged time course of the familiarity decision when orientation is not upright.