The human visual system is extremely sensitive to the directions of the gazes of others. However, the mechanism underlying gaze direction perception has yet to be clarified. The primary aim of the present study is to investigate whether the relational property between the local eye region and other facial regions serves as the primary visual system cue in detecting a direct gaze. Our results showed that search efficiency was determined primarily by the gaze direction indicated by the relational property regardless of the direction indicated by the local feature information of the eye region; this was true even when the gaze directions indicated by these two types of information were conflicting. These results bolster the hypothesis that the human visual system primarily accesses socially meaningful information in searching for a deviant gaze.