Faces are a major source of information in social interaction. The ability to perceive and interpret faces thus carries paramount importance in people's social lives. However, this crucial ability is not yet fully understood. Individual differences studies show that the speed and accuracy of face cognition (including perception and memory/recognition), the two facets targeted when measuring cognitive performance, are relatively independent traits. Unlike accuracy data, individual differences in reaction times (RTs) measured in perceptual decision tasks with or without memory load using faces and objects, do not show face-specific variance. Here, we applied the diffusion model to RT and accuracy data captured by simple perceptual decision tasks to improve understanding of the lack of face specificity. If performance speed in face cognition tasks is truly a global, nonspecific individual ability, no parameter of the diffusion model should hold face specificity. In a study on adults (N = 217), we administered two tasks of face and object perception. We used individually estimated diffusion model parameters as manifest variables to study face specificity in drift rate (ν), boundary separation (a), and nondecision time (Ter) using structural equation modeling. Furthermore, to study differential relationships between diffusion model parameters and measures of cognitive abilities, we regressed factors of face and object cognition accuracy on factors of diffusion model parameters. The results revealed face specificity only in boundary separation. This suggests face-specific adjustment in the cautiousness of information processing. (PsycINFO Database Record (c) 2018 APA, all rights reserved).