In human lateral temporal cortex, some regions show specific sensitivity to human motion. Here we examine whether such effects reflect a general biological-nonbiological organizational principle or a process specific to human-agent processing by comparing processing of human, animal, and tool motion in a functional magnetic resonance imaging (fMRI) experiment with healthy participants and a voxel-based lesion-symptom mapping (VLSM) study of patients with brain damage (77 stroke patients). The fMRI experiment revealed that in the lateral temporal cortex, the posterior superior temporal sulcus shows a preference for human and animal motion, whereas the middle part of the right superior temporal sulcus/gyrus (mSTS/STG) shows a preference for human and functional tool motion. VLSM analyses also revealed that damage to this right mSTS/STG region led to more severe impairment in the recognition of human and functional tool motion relative to animal motion, indicating the causal role of this brain area in human-agent motion processing. The findings for the right mSTS/STG cannot be reduced to a preference for articulated motion or processing of social variables since neither factor is involved in functional tool motion recognition. We conclude that a unidimensional biological-nonbiological distinction cannot fully explain the visual motion effects in lateral temporal cortex. Instead, the results suggest the existence of distinct components in right posterior temporal cortex and mSTS/STG that are associated, respectively, with biological motion and human-agent motion processing.