Safe and effective transcranial magnetic stimulation (TMS) requires accurate intensity calibration. Output is typically calibrated to individual motor cortex excitability and applied to nonmotor brain areas, assuming that it captures a site nonspecific factor of excitability. We tested this assumption by correlating the effect of TMS at motor and visual cortex. In 30 participants, we measured motor threshold (MT) and phosphene threshold (PT) at the scalp surface and at coil-scalp distances of 3.17, 5.63, and 9.03 mm. We also modeled the effect of TMS in a simple head model to test the effect of distance. Four independent tests confirmed a significant correlation between PT and MT. We also found similar effects of distance in motor and visual areas, which did not correlate across participants. Computational modeling suggests that the relationship between the effect of distance and the induced electric field is effectively linear within the range of distances that have been explored empirically. We conclude that MT-guided calibration is valid for nonmotor brain areas if coil-cortex distance is taken into account. For standard figure-of-eight TMS coils connected to biphasic stimulators, the effect of cortical distance should be adjusted using a general correction factor of 2.7% stimulator output per millimeter.