High-quality optical resonant cavities require low optical loss, typically on the scale of parts per million. However, unintended micron-scale contaminants on the resonator mirrors that absorb the light circulating in the cavity can deform the surface thermoelastically and thus increase losses by scattering light out of the resonant mode. The point absorber effect is a limiting factor in some high-power cavity experiments, for example, the Advanced LIGO gravitational-wave detector. In this Letter, we present a general approach to the point absorber effect from first principles and simulate its contribution to the increased scattering. The achievable circulating power in current and future gravitational-wave detectors is calculated statistically given different point absorber configurations. Our formulation is further confirmed experimentally in comparison with the scattered power in the arm cavity of Advanced LIGO measured by in situ photodiodes. The understanding presented here provides an important tool in the global effort to design future gravitational-wave detectors that support high optical power and thus reduce quantum noise.