Increasing degradation of coral reef ecosystems and specifically, loss of corals is causing significant and widespread declines in the abundance of coral reef fishes, but the proximate cause(s) of these declines are largely unknown. Here, we examine specific responses to host coral mortality for three species of coral-dwelling damselfishes (Dascyllus aruanus, D. reticulatus, and Pomacentrus moluccensis), explicitly testing whether these fishes can successfully move and recolonize nearby coral hosts. Responses of fishes to localized coral loss was studied during population irruptions of coral feeding crown-of-thorns starfish, where starfish consumed 29 (34%) out of 85 coral colonies, of which 25 (86%) were occupied by coral-dwelling damselfishes. Damselfishes were not tagged or individually recognizable, but changes in the colonization of different coral hosts was assessed by carefully assessing the number and size of fishes on every available coral colony. Most damselfishes (> 90%) vacated dead coral hosts within 5 days, and either disappeared entirely (presumed dead) or relocated to nearby coral hosts. Displaced fishes only ever colonized corals already occupied by other coral-dwelling damselfishes (mostly conspecifics) and colonization success was strongly size-dependent. Despite movement of damselfishes to surviving corals, the local abundance of coral-dependent damselfishes declined in approximate accordance with the proportional loss of coral habitat. These results suggest that even if alternative coral hosts are locally abundant, there are significant biological constraints on movement of coral-dwelling damselfishes and recolonization of alternative coral habitats, such that localized persistence of habitat patches during moderate or patchy disturbances do not necessarily provide resilience against overall habitat loss.