Materials containing a high proportion of grain boundaries offer significant potential for the development of radiation-resistant structural materials. However, a proper understanding of the connection between the radiation-induced microstructural behavior of a grain boundary and its impact at long natural time scales is still missing. In this Letter, point defect absorption at interfaces is summarized by a jump Robin-type condition at a coarse-grained level, wherein the role of interface microstructure is effectively taken into account. Then a concise formula linking the sink strength of a polycrystalline aggregate with its grain size is introduced and is well compared with experimental observation. Based on the derived model, a coarse-grained formulation incorporating the coupled evolution of grain boundaries and point defects is proposed, so as to underpin the study of long-time morphological evolution of grains induced by irradiation. Our simulation results suggest that the presence of point defect sources within a grain further accelerates its shrinking process, and radiation tends to trigger the extension of twin boundary sections.