We present a theory describing the single-ion anisotropy of rare-earth (RE) magnets in the presence of point defects. Taking the RE-lean 1∶12 magnet class as a prototype, we use first-principles calculations to show how the introduction of Ti substitutions into SmFe_{12} perturbs the crystal field, generating new coefficients due to the lower symmetry of the RE environment. We then demonstrate that these perturbations can be described extremely efficiently using a screened point charge model. We provide analytical expressions for the anisotropy energy that can be straightforwardly implemented in atomistic spin dynamics simulations, meaning that such simulations can be carried out for an arbitrary arrangement of point defects. The significant crystal field perturbations calculated here demonstrate that a sample that is single phase from a structural point of view can nonetheless have a dramatically varying anisotropy profile at the atomistic level if there is compositional disorder, which may influence localized magnetic objects like domain walls or skyrmions.