Using a Wigner Lorentzian random matrix ensemble, we study the fidelity, F(t), of systems at the Anderson metal-insulator transition, subject to small perturbations that preserve the criticality. We find that there are three decay regimes as perturbation strength increases: the first two are associated with a Gaussian and an exponential decay, respectively, and can be described using linear response theory. For stronger perturbations F(t) decays algebraically as F(t) approximately t(-D2(mu)), where D2(mu) is the correlation dimension of the local density of states.