We systematically study the decay of quasi-two-dimensional vortices in an oblate strongly interacting Fermi gas over a wide interaction range and observe that, as the system temperature is lowered, the vortex lifetime increases in the Bose-Einstein condensate (BEC) regime but decreases at unitarity and in the Bardeen-Cooper-Schrieffer (BCS) regime. The observations can be qualitatively captured by a phenomenological model simply involving diffusion and two-body collisional loss, in which the vortex lifetime is mostly determined by the slower process of the two. In particular, the counterintuitive vortex decay in the BCS regime can be interpreted by considering the competition between the temperature dependence of the vortex annihilation rate and that of unpaired fermions. Our results suggest a competing mechanism for the complex vortex decay dynamics in the BCS-BEC crossover for the fermionic superfluids.