Fatigue cracking is an essential problem of asphalt concrete that contributes to pavement damage. Although stone matrix asphalt (SMA) has significantly provided resistance to rutting failure, its resistance to fatigue failure is yet to be fully addressed. The aim of this study is to evaluate the effect of crumb rubber modifier (CRM) on stiffness and fatigue properties of SMA mixtures at optimum binder content, using four different modification levels, namely, 6%, 8%, 10%, and 12% CRM by weight of the bitumen. The testing undertaken on the asphalt mix comprises the dynamic stiffness (indirect tensile test), dynamic creep (repeated load creep), and fatigue test (indirect tensile fatigue test) at temperature of 25°C. The indirect tensile fatigue test was conducted at three different stress levels (200, 300, and 400 kPa). Experimental results indicate that CRM-reinforced SMA mixtures exhibit significantly higher fatigue life compared to the mixtures without CRM. Further, higher correlation coefficient was obtained between the fatigue life and resilient modulus as compared to permanent strain; thus resilient modulus might be a more reliable indicator in evaluating the fatigue life of asphalt mixture.