The purpose of this study was to determine the relationships between selected physical qualities, change-of-direction (COD) speed, and defensive agility performance in Australian Rules football players. Twenty-four male community-level players were assessed on sprint acceleration (10-m time), maximum strength (3 repetition-maximum half squat), leg power (countermovement jump), reactive strength (drop jump), and a single COD speed test and a defensive agility test. Change-of-direction speed was correlated with reactive strength (r = -0.645, p = 0.001) and sprint acceleration (r = 0.510, p = 0.011). Multiple regression indicated that the combined physical qualities explained 56.7% of the variance associated with COD speed (adjusted R = 0.567, p ≤ 0.05). Participants were median split into faster and slower COD speed groups, and these were compared by independent t-tests. The faster group was significantly better (p ≤ 0.05) on the sprint acceleration and reactive strength tests (large effect size). The correlations between physical qualities and agility were trivial to small (r = -0.101 to 0.123, p > 0.05) and collectively explained only 14.2% of the variance associated with agility performance (adjusted R = -0.142, p > 0.05). When faster and slower agility groups were compared, there were trivial to moderate differences (p > 0.05) in all physical qualities. It was concluded that reactive strength and sprint acceleration are important for COD speed, but the physical qualities assessed are not associated with defensive agility performance. For agility tasks similar to those in this study, sprint and resistance training should not be emphasized, and training other factors, such as the development of sport-specific technique and cognitive skill, is recommended.