Residual stress birefringence of highly reflective mirrors is a challenging problem due to its dubious origin and intricate nature. In this paper, the birefringences of highly reflective mirrors manufactured under the same deposition parameters but structured with different numbers of high- and low-refractive-index (HL) layer pairs are measured with the cavity ring-down technique by the mirror rotation method together with a differential loss approximation model. Experimental results show that birefringence retardation increases with the increasing number of HL layer pairs. Further measurements across the mirror surface indicate a non-uniform birefringence distribution, while curvature analysis of the stress-deformed surface provides more clues to the origins of birefringence.