While the thermomechanical properties of sapphire make it an excellent candidate of test mass for advanced laser interferometers, its optical quality is not well understood or well controlled. We have studied the results from high-resolution measurements of scattering, absorption, and birefringence in test-mass samples to better understand issues of quality. Samples show large-scale scattering structures clearly linked to the crystal-growth process. Samples characterized by the presence of point defects have significantly lower scattering (except at the point defects). In general on a large scale, high scattering also correlates with higher absorption and higher average birefringence inhomogeneity. However, on a smaller scale there is not a clear point-to-point correlation between scattering and absorption. Often a large-scale scattering structure is spatially displaced by tens of millimeters from a similar absorption structure, indicating that quite separate microscopic mechanisms give rise to scattering and absorption. The spatial displacements indicate that absorption centers and scattering centers are laid down during crystal growth at different distances from the solid-liquid interface. We suggest that absorption may be linked to F centers, while scattering may be linked to impurities such as iron.