Texture compression is widely used in real-time rendering to reduce storage and bandwidth requirements. Recent research in compression algorithms has explored both reduced fixed bit rate and variable bit rate algorithms. The results are evaluated at the individual texture level using mean square error, peak signal-to-noise ratio, or visual image inspection. We argue this is the wrong evaluation approach. Compression artifacts in individual textures are likely visually masked in final rendered images and this masking is not accounted for when evaluating individual textures. This masking comes from both geometric mapping of textures onto models and the effects of combining different textures on the same model such as diffuse, gloss, and bump maps. We evaluate final rendered images using rigorous perceptual error metrics. Our method samples the space of viewpoints in a scene, renders the scene from each viewpoint using variations of compressed textures, and then compares each to a ground truth using uncompressed textures from the same viewpoint. We show that masking has a significant effect on final rendered image quality, masking effects and perceptual sensitivity to masking varies by the type of texture, graphics hardware compression algorithms are too conservative, and reduced bit rates are possible while maintaining final rendered image quality.