Augmented reality (AR) blends the digital and physical worlds by overlapping a virtual image onto the see-through physical environment. However, contrast reduction and noise superposition in an AR head-mounted display (HMD) can substantially limit image quality and human perceptual performance in both the digital and physical spaces. To assess image quality in AR, we performed human and model observer studies for various imaging tasks with targets placed in the digital and physical worlds. A target detection model was developed for the complete AR system including the optical see-through. Target detection performance using different observer models developed in the spatial frequency domain was compared with the human observer results. The non-prewhitening model with eye filter and internal noise results closely track human perception performance as measured by the area under the receiver operating characteristic curve (AUC), especially for tasks with high image noise. The AR HMD non-uniformity limits the low-contrast target (less than 0.02) observer performance for low image noise. In augmented reality conditions, the detectability of a target in the physical world is reduced due to the contrast reduction by the overlaid AR display image (AUC less than 0.87 for all the contrast levels evaluated). We propose an image quality optimization scheme to optimize the AR display configurations to match observer detection performance for targets in both the digital and physical worlds. The image quality optimization procedure is validated using both simulation and bench measurements of a chest radiography image with digital and physical targets for various imaging configurations.