Many new clinical investigations of microwave breast imaging have been published in recent years. Trials with over one hundred participants have indicated the potential of microwave imaging to detect breast cancer, with particularly encouraging sensitivity results reported from women with dense breasts. The next phase of clinical trials will involve larger and more diverse populations, including women with no breast abnormalities or benign breast diseases. These trials will need to address clinical efficacy in terms of sensitivity and specificity. A number of challenges exist when using microwave imaging with broad populations: 1) addressing the substantial variance in breast composition observed in the population and 2) achieving high specificity given differences between individuals. This paper analyses these challenges using a diverse phantom set which models the variance in breast composition and tumor shape and size seen in the population. The data show that the sensitivity of microwave breast imaging in breasts of differing density can suffer if patient-specific beamforming is not used. Moreover, the results suggest that achieving high specificity in dense breasts may be difficult, but that patient-specific beamforming does not adversely affect the expected specificity. In summary, this paper finds that patient-specific beamforming has a tangible impact on expected sensitivity in experimental cases and that achieving high specificity in dense breasts may be challenging.