A wavelength selection method that combines an inverse Monte Carlo model of reflectance and a genetic algorithm for global optimization was developed for the application of spectral imaging of breast tumor margins. The selection of wavelengths impacts system design in cost, size, and accuracy of tissue quantitation. The minimum number of wavelengths required for the accurate quantitation of tissue optical properties is 8, with diminishing gains for additional wavelengths. The resulting wavelength choices for the specific probe geometry used for the breast tumor margin spectral imaging application were tested in an independent pathology-confirmed ex vivo breast tissue data set and in tissue-mimicking phantoms. In breast tissue, the optical endpoints (hemoglobin, β-carotene, and scattering) that provide the contrast between normal and malignant tissue specimens are extracted with the optimized 8-wavelength set with <9% error compared to the full spectrum (450-600 nm). A multi-absorber liquid phantom study was also performed to show the improved extraction accuracy with optimization and without optimization. This technique for selecting wavelengths can be used for designing spectral imaging systems for other clinical applications.