A miniaturized portable short-wave infrared imaging spectroscopy optical system is designed based on a prism-grating dispersion module. We established a prism-grating model to calculate the optimal combination of prism and grating parameters to balance spectral smile over the entire band. The design method for the telescopic system and spectroscopic system combines independent design with integrated optimization. The system's spectral smile and spectral keystone are less than 15 μm and less than half a pixel, respectively. The total optical system length is 230 mm, which meets the miniaturization requirements for airborne systems. The system's spatial resolution is 1 mrad, and its average spectral resolution is 6.2 nm. The system offers the advantages of large relative aperture, excellent imaging quality, reduced spectral smile, and spectral keystone, miniaturization, and portability.