To address the needs of polarized light navigation for accurate position information of feature points in the sky, an accurate solar position detection method based on an all-sky polarization pattern imaging system is proposed. Unlike the traditional spot-based solar position detection method, this method uses the polarization information inherent in the atmosphere to accurately measure solar position. This approach is characterized by simple detection, high accuracy, and wide application range. The optical acquisition system is composed of three miniature large-field camera modules and polarizers, which enables a more compact structure, smaller size, and lesser height. Based on this principle, the solar position solution algorithm was simulated and then verified in various weather environments using the optical acquisition system built as part of this study. Solar position was detected at different moments on the same day in clear weather, and the accuracy of the measured solar altitude and azimuth angles was 0.024° and 0.03°, respectively. The accuracy of the measured solar altitude and azimuth angles was 0.08° and 0.05°, respectively, when the sun was shielded by high-rise buildings and 0.3° and 0.1° when the sun was shielded by branches and tree leaves. Aerosol concentrations exceeding a certain amount destroyed the Rayleigh distribution pattern of polarized light, thus affecting solar position detection accuracy. It is concluded that this novel detection method can not only meet the needs of polarized light navigation for solar position, but also provide a new exploration idea for enthusiasts who are eager to explore the mysteries of the universe.