III-V growth and surface conditions strongly influence the physical structure and resulting optical properties of self-assembled quantum dots (QDs). Beyond the design of a desired active optical wavelength, the polarization response of QDs is of particular interest for optical communications and quantum information science. Previous theoretical studies based on a pure InAs QD model failed to reproduce experimentally observed polarization properties. In this work, multi-million atom simulations are performed in an effort to understand the correlation between chemical composition and polarization properties of QDs. A systematic analysis of QD structural parameters leads us to propose a two-layer composition model, mimicking In segregation and In-Ga intermixing effects. This model, consistent with mostly accepted compositional findings, allows us to accurately fit the experimental PL spectra. The detailed study of QD morphology parameters presented here serves as a tool for using growth dynamics to engineer the strain field inside and around the QD structures, allowing tuning of the polarization response.