Despite comparable levels of virus replication, simian immunodeficiency viruses (SIV) infection is non-pathogenic in natural hosts, such as sooty mangabeys (SM), whereas it is pathogenic in non-natural hosts, such as rhesus macaques (RM). Comparative studies of pathogenic and non-pathogenic SIV infection can thus shed light on the role of specific factors in SIV pathogenesis. Here, we determine the impact of target-cell limitation, CD8+ T cells, and Natural Killer (NK) cells on virus replication in the early SIV infection. To this end, we fit previously published data of experimental SIV infections in SMs and RMs with mathematical models incorporating these factors and assess to what extent the inclusion of individual factors determines the quality of the fits. We find that for both rhesus macaques and sooty mangabeys, target-cell limitation alone cannot explain the control of early virus replication, whereas including CD8+ T cells into the models significantly improves the fits. By contrast, including NK cells does only significantly improve the fits in SMs. These findings have important implications for our understanding of SIV pathogenesis as they suggest that the level of early CD8+ T cell responses is not the key difference between pathogenic and non-pathogenic SIV infection.