Viral replication requires the help of host cell factors, whose species specificity may affect viral tropism. On the other hand, there exist host factors that restrict viral replication. The anti-viral system mediated by some of these restriction factors, which is termed intrinsic immunity and is distinguished from conventional innate and adaptive immunity, has been described as playing an important role in making species-specific barriers against viral infection. Here, we describe the current progress in understanding of such restriction factors against retroviral replication, focusing on TRIM5alpha and APOBEC, whose anti-retroviral effects have recently been recognized. Additionally, we mention cyclophilin A, which is essential for HIV-1 replication in human cells and may affect viral tropism. Understanding of these host factors would contribute to identification of the determinants for viral tropism.