The efficiency of a virus to establish its infection in host cells varies broadly among viruses. It remains unclear if there is a key step in this process that controls viral infectivity. To address this question, we use single-particle tracking and Brownian dynamics simulation to examine human immunodeficiency virus type 1 (HIV-1) infection in cell culture. We find that the frequency of viral-cell encounters is consistent with diffusion-limited interactions. However, even under the most favorable conditions, only 1% of the viruses can become immobilized on cell surface and subsequently enter the cell. This is a result of weak interaction between viral surface gp120 and CD4 receptor, which is insufficient to form a stable complex the majority of the time. We provide the first direct quantitation for efficiencies of these events relevant to measured HIV-1 infectivity and demonstrate that immobilization on host cell surface post-virion-diffusion is the key step in viral infection. Variation of its probability controls the efficiency of a virus to infect its host cells. These results explain the low infectivity of cell-free HIV-1 in vitro and offer a potential rationale for the pervasive high efficiency of cell-to-cell transmission of animal viruses.