Cytomegaloviruses (CMVs), members of the herpesvirus family, have evolved a variety of mechanisms to evade the immune response to survive in infected hosts and to establish latent infection. They effectively hide infected cells from the effector mechanisms of adaptive immunity by eliminating cellular proteins (major histocompatibility Class I and Class II molecules) from the cell surface that display viral antigens to CD8 and CD4 T lymphocytes. CMVs also successfully escape recognition and elimination of infected cells by natural killer (NK) cells, effector cells of innate immunity, either by mimicking NK cell inhibitory ligands or by downregulating NK cell-activating ligands. To accomplish these immunoevasion functions, CMVs encode several proteins that function in the biosynthetic pathway by inhibiting the assembly and trafficking of cellular proteins that participate in immune recognition and thereby, block their appearance at the cell surface. However, elimination of these proteins from the cell surface can also be achieved by perturbation of their endosomal route and subsequent relocation from the cell surface into intracellular compartments. Namely, the physiological route of every cellular protein, including immune recognition molecules, is characterized by specific features that determine its residence time at the cell surface. In this review, we summarize the current understanding of endocytic trafficking of immune recognition molecules and perturbations of the endosomal system during infection with CMVs and other members of the herpesvirus family that contribute to their immune evasion mechanisms.