In recent studies, the nuclear domain 10 (ND10) components PML and hDaxx were identified as cellular restriction factors that inhibit the initiation of human cytomegalovirus (HCMV) replication. The antiviral function of ND10, however, is antagonized by the IE1 protein, which induces ND10 disruption. Here we show that IE1 not only de-SUMOylates PML immediately upon infection but also directly targets Sp100. IE1 expression alone was sufficient to downregulate endogenous Sp100 independently of the presence of PML. Moreover, cotransfection experiments revealed that IE1 negatively interferes with the SUMOylation of all Sp100 isoforms. The modulation of Sp100 at immediate-early (IE) times of infection, indeed, seemed to have an in vivo relevance for HCMV replication, since knockdown of Sp100 resulted in more cells initiating the viral gene expression program. In addition, we observed that Sp100 was degraded in a proteasome-dependent manner at late times postinfection, suggesting that Sp100 may play an additional antiviral role during the late phase. Infection experiments conducted with Sp100 knockdown human foreskin fibroblasts (HFFs) confirmed this hypothesis: depletion of Sp100 resulted in augmented release of progeny virus particles compared to that from control cells. Consistent with this observation, we noted increased amounts of viral late gene products in the absence of Sp100. Importantly, this elevated late gene expression was not dependent on enhanced viral IE gene expression. Taken together, our data provide evidence that Sp100 is the first ND10-related factor identified that not only possesses the potential to restrict the initial stage of infection but also inhibits HCMV replication during the late phase.