Asphaltene aggregation is critical to many natural and industrial processes, from groundwater contamination and remediation to petroleum utilization. Despite extensive research in the past few decades, the fundamental process of sulfur-rich asphaltene aggregation still remains not fully understood. In this work, we have investigated the particle-by-particle growth of aggregates formed with sulfur-rich asphaltene by a combined approach of in situ microscopy and molecular simulation. The experimental results show that aggregates assembled from sulfur-rich asphaltene have morphologies with time-dependent structural self-similarity, and their growth rates are aligned with a crossover behavior between classic reaction-limited aggregation and diffusion-limited aggregation. Although the particle size distribution predicted using the Smoluchowski equation deviates from the observations at the initial stage, it provides a reasonable prediction of aggregate size distribution at the later stage, even if the observed cluster coalescence has an important effect on the corresponding cluster size distribution. The simulation results show that aliphatic sulfur exerts nonmonotonic effects on asphaltene nanoaggregate formation depending on the asphaltene molecular structure. Specifically, aliphatic sulfur has a profound effect on the structure of rod-like nanoaggregates, especially when asphaltene molecules have small aromatic cores. Interactions between aliphatic sulfur and the side chain of neighboring molecules account for the repulsive forces that largely explain the polydispersity in the nanoaggregates and corresponding colloidal aggregates. These results can improve our current understanding of the complex process of sulfur-rich asphaltene aggregation and sheds light on designing efficient crude oil utilization and remediation technologies.