Using published high-resolution data on S-phase replication timing, we determined the three-dimensional (3D) nuclear arrangement of 33 very-early-replicating and 31 very-late-replicating loci. We analyzed diploid human, non-human primate and rearranged tumor cells by 3D fluorescence in situ hybridization with the aim of investigating the impact of chromosomal structural changes on the nuclear organization of these loci. Overall, their topology was found to be largely conserved between cell types, species and in tumor cells. Early-replicating loci were localized in the nuclear interior, whereas late-replicating loci showed a broader distribution with a higher preference for the periphery than for late-BrdU-incorporation foci. However, differences in the spatial arrangement of early and late loci of chromosome 2, as compared with those from chromosome 5, 7 and 17, argue against replication timing as a major driving force for the 3D radial genome organization in human lymphoblastoid cell nuclei. Instead, genomic properties, and local gene density in particular, were identified as the decisive parameters. Further detailed comparisons of chromosome 7 loci in primate and tumor cells suggest that the inversions analyzed influence nuclear topology to a greater extent than the translocations, thus pointing to geometrical constraints in the 3D conformation of a chromosome territory.