Mammalian females repress gene expression from one of their two X chromosomes to compensate for the gene dosage difference between females and males, via a process called X chromosome inactivation (XCI). Since the first discovery of XCI 50 years ago, the knowledge of this phenomenon has greatly contributed to a better understanding of the molecular mechanism that controls the epigenetic regulation of gene expression. The key molecule that organizes the chromatin-level repression is an X-linked 17-kb non-coding RNA named Xist. The transcripts of Xist are localized along the entire length of the X chromosome and subsequently recruit a chromatin remodeling complex that introduces the repressive epigenetic modifications. In the present review, we will highlight the recent findings that have illustrated the close relationship between XCI and the structural component of the nucleus called the nuclear scaffold/matrix, with an emphasis on the function of the bona-fide scaffold/matrix-binding protein hnRNP U/SAF-A.