Xist RNA, the master regulator of X chromosome inactivation, acts in cis to induce chromosome-wide silencing. Whilst recent studies have defined candidate silencing factors, their relative contribution to repressing different genes, and their relationship with one another is poorly understood. Here we describe a systematic analysis of Xist-mediated allelic silencing in mouse embryonic stem cell-based models. Using a machine learning approach we identify distance to the Xist locus and prior gene expression levels as key determinants of silencing efficiency. We go on to show that Spen, recruited through the Xist A-repeat, plays a central role, being critical for silencing of all except a subset of weakly expressed genes. Polycomb, recruited through the Xist B/C-repeat, also plays a key role, favouring silencing of genes with pre-existing H3K27me3 chromatin. LBR and the Rbm15/m6A-methyltransferase complex make only minor contributions to gene silencing. Together our results provide a comprehensive model for Xist-mediated chromosome silencing.