Interphase chromosomes in Saccharomyces cerevisiae are tethered to the nuclear envelope at their telomeres and to the spindle pole body (SPB) at their centromeres. Using a polymer model of yeast chromosomes that includes these interactions, we show theoretically that telomere attachment to the nuclear envelope is a major determinant of gene positioning within the nucleus only for genes within 10 kb of the telomeres. We test this prediction by measuring the distance between the SPB and the silent mating locus (HML) on chromosome III in wild-type and mutant yeast strains that contain altered chromosome-tethering interactions. In wild-type yeast cells we find that disruption of the telomere tether does not dramatically change the position of HML with respect to the SPB, in agreement with theoretical predictions. Alternatively, using a mutant strain with a synthetic tether that localizes an HML-proximal site to the nuclear envelope, we find a significant change in the SPB-HML distance, again as predicted by theory. Our study quantifies the importance of tethering at telomeres on the organization of interphase chromosomes in yeast, which has been shown to play a significant role in determining chromosome function such as gene expression and recombination.