Understanding glass structure is still challenging due to the result of disorder, although novel materials design on the basis of atomistic structure has been strongly demanded. Here we report on the atomic structures of the zinc phosphate glass determined by reverse Monte Carlo modelling based on diffraction and spectroscopic data. The zinc-rich glass exhibits the network formed by ZnOx (averaged x<4) polyhedra. Although the elastic modulus, refractive index and glass transition temperature of the zinc phosphate glass monotonically increase with the amount of ZnO, we find for the first time that the thermal expansion coefficient is very sensitive to the substitution of the phosphate chain network by a network consisting of Zn-O units in zinc-rich glass. Our results imply that the control of the structure of intermediate groups may enable new functionalities in the design of oxide glass materials.