Here, we report a high-pressure study of orthorhombic structured β-Sb2O3 (valentinite) by the combination of synchrotron in situ X-ray diffraction and first-principles theoretical calculations at pressures up to 40.5 GPa. Our results reveal that the metastable β-Sb2O3 undergoes an isostructural phase transition at high pressure, yielding a distorted β phase at 7-15 GPa through symmetry breaking and structural distortion as inferred from our XRD analyses and DFT theoretical calculations where pressure-induced elasticity softening is observed at pressures of 7-15 GPa. At pressures higher than 15 GPa, a new high-pressure monoclinic phase is discovered from the current synchrotron X-ray diffraction data. Upon further compression up to ∼33 GPa, the monoclinic Sb2O3 starts to lose its long-range order and forms an amorphous component coexisting with the monoclinic one. To further explore the structural instability and understand the origin of pressure-induced phase transitions in β-Sb2O3 upon compression, we have performed first-principles calculations to track the evolution of its phonon velocities, density of states and phonon dispersion curves under high pressure. Our results may play an important role in determining the local structures as well as their structural relationship among sesquioxides.