Akin to the enormous number of discoveries made through traditional semiconductor alloys, alloying selected 2D semiconductors enables engineering of their electronic structure for a wide range of new applications. 2D alloys have been demonstrated when two components crystallized in the same phase, and their bandgaps displayed predictable monotonic variation. By stabilizing previously unobserved compositions and phases of GaSe1-xTe x at nanoscales on GaAs(111), we demonstrate abnormal band bowing effects and phase instability region when components crystallize in different phases. Advanced microscopy and spectroscopy measurements show as tellurium is alloyed into GaSe, nanostructures undergo hexagonal to monoclinic and isotropic to anisotropic transition. There exists an instability region (0.56 < x < 0.67) where both phases compete and coexist, and two different bandgap values can be found at the same composition leading to anomalous band bowing effects. Results highlight unique alloying effects, not existing in single-phase alloys, and phase engineering routes for potential applications in photonic and electronics.