The Mg-Y-Zn ternary alloy system contains a series of novel structures known as long-period stacking ordered (LPSO) structures. The formation process and its key concept from a viewpoint of phase transition are not yet clear. The current study reveals that the phase transformation process is not a traditional spinodal decomposition or structural transformation but, rather a novel hierarchical phase transformation. In this transformation, clustering occurs first, and the spatial rearrangement of the clusters induce a secondary phase transformation that eventually lead to two-dimensional ordering of the clusters. The formation process was examined using in situ synchrotron radiation small-angle X-ray scattering (SAXS). Rapid quenching from liquid alloy into thin ribbons yielded strongly supersaturated amorphous samples. The samples were heated at a constant rate of 10 K/min. and the scattering patterns were acquired. The SAXS analysis indicated that small clusters grew to sizes of 0.2 nm after they crystallized. The clusters distributed randomly in space grew and eventually transformed into a microstructure with two well-defined cluster-cluster distances, one for the segregation periodicity of LPSO and the other for the in-plane ordering in segregated layer. This transformation into the LPSO structure concomitantly introduces the periodical stacking fault required for the 18R structures.