Identifying ordering in non-crystalline solids has been a focus of natural science since the publication of Zachariasen's random network theory in 1932, but it still remains as a great challenge of the century. Literature shows that the hierarchical structures, from the short-range order of first-shell polyhedra to the long-range order of translational periodicity, may survive after amorphization. Here, in a piece of AlPO4, or berlinite, we combine X-ray diffraction and stochastic free-energy surface simulations to study its phase transition and structural ordering under pressure. From reversible single crystals to amorphous transitions, we now present an unambiguous view of the topological ordering in the amorphous phase, consisting of a swarm of Carpenter low-symmetry phases with the same topological linkage, trapped in a metastable intermediate stage. We propose that the remaining topological ordering is the origin of the switchable "memory glass" effect. Such topological ordering may hide in many amorphous materials through disordered short atomic displacements.