Nanosized α-zirconium phosphate, α-ZrP, undergoes a phase transition at 120 °C, which is not observed with microcrystalline α-ZrP in the same conditions, and which leads to a new 3D phase. The new compound, with formula Zr(HPO4)2 (τ'-ZrP), consists of cubelike nanoparticles and has a tetragonal unit cell (space group P43212, a = 7.955 Å, c = 10.744 Å). The structure of τ'-ZrP is in close relationship with that of the already known τ-ZrP. Both structures are made of packed chains of eight-membered rings, composed of Zr atoms connected to bridging HPO4 groups. The main difference between the two structures concerns the different orientation of the uncoordinated P-OH groups, pointing into the channels. The in situ XRPD analysis on nanosized α-ZrP, performed at 120 °C as a function of time, provided information about the kinetics of the formation of τ'-ZrP, showing that the α-ZrP phase is directly transformed into τ'-ZrP. Moreover, τ'-ZrP is converted into α-ZrP at room temperature in the presence of water vapor. It was proved that the free phosphoric acid, which is originally present in small amounts in nanosized α-ZrP and τ'-ZrP, is necessary for the interconversion between the two phases. As a matter of fact, the removal of phosphoric acid, by washing α-ZrP and τ'-ZrP with anhydrous ethanol, inhibits the above conversion.