In this work we report the metastability and the energetics of the phase transitions of three different polymorphs of BiPO4, namely trigonal (Phase-I, space group P3(1)21), monoclinic monazite-type (Phase-II, space group P2(1)/n) and SbPO4-type monoclinic (Phase-III, space group P2(1)/m) from ambient and non-ambient temperature powder XRD and neutron diffraction studies as well as ab initio density functional theory (DFT) calculations. The symmetry ambiguity between P2(1) and P2(1)/m of the high temperature polymorph of BiPO4 has been resolved by a neutron diffraction study. The structure and vibrational properties of these polymorphs of the three polymorphs have also been reported in detail. Total energy calculations have been used to understand the experimentally observed metastable behavior of trigonal and monazite-type BiPO4. Interestingly, all of the three phases were found to coexist after heating a single phasic trigonal BiPO4 to 773 K. The irreversible nature of these phase transitions has been explained by the concepts of the interplay of the structural distortion, molar volume and total energy.