In this paper, we propose a hierarchical simulated annealing of erosion method (HSAE) to improve the computational efficiency of multiphase microstructure reconstruction, whose computational efficiency can be improved by an order of magnitude. Reconstruction of the two-dimensional (2D) and three-dimensional (3D) multiphase microstructures (pore, grain, and clay) based on simulated annealing (SA) and HSAE are performed. In the reconstruction of multiphase microstructure with HSAE and SA, three independent two-point correlation functions are chosen as the morphological information descriptors. The two-point cluster function which contains significant high-order statistical information is used to verify the reconstruction results. From the analysis of 2D reconstruction, it can find that the proposed HSAE technique not only improves the quality of reconstruction, but also improves the computational efficiency. The reconstructions of our proposed method are still imperfect. This is because the used two-point correlation functions contain insufficient information. For the 3D reconstruction, the two-point correlation functions of the 3D generation are in excellent agreement with those of the original 2D image, which illustrates that our proposed method is effective for the reconstruction of 3D microstructure. The comparison of the energy vs computational time between the SA and HSAE methods shows that our presented method is an order of magnitude faster than the SA method. That is because only some of the pixels in the overall hierarchy need to be considered for sampling.