Crystal structure prediction (CSP) for inorganic materials is one of the central and most challenging problems in materials science and computational chemistry. This problem can be formulated as a global optimization problem in which global search algorithms such as genetic algorithms (GAs) and particle swarm optimization have been combined with first-principles free-energy calculations to predict crystal structures given only the material composition or a chemical system. These DFT-based ab initio CSP algorithms are computationally demanding and can usually be used only to predict crystal structures of relatively small systems. The vast coordinate space and the expensive DFT free-energy calculations limit their inefficiency and scalability. On the other hand, a similar structure prediction problem has been intensively investigated in parallel in the protein structure prediction (PSP) community of bioinformatics, in which the dominating predictors are knowledge-based approaches including homology modeling and threading that exploit known protein structures. Surprisingly, the CSP field has mainly focused on ab initio approaches in the past decade. Inspired by the knowledge-rich PSP approaches, herein, we explore whether known geometric constraints such as the pairwise atomic distances of a target crystal material can help predict/reconstruct its structure given its space group and lattice information. We propose DMCrystal, a GA-based crystal structure reconstruction algorithm based on predicted pairwise atomic distances. Based on extensive experiments, we show that the predicted distance matrix can dramatically help reconstruct the crystal structure and usually achieves much better performance than that of CMCrystal, an atomic contact map-based CSP algorithm. This implies that the knowledge of atomic interaction information learned from the existing materials can be used to significantly improve the CSP in terms of both speed and quality.