The interaction between the swarm individuals affects the dynamic behavior of the swarm, but it is difficult to obtain directly from outside observation. Therefore, the problem we focus on is inferring the structure of the interactions in the swarm from the individual behavior trajectories. Similar inference problems that existed in network science are named network reconstruction or network inference. It is a fundamental problem pervading research on complex systems. In this paper, a new method, called Motion Trajectory Similarity, is developed for inferring direct interactions from the motion state of individuals in the swarm. It constructs correlations by combining the similarity of the motion trajectories of each cross section of the time series, in which individuals with highly similar motion states are more likely to interact with each other. Experiments on the flocking systems demonstrate that our method can produce a reliable interaction inference and outperform traditional network inference methods. It can withstand a high level of noise and time delay introduced into flocking models, as well as parameter variation in the flocking system, to achieve robust reconstruction. The proposed method provides a new perspective for inferring the interaction structure of a swarm, which helps us to explore the mechanisms of collective movement in swarms and paves the way for developing the flocking models that can be quantified and predicted.