It is fascinating that fish groups spontaneously form different formations. The collective locomotions of two and multiple undulatory self-propelled foils swimming in a fluid are numerically studied and the deep reinforcement learning (DRL) is applied to control the locomotion. We explored whether typical patterns emerge spontaneously under the driven two DRL strategies. One strategy is that only the following fish gets hydrodynamic advantages. The other is that all individuals in the group take advantage of the interaction. In the DRL strategy, we use swimming efficiency as the reward function, and the visual information is included. We also investigated the effect of involving hydrodynamic force information, which is an analogy to that detected by the lateral line of fish. Each fish can adjust its undulatory phase to achieve the goal. Under the two strategies, collective patterns with different characteristics, i.e., the staggered-following, tandem-following phalanx and compact modes emerge. They are consistent with the results in the literature. The hydrodynamic mechanism of the above high-efficiency collective traveling modes is analyzed by the vortex-body interaction and thrust. We also found that the time sequence feature and hydrodynamic information in the DRL are essential to improve the performance of collective swimming. Our research can reasonably explain the controversial issue observed in the relevant experiments. The paper may be helpful for the design of bionic fish.