Recognition of different types of fiber vibration signals is important for the optical fiber prewarning system (OFPS). Nowadays, the recognition of fiber vibration signals with neural networks is one of the common methods in this field. As a small, well-trained network, stochastic configuration networks (SCNs) can achieve good results when applied to fiber vibration signal recognition. However, in the case of a limited number of vibration signals, the recognition rate of SCNs is also limited. In order to improve the recognition rate of vibration signals, this paper proposes the AdaBoost-SCN algorithm. It integrates different SCNs as base classifiers in AdaBoost. The experiments show that the testing accuracy of the AdaBoost-SCN algorithm is 12.1% higher than that of the original SCN when training with a small vibration signal set. The algorithm proposed in this paper not only increases the recognition rate of fiber vibration signals, but also improves the generalization ability of the original SCN in the case of a limited number of vibration signal samples.