Automatic efficient spike sorting is one of the biggest challenges for the neural recording microsystem online. An unsupervised spike sorting method is proposed in this paper, based on the hybrid neural network with principal component analysis network (PCAN) and normal boundary response (NBR) self-organizing map network (SOMN) classifier. The PCAN extracted the spike features with the dimension reduced and correlation eliminated; The SOM network perform the spike distribution in the feature space, thus after convergence, the weights of the neurons demonstrate the spike cluster distribution in the feature space; At last the spike sorting was finished by computing the neurons' Normal Boundary Response (NBR) which determined the neurons' classes. The experimental results show that, based on hybrid neural network spiking sorting algorithm, it can achieve the accuracy above 97.91% with signals containing five classes. The novel classification algorithm proposed is to further improve the efficient and adaptive of classification system.