Haptic temporal signal recognition plays an important supporting role in robot perception. This paper investigates how to improve classification performance on multiple types of haptic temporal signal datasets using a Transformer model structure. By analyzing the feature representation of haptic temporal signals, a Transformer-based two-tower structural model, called Touchformer, is proposed to extract temporal and spatial features separately and integrate them using a self-attention mechanism for classification. To address the characteristics of small sample datasets, data augmentation is employed to improve the stability of the dataset. Adaptations to the overall architecture of the model and the training and optimization procedures are made to improve the recognition performance and robustness of the model. Experimental comparisons on three publicly available datasets demonstrate that the Touchformer model significantly outperforms the benchmark model, indicating our approach's effectiveness and providing a new solution for robot perception.