Liver cancer is one of the most common cancers leading to death in the world. Microvascular invasion (MVI) is a principal reason for the poor long-term survival rate after liver cancer surgery. Early detection and treatment are very important for improving the survival rate. Manual examination of MVI based on histopathological images is very inefficient and time consuming. MVI automatic diagnosis based on deep learning methods can effectively deal with this problem, reduce examination time, and improve detection efficiency. In recent years, deep learning-based methods have been widely used in histopathological image analysis because of their impressive performance. However, it is very challenging to identify MVI directly using deep learning methods, especially under the interference of hepatocellular carcinoma (HCC) because there is no obvious difference in the histopathological level between HCC and MVI. To cope with this problem, we adopt a method of classifying the MVI boundary to avoid interference from HCC. Nonetheless, due to the specificity of the histopathological tissue structure with the MVI boundary, the effect of transfer learning using the existing models is not obvious. Therefore, in this paper, according to the features of the MVI boundary histopathological tissue structure, we propose a new classification model, i.e., the PCformer, which combines the convolutional neural network (CNN) method with a visual transformer and improves the recognition performance of the MVI boundary histopathological image. Experimental results show that our method has better performance than other models based on a CNN or a transformer.