Liver cancer is a part of the common causes of cancer death worldwide, and the accurate diagnosis of hepatic malignancy is important for effective next treatment. In this paper, we propose a convolutional neural network (CNN) based on a spatiotemporal excitation (STE) module for identification of hepatic malignancy in four-phase computed tomography (CT) images. To enhance the display detail of lesion, we expand single-channel CT images into three channels by using the channel expansion method. Our proposed STE module consists of a spatial excitation (SE) module and a temporal interaction (TI) module. The SE module calculates the temporal differences of CT slices at the feature level, which is used to excite shape-sensitive channels of the lesion features. The TI module shifts a portion of the channels in the temporal dimension to exchange information among the current CT slice and adjacent CT slices. Four-phase CT images of 398 patients diagnosed with hepatocellular carcinoma (HCC) and intrahepatic cholangiocarcinoma (ICC) are used for experiments and five cross-validations are performed. Our model achieved average accuracy of 85.00% and average AUC of 88.91% for classifying HCC and ICC.Clinical Relevance- The proposed deep learning-based model can perform HCC and ICC recognition tasks based on four-phase CT images, assisting doctors to obtain better diagnostic performance.