Liver cancer has been one of the top causes of cancer-related death. For developing an accurate treatment strategy and raising the survival rate, the differentiation of liver cancers is essential. Multiphase CT recently acts as the primary examination method for clinical diagnosis. Deep learning techniques based on multiphase CT have been proposed to distinguish hepatic cancers. However, due to the recurrent mechanism, RNN-based approaches require expensive calculations whereas CNN-based models fail to explicitly establish temporal correlations among phases. In this paper, we proposed a phase difference network, termed as Phase Difference Network (PDN), to identify two liver cancer, hepatocellular carcinoma and intrahepatic cholangiocarcinoma, from four-phase CT. Specifically, the phase difference was used as interphase temporal information in a differential attention module, which enhanced the feature representation. Additionally, utilizing a multihead self-attention module, a transformer-based classification module was employed to explore the long-term context and capture the temporal relation between phases. Clinical datasets are used in experiments to compare the performance of the proposed strategy versus conventional approaches. The results indicate that the proposed method outperforms the traditional deep learning based methods.