High early recurrence (ER) rate is the main factor leading to the poor outcome of patients with hepatocellular carcinoma (HCC). Accurate preoperative prediction of ER is thus highly desired for HCC treatment. Many radiomics solutions have been proposed for the preoperative prediction of HCC using CT images based on machine learning and deep learning methods. Nevertheless, most current radiomics approaches extract features only from segmented tumor regions that neglect the liver tissue information which is useful for HCC prognosis. In this work, we propose a deep prediction network based on CT images of full liver combined with tumor mask that provides tumor location information for better feature extraction to predict the ER of HCC. While, due to the complex imaging characteristics of HCC, the image-based ER prediction methods suffer from limited capability. Therefore, on the one hand, we propose to employ supervised contrastive loss to jointly train the deep prediction model with cross-entropy loss to alleviate the problem of intra-class variation and inter-class similarity of HCC. On the other hand, we incorporate the clinical data to further improve the prediction ability of the model. Experiments are extensively conducted to verify the effectiveness of our proposed deep prediction model and the contribution of liver tissue for prognosis assessment of HCC.