Survival prediction of esophageal cancer is an essential task for doctors to make personalized cancer treatment plans. However, handcrafted features from medical images need prior medical knowledge, which is usually limited and not complete, yielding unsatisfying survival predictions. To address these challenges, we propose a novel and efficient deep learning-based survival prediction framework for evaluating clinical outcomes before concurrent chemoradiotherapy. The proposed model consists of two key components: a 3D Coordinate Attention Convolutional Autoencoder (CACA) and an uncertainty-based jointly Optimizing Cox Model (UOCM). The CACA is built upon an autoencoder structure with 3D coordinate attention layers, capturing latent representations and encoding 3D spatial characteristics with precise positional information. Additionally, we designed an Uncertainty-based jointly Optimizing Cox Model, which jointly optimizes the CACA and survival prediction task. The survival prediction task models the interactions between a patient's feature signatures and clinical outcome to predict a reliable hazard ratio of patients. To verify the effectiveness of our model, we conducted extensive experiments on a dataset including computed tomography of 285 patients with esophageal cancer. Experimental results demonstrated that the proposed method achieved a C-index of 0.72, outperforming the state-of-the-art method.