Detection of patients with esophageal squamous cell carcinoma (ESCC) who do not benefit from standard chemoradiation (CRT) is an important medical need. Radiomics using 18-fluorodeoxyglucose (FDG) positron emission tomography (PET) is a promising approach. In this retrospective study of 184 patients with locally advanced ESCC. 152 patients from one center were grouped into a training cohort (n = 100) and an internal validation cohort (n = 52). External validation was performed with 32 patients treated at a second center. Primary endpoint was disease-free survival (DFS), secondary endpoints were overall survival (OS) and local control (LC). FDG-PET radiomics features were selected by Lasso-Cox regression analyses and a separate radiomics signature was calculated for each endpoint. In the training cohort radiomics signatures containing up to four PET derived features were able to identify non-responders in regard of all endpoints (DFS p < 0.001, LC p = 0.003, OS p = 0.001). After successful internal validation of the cutoff values generated by the training cohort for DFS (p = 0.025) and OS (p = 0.002), external validation using these cutoffs was successful for DFS (p = 0.002) but not for the other investigated endpoints. These results suggest that pre-treatment FDG-PET features may be useful to detect patients who do not respond to CRT and could benefit from alternative treatment.