Diabetic retinopathy (DR) and especially diabetic macular edema (DME) are common causes of vision loss as complications of diabetes. In this work, we consider an ensemble that organizes a convolutional neural network (CNN) and traditional hand-crafted features into a single architecture for retinal image classification. This approach allows the joint training of a CNN and the fine-tuning of the weights of handcrafted features to provide a final prediction. Our solution is dedicated to the automatic classification of fundus images according to the severity level of DR and DME. For an objective evaluation of our approach, we have tested its performance on the official test datasets of the IEEE International Symposium on Biomedical Imaging (ISBI) 2018 Challenge 2: Diabetic Retinopathy Segmentation and Grading Challenge, section B. Disease Grading: Classification of fundus images according to the severity level of diabetic retinopathy and diabetic macular edema. As for our experimental results based on testing on the Indian Diabetic Retinopathy Image Dataset (IDRiD), the classification accuracies have been found to be 90.07% for the 5-class DR challenge, and 96.85% for the 3-class DME one.