The gastric cancer grading of patients determines their clinical treatment plan. We use hyperspectral imaging (HSI) gastric cancer section data to automatically classify the three different cancer grades (low grade, intermediate grade, and high grade) and healthy tissue. This paper proposed the use of HSI data combined with a shallow residual network (SR-Net) as the classifier. We collected hyperspectral data from gastric sections of 30 participants, with the wavelength range of hyperspectral data being 374 nm to 990 nm. We compared the classification results between hyperspectral data and color images. The results show that using hyperspectral data and a SR-Net an average classification accuracy of 91.44% could be achieved, which is 13.87% higher than that of the color image. In addition, we applied a modified SR-Net incorporated direct down-sampling, asymmetric filters, and global average pooling to reduce the parameters and floating-point operations. Compared with the regular residual network with the same number of blocks, the floating-point operations of a SR-Net are one order of magnitude less. The experimental results show that hyperspectral data with a SR-Net can achieve cutting-edge performance with minimum computational cost and therefore have potential in the study of gastric cancer grading.