We present a speckle-based deep learning approach for orbital angular momentum (OAM) mode classification. In this method, we have simulated the speckle fields of the Laguerre-Gauss (LG), Hermite-Gauss (HG), and superposition modes by multiplying these modes with a random phase function and then taking the Fourier transform. The intensity images of these speckle fields are fed to a convolutional neural network (CNN) for training a classification model that classifies modes with an accuracy >99%. We have trained and tested our method against the influence of atmospheric turbulence by training the models with perturbed LG, HG, and superposition modes and found that models are still able to classify modes with an accuracy >98%. We have also trained and tested our model with experimental speckle images of LG modes generated by three different ground glasses. We have achieved a maximum accuracy of 96% for the most robust case, where the model is trained with all simulated and experimental data. The novelty of the technique is that one can do the mode classification just by using a small portion of the speckle fields because speckle grains contain the information about the original mode, thus eliminating the need for capturing the whole modal field, which is modal dependent.