Diabetic Retinopathy (DR), the complication leading to vision loss, is generally graded according to the amalgamation of various structural factors in fundus photography such as number of microaneurysms, hemorrhages, vascular abnormalities, etc. To this end, Convolution Neural Network (CNN) with impressively representational power has been exhaustively utilized to address this problem. However, while existing multi-stream networks are costly, the conventional CNNs do not consider multiple levels of semantic context, which suffers from the loss of spatial correlations between the aforementioned DR-related signs. Therefore, this paper proposes a Densely Reversed Attention based CNN (DRAN) to leverage the learnable integration of channel-wise attention at multi-level features in a pretrained network for unambiguously involving spatial representations of important DR-oriented factors. Consequently, the proposed approach gains a quadratic weighted kappa of 85.6% on Kaggle DR detection dataset, which is competitive with the state-of-the-arts.