Graphene quantum dots (GQDs) have been suggested to have a wide range of applications due to their unique electronic and optical properties. Moreover, heteroatom doping has become a viable way to fine-tune the properties of GQDs. However, the working principle of the doping strategy is still not conclusive. In this study, the effects of size, configuration of the nitrogen dopant, and N/C ratio on the electronic and optical properties of GQDs have been carefully examined. First, the variation of the adsorption wavelength of pristine GQDs was evaluated for which a linear relation is established against different diameters. Moreover, it is found that both the configuration and content of nitrogen dopants have a significant impact on the adsorption wavelength and band gap of GQDs. In particular, different nitrogen species could have exactly opposite effects on the adsorption behavior. The origin of the nitrogen doping effect is calibrated from orbital localization, charge analysis, natural transition orbitals, and atomic contribution towards excitation. It is noted that nitrogen doping can simultaneously reduce both light adsorption energy and emission energy compared with the pristine one. This study provides an insightful explanation for the electronic and optical properties of GQDs and consolidates the theory base of the doping strategy.