In the mid-infrared and terahertz (THz) regime, graphene supports tunable surface plasmon resonance (SPR) by controlling the chemical potential, which promotes light-matter interaction at the selected wavelength, showing exceptional promise for optoelectronic applications. In this article, we show that the electromagnetic (EM) response of graphene oligomers can be substantially modified by the modification of the local chemical potential, strengthening or reducing the intrinsic plasmonic modes. The effect mechanism is corroborated by a graphene nanocluster composed of 13 nanodisks with D6h symmetry; by transforming to D3h symmetry, the effect mechanism was retained and more available plasmonic resonance modes appeared. The intriguing properties open a new way to design nanodevices made of graphene oligomers with highly efficient photoresponse enhancement and tunable spectral selectivity for highly accurate photodetection.