Charge noise is critical in the performance of gate-controlled quantum dots (QDs). Such information is not yet available for QDs made out of the new material graphene, where both substrate and edge states are known to have important effects. Here we show the 1/f noise for a microscopic graphene QD is substantially larger than that for a macroscopic graphene field-effect transistor (FET), increasing linearly with temperature. To understand its origin, we suspended the graphene QD above the substrate. In contrast to large area graphene FETs, we find that a suspended graphene QD has an almost-identical noise level as an unsuspended one. Tracking noise levels around the Coulomb blockade peak as a function of gate voltage yields potential fluctuations of order 1 μeV, almost one order larger than in GaAs/GaAlAs QDs. Edge states and surface impurities rather than substrate-induced disorders, appear to dominate the 1/f noise, thus affecting the coherency of graphene nano-devices.