We investigate the dependence of decoherence on the mode number M in a multiple-mode Aharonov-Bohm (AB) interferometer. The design of the AB interferometer allows us to precisely determine M by the additivity rule of ballistic conductors; meanwhile, the decoherence rate is simultaneously deduced by the variance of the AB oscillation amplitude. The AB amplitude decreases and fluctuates with depopulating M. Moreover, the normalized amplitude exhibits a maximum at a specific M (∼9). Data analysis reveals that the charge-fluctuation-induced dephasing, which depends on the geometry and the charge relaxation resistance of the system, could play an essential role in the decoherence process. Our results suggest that the phase coherence, in principle, can be optimized using a deliberated design and pave one of the ways toward the engineering of quantum coherence.