We investigate the charging process of open quantum battery in the weak system-environment coupling regime. A method to improve the performance of open quantum battery in a reservoir environment, which described by a band-gap environment model or a two-Lorentzian environment model, is proposed by manipulating the spectral density of environment. We find that the optimal quantum battery, characterized by fast charging time and large ergotropy, in the band-gap environment can be obtained by increasing the weights of two Lorentzians and the spectral width of the second Lorentzian, which is in sharp contrast to the quantum battery in two-Lorentzian environment. Then we extend our discussion to multiple coupled reservoir environments, which are composed of N coupled dissipative cavities. We show that, the performance of quantum battery can be enhanced by increasing the coupling strength between the nearest-neighbor environments and decreasing the size of the environments. In particular, to fully charge and extract the total stored energy as work for quantum battery can be achieved by manipulating the coupling strength between the nearest-neighbor environments. Our results provide a practical approach for the realization of the optimal quantum batteries in future experiments.