Due to global energy crisis and severe environmental pollution, the photovoltaic (PV) system has become one of the most important renewable energy sources. Many previous studies on solar charger integrated system only focus on load charge control or switching Maximum Power Point Tracking (MPPT) and charge control modes. This study used two-stage system, which allows the overall portable solar energy charging system to implement MPPT and optimal charge control of Li-ion battery simultaneously. First, this study designs a DC/DC boost converter of solar power generation, which uses variable step size incremental conductance method (VSINC) to enable the solar cell to track the maximum power point at any time. The voltage was exported from the DC/DC boost converter to the DC/DC buck converter, so that the voltage dropped to proper voltage for charging the battery. The charging system uses constant current/constant voltage (CC/CV) method to charge the lithium battery. In order to obtain the optimum PI charge controller parameters, this study used intelligent algorithm to determine the optimum parameters. According to the simulation and experimental results, the control parameters resulted from PSO have better performance than genetic algorithms (GAs).