The energy exchange between orthogonal polarization components is crucial for the build-up of vector solitons (VSs). Unlike previous observations of energy exchange in the frequency domain, our experiments analyzed pulse energy flows in the time domain. We provide evidence to demonstrate the influence of the four-wave mixing (FWM) and cross-phase modulation (XPM) effect on VSs build-up in passive mode-locked fiber lasers through a perspective of pulse energy exchange. The results indicate that the energy exchange of PRVS caused by FWM and XPM is stronger than that of PLVS. The liner energy exchange caused by the birefringence of fiber and PC influences the period of energy exchange. After stabilization, the intra-cavity energy evolution period is one roundtrip for PLVS while serval roundtrips for PRVS. In the future, we can achieve PLVS by adjusting the linear energy exchange through cavity birefringence, thereby meeting the industrial demand for stable and uniform pulse trains.