Optical memory based on the electromagnetically induced transparency (EIT) in a double-[Formula: see text] atomic system provides a convenient way to convert the frequency, bandwidth or polarization of an optical pulse by storing it in one [Formula: see text] channel and retrieving it from another. This memory-based optical converter can be used to bridge the quantum nodes which have different physical properties in a quantum network. However, in real atoms, each energy level usually contains degenerate Zeeman states, which may lead to additional energy loss, as has been discussed in our recent theoretical paper (Tsai et al. in Phys. Rev. A 100, 063843). Here, we present an experimental study on the efficiency variation in the EIT-memory-based optical polarization conversion in cold cesium atoms under Zeeman-state optical pumping. The experimental results support the theoretical predictions. Our study provides quantitative knowledge and physical insight useful for practical implementation of an EIT-memory-based optical converter.