We search for nuclear recoil signals of dark matter models with a light mediator in PandaX-II, a direct detection experiment in the China Jinping underground laboratory. Using data collected in 2016 and 2017 runs, corresponding to a total exposure of 54 ton day, we set upper limits on the zero-momentum dark matter-nucleon cross section. These limits have a strong dependence on the mediator mass when it is comparable to or below the typical momentum transfer. We apply our results to constrain self-interacting dark matter models with a light mediator mixing with standard model particles, and set strong limits on the model parameter space for the dark matter mass ranging from 5 GeV to 10 TeV.