In this paper, we investigated the demulsification behavior of oil-in-water (O/W) emulsions during membrane permeation in the oil-water separation process using a numerical simulation approach. To accurately deal with the large deformation of the oil-water interface by coalescence and wetting, and to estimate the volume of the coalesced oil droplet, the coupled level set and volume-of-fluid method was used as the interface capturing method. We applied the simulation model to the permeation of O/W emulsions through a membrane pore, and then investigated the effects of the wettability of the membrane surface, filtration flux, and pore size on the demulsification efficiency. The results showed that oil droplets were likely to coalesce on the outlet membrane surface. High wettability on the membrane surface and low fluid velocity inside the pore increased the demulsification efficiency. This is the first work to numerically simulate the demulsification behavior of emulsions through membranes in the oil-water separation process.