In this paper, a novel, to the best of our knowledge, iterative approach for highlight removal is proposed using lenselet-based plenoptic cameras without multiple exposures. An unsupervised k-means clustering approach that relates unsaturated pixels to chromatic dispersion based on the intrinsic decomposition and dichromatic reflection model is proposed to recover unsaturated highlights. Meanwhile, an adaptive direction method along with a Gaussian probability distribution model is designed to recover the saturated highlights. Finally, a method that combines the specular residual ratio with information entropy is built to quantitatively evaluate the quality of highlight removal. Generally, our method not only fully removes specular highlights, but also has low spatial complexity of image acquisition, more stability, and outstanding restoration for complex scenes.