We report on the contact line dynamics of a triple-phase system silica/oil/water. When oil advances onto silica within a water film squeezed between oil and silica, a rim forms in water and recedes at constant velocity. We evidence a sharp (three orders of magnitude) decrease of the contact line velocity upon the addition of cationic surfactants above a threshold concentration, which is slightly smaller than the critical micellar concentration. We show that, with or without surfactant, and within the range of small capillary numbers investigated, the contact line dynamics can be described by a friction term that does not reduce to pure hydrodynamical effects. In addition, we derive a model that successfully accounts for the selected contact line velocity of the rim. We further demonstrate the strong increase of the friction coefficient with surfactant bulk concentration results from the strongly nonlinear adsorption isotherm of surfactants on silica. From the variations of the friction coefficient and spreading parameter with surface concentration, we suggest a picture in which the part of the adsorbed surfactants that are strongly bound to the silica interface is trapped under the oil droplet and is responsible for the large increase in line friction.