Adapting to particular features of a haptic shape, for example, the slant of a surface, affects how a subsequently touched shape is perceived (aftereffect). Previous studies showed that this adaptation is largely based on our proprioceptive sense of hand posture, yet the influence of vision on haptic shape adaptation has been relatively unexplored. Here, using a slant-adaptation paradigm, we investigated whether visual information affects haptic adaptation and, if so, how. To this end, we varied the available visual cues during the adaptation period. This process ranged from providing visual information only about the slant of the surface, or the reference frame in which it is presented, to only providing visual information about the location of the fingertips. Additionally, we tested several combinations of these visual cues. We show that, as soon as the visual information can be used as a spatial reference to link the own fingertip position to the surface slant, haptic adaptation is very much reduced. This result means that, under these viewing conditions, vision dominates touch and is one reason why we do not easily adapt to haptic shape in our daily life, because we usually have visual information about both hand and object available simultaneously.