Fat perception of food emulsions has been found to relate to in-mouth friction. Previously, we have shown that friction under mouth-like conditions strongly depends on the sensitivity of protein-stabilized emulsion droplets to coalescence. Here, we investigated whether this also implies that oral fat retention depends in a similar manner on the stability of the emulsion droplets against coalescence. We investigate the separate contributions of droplet adhesion and droplet spreading to fat retention at the tongue, as well as the role of saliva. We perform ex vivo (Confocal Raman Spectroscopy; Confocal Scanning Laser Microscopy) experiments using pig's tongue surfaces in combination with human in vivo experiments. These reveal that protein-poor (unstable) emulsions are retained more at the tongue than protein-rich (stable) emulsions. Furthermore, the layer formed by adhering protein-poor droplets is more stable against rinsing. Saliva is found to be very efficient in removing fat and emulsion droplets from the oral surface but its role in fat retention needs further research. We relate our results to the colloidal forces governing droplet adhesion and spreading.