Under environmental duress, many organisms accumulate large amounts of osmolytes - molecularly small organic solutes. Osmolytes are known to counteract stress, driving proteins to their compact native states by their exclusion from protein surfaces. In contrast, the effect of osmolytes on lipid membranes is poorly understood and widely debated. Many fully membrane-permeable osmolytes exert an apparent attractive force between lipid membranes, yet all proposed models fail to fully account for the origin of this force. We follow the quintessential osmolyte trimethylamine N-oxide (TMAO) and its interaction with dimyristoyl phosphatidylcholine (DMPC) membranes in aqueous solution. We find that by partitioning away from the inter-bilayer space, TMAO pushes adjacent membranes closer together. Experiments and simulations further show that the partitioning of TMAO away from the volume between bilayers stems from its exclusion from the lipid-water interface, similar to the mechanism of protein stabilization by osmolytes. We extend our analysis to show that the preferential interaction of other physiologically relevant solutes (including sugars and DMSO) also correlates with their effect on membrane bilayer interactions. Our study resolves a long-standing puzzle, explaining how osmolytes can increase membrane-membrane attraction or repulsion depending on their preferential interactions with lipids.