Retrograde transport links early/recycling endosomes to the trans-Golgi network (TGN), thereby connecting the endocytic and the biosynthetic/secretory pathways. To determine how internalized molecules are targeted to the retrograde route, we have interfered with the function of clathrin and that of two proteins that interact with it, AP1 and epsinR. We found that the glycosphingolipid binding bacterial Shiga toxin entered cells efficiently when clathrin expression was inhibited. However, retrograde transport of Shiga toxin to the TGN was strongly inhibited. This allowed us to show that for Shiga toxin, retrograde sorting on early/recycling endosomes depends on clathrin and epsinR, but not AP1. EpsinR was also involved in retrograde transport of two endogenous proteins, TGN38/46 and mannose 6-phosphate receptor. In conclusion, our work reveals the existence of clathrin-independent and -dependent transport steps in the retrograde route, and establishes a function for clathrin and epsinR at the endosome-TGN interface.