Intestinal epithelial cells (IECs) act as a physical barrier separating the commensal-containing intestinal tract from the sterile interior. These cells have found a complex balance allowing them to be prepared for pathogen attacks while still tolerating the presence of bacterial or viral stimuli present in the lumen of the gut. Using primary human IECs, we probed the mechanisms that allow for such a tolerance. We discovered that viral infections emanating from the basolateral side of IECs elicit a stronger intrinsic immune response in comparison to lumenal apical infections. We determined that this asymmetric immune response is driven by the clathrin-sorting adaptor AP-1B, which mediates the polarized sorting of Toll-like receptor 3 (TLR3) towards the basolateral side of IECs. Mice and human IECs lacking AP-1B showed an exacerbated immune response following apical stimulation. Together, these results suggest a model where the cellular polarity program plays an integral role in the ability of IECs to partially tolerate apical commensals while remaining fully responsive to invasive basolateral pathogens.