Toll-like receptors (TLRs) are a family of proteins with a key role in the innate immune system. They are specialized in the recognition of molecular patterns present in microbial components, through mechanisms not yet unraveled at atomic level. Improvement in the understanding of the molecular mechanisms that drive TLR signaling is of paramount importance to grasp key aspects of immunity, potentially leading to the design of new molecules able to modulate their functions. Toll-like receptor 4 (TLR4), along with its accessory protein myeloid differentiation factor 2 (MD-2), builds a heterodimeric complex that specifically recognizes lipopolysaccharides (LPS), which are present on the cell wall of gramnegative bacteria, activating the immune response. Some TLR4 modulators are undergoing preclinical and clinical evaluation for the treatment of sepsis, inflammatory diseases, cancer, and rheumatoid arthritis. Reported X-ray crystal structures together with molecular modeling studies, not reviewed before in the literature, have recently contributed to the elucidation of key interactions at atomic level of the binding between the TLR4/MD-2 system and different TLR4/MD-2 ligands. The purpose of this review is to summarize these reported studies which may account for the SAR rationalization of natural/ synthetic agonist/antagonist TLR4 binders and may also guide further design of novel TLR4 modulators.