HIV cellular entry is a multistep process that requires the interaction of a viral envelope glycoprotein (gp120) and a host receptor (CD4) followed by binding to a co-receptor. The CC-chemokine receptor 5 (CCR5) and CXC-chemokine receptor 4 (CXCR4) have been identified as the major HIV co-receptors and therefore are promising targets for the development of new anti-HIV drugs. CXCR4 is also involved in several diseases such as angiogenesis, metabolic and neurological disorders, rheumatoid arthritis and in different forms of metastatic cancer. Herein, we present a review focusing on small molecule CXCR4 antagonists. These compounds are divided into 11 classes that include cyclic penta- and tetrapeptides, diketopiperazine mimetics, bicyclams, non-bicyclams, tetrahydroquinolines, thiazolylisothiourea derivatives, benzodiazepines, alkyl amine analogs and non-peptides derivatives, dipicolylamine-zinc(II) complexes, ampelopsin and distamycin analogs. The most advanced CXCR4 antagonists documented are bicyclam derivatives, which are specific CXCR4 antagonists and exhibit potency in the nanomolar range. Further development of selective CXCR4 antagonists continues to be crucial for the design of second generation of anti-HIV drugs. We aim to provide a comprehensive summary of diverse structural templates that could be useful for optimization and discovery of novel CXCR4 antagonists.