The testis is an immunological privileged tissue as evidenced by its ability to support grafts with minimal rejection. Immune privilege is essential for the tolerance of neo-antigens from developing germ cells that appear after the constitution of self-tolerance, but imposes the paradoxical task of also providing efficient protection against pathogens and tumor cells. It is becoming increasingly clear that immune privilege cannot be attributed to a single factor such as the sequestration of neo-antigens from the immune system behind the blood-testis barrier, but is based on a complex multifaceted interplay between cells and factors that are essential for the reproductive function of the testis and the testicular immune system. This review summarizes the evidence that has accumulated regarding the role of Sertoli cells, androgens, and selected population of leukocytes in the maintenance of immune privilege and its perturbation in testicular inflammatory sub- and infertility.