Sex and gender matter in all aspects of life. Humans exhibit sexual dimorphism in anatomy, physiology, but also pathology. Many of the differences are due to sex chromosomes and, thus, genetics, other due to endocrine factors such as sex hormones, some are of social origin. Over the past decades, huge number of scientific studies have revealed striking sex differences of the human brain with remarkable behavioral and cognitive consequences. Prenatal and postnatal testosterone influence brain structures and functions, respectively. Cognitive sex differences include especially certain spatial and language tasks, but they also affect many other aspects of the neurotypical brain. Sex differences of the brain are also relevant for the pathogenesis of neuropsychiatric disorders such as autism spectrum disorders, which are much more prevalent in the male population. Structural dimorphism in the human brain was well-described, but recent controversies now question its importance. On the other hand, solid evidence exists regarding gender differences in several brain functions. This review tries to summarize the current understanding of the complexity of the effects of testosterone on brain with special focus on their role in the known sex differences in healthy individuals and people in the autism spectrum.