This review aims to summarize what is currently known about neurocognitive outcome and quality of life in patients with brain tumors treated with radiotherapy. Whether potential tumor-controlling benefits of radiotherapy outweigh its potential toxicity in the natural history of brain tumors is a matter of debate. This review focuses on some of the adult main brain tumors, for which the issue of neurocognitive decline has been thoroughly studied: low-grade gliomas, brain metastases, and primary central nervous system lymphomas. The aims of this review are: (1) the analysis of existing data regarding the relationship between radiotherapy and neurocognitive outcome; (2) the identification of strategies to minimize radiotherapy-related neurotoxicity by reducing the dose or the volume; (3) the evidence-based data concerning radiotherapy withdrawal; and (4) the definition of patients subgroups that could benefit from immediate radiotherapy. For high grade gliomas, the main findings from literature are summarized and some strategies to reduce the neurotoxicity of the treatment are presented. Although further prospective studies with adequate neuropsychological follow-up are needed, this article suggests that cognitive deficits in patients with brain tumor have a multifactorial genesis: radiotherapy may contribute to the neurocognitive deterioration, but the causes of this decline include the tumor itself, disease progression, other treatment modalities and comorbidities. Treatment variables, such as total and fractional dose, target volume, and irradiation technique can dramatically affect the safety of radiotherapy: optimizing radiation parameters could be an excellent approach to improve outcome and to reduce neurotoxicity. At the same time, delayed radiotherapy could be a valid option for highly selected patients.