In recent years, gadolinium-based contrast agents have been associated with different types of toxicity. In particular, nephrogenic systemic fibrosis, a progressive sclerotic-myxedematous systemic disease of unknown etiology, is related to gadolinium-based contrast agent administration in patients with kidney dysfunction. More recently, evidence of magnetic resonance signal intensity changes on pre-contrast T1-weighted images after multiple gadolinium-based contrast agent administrations resulted in the hypothesis of gadolinium brain accumulation in patients with normal renal function, subsequently confirmed in pathological samples. However, there is limited current data and further investigations are necessary before drawing definite conclusions on the clinical consequences of gadolinium-based contrast agent accumulation in human tissues and particularly in the brain. Gadolinium-based contrast agent-related toxicity appears connected to molecular stability, which varies together with the pharmacokinetic properties of the compound and depends on the individual characteristics of the subject. During a lifetime, the physiological changes occurring in the human body may influence its interaction with gadolinium-based contrast agents: the integrity and developmental stage of the organs has an effect on the dynamics of gadolinium-based contrast agent distribution and excretion, thus leading to different possible mechanisms of deposition and toxicity. Therefore, the aim of this work is to discuss the pharmacokinetics and pharmacodynamics of gadolinium-based contrast agents, with a special focus on the brain, and to explore potential predominant gadolinium-based contrast agent-related toxicity in two cornerstone periods of the human life cycle: fetal/neonatal and adulthood/aged.