Cigarette smoking is associated with increased risk for myriad health consequences including cognitive decline and dementia, but research on the link between smoking and brain structure is nascent. In the current study, we assessed the relationship of cigarette smoking with gray matter (GM) and white matter (WM) in the UK Biobank, controlling for numerous confounding demographic and health variables. We used negative-binomial regression to model the association of cigarette smoking (having ever smoked regularly, cigarettes per day, and duration smoked) with GM and WM (GM N = 19,615; WM N = 17,760), adjusting for confounders. Ever smoked and duration were associated with smaller total GM volume. Ever smoked was associated with reduced volume of the right VIIIa cerebellum and elevated WM hyperintensity volume. Smoking duration was associated with reduced total WM volume. Regarding specific tracts, ever smoked was associated with reduced fractional anisotropy in the left cingulate gyrus part of the cingulum, left posterior thalamic radiation, and bilateral superior thalamic radiation, and increased mean diffusivity in the middle cerebellar peduncle, right medial lemniscus, bilateral posterior thalamic radiation, and bilateral superior thalamic radiation. This study identified significant associations of cigarette exposure with global measures of GM and WM, and select associations of ever smoked, but not cigarettes per day or duration, with specific GM and WM regions. By controlling for important sociodemographic and health confounders, such as alcohol use, this study identifies distinct associations between smoking and brain structure, highlighting potential mechanisms of risk for common neurological sequelae (e.g., dementia).