It is well known that neuropsychological impairment can be associated with chronic epilepsy. This review suggests that a broad lifespan perspective of cognition in epilepsy should include consideration of: a) neurobiological factors that antedate the first seizure and influence cognition, b) epilepsy-related factors that influence brain growth and cognitive development after epilepsy is diagnosed and treated, c) clinical epilepsy and other risk factors associated with poor cognitive prognosis in the context of chronic pharmacoresistant epilepsy, and d) the modifiable and non-modifiable risk factors that influence cognitive aging in the general population.