Many pathogens continuously change their protein structure in response to immune-driven selection, resulting in weakened protection. In addition, for some pathogens such as dengue virus, poorly targeted immunity is associated with increased risk of severe disease, through a mechanism known as antibody-dependent enhancement. However, it remains a mystery whether the antigenic distance between an individual's first infection and subsequent exposures dictate disease risk, explaining the observed large-scale differences in dengue hospitalisations across years. Here we develop an inferential framework that combines detailed antigenic and genetic characterisation of viruses, and hospitalised cases from 21 years of surveillance in Bangkok, Thailand to identify the role of the antigenic profile of circulating viruses in determining disease risk. We find that the risk of hospitalisation depends on both the specific order of infecting serotypes and the antigenic distance between an individual's primary and secondary infections, with risk maximised at intermediate antigenic distances. These findings suggest immune imprinting helps determine dengue disease risk, and provides a pathway to monitor the changing risk profile of populations and to quantifying risk profiles of candidate vaccines.