The circadian system affects physiological, psychological, and molecular mechanisms in the body, resulting in varying physical performance over the day. The timing and relative size of these effects are important for optimizing sport performance. In this study, Olympic swim times (from 2004 to 2016) were used to determine time-of-day and circadian effects under maximal motivational conditions. Data of athletes who made it to the finals (N = 144, 72 female) were included and normalized on individual levels based on the average swim times over race types (heat, semifinal, and final) per individual for each stroke, distance and Olympic venue. Normalized swim times were analyzed with a linear mixed model and a sine fitted model. Swim performance was better during finals as compared to semi-finals and heats. Performance was strongly affected by time-of-day, showing fastest swim times in the late afternoon around 17:12 h, indicating 0.32% improved performance relative to 08:00 h. This study reveals clear effects of time-of-day on physical performance in Olympic athletes. The time-of-day effect is large, and exceeds the time difference between gold and silver medal in 40%, silver and bronze medal in 64%, and bronze or no medal in 61% of the finals.