The aim of this study was to examine the relationship between physical activity level and impaired left ventricular (LV) relaxation in a large sample of apparently healthy men and women. We conducted a cross-sectional study in 57,449 adults who underwent echocardiography as part of a comprehensive health examination between March 2011 and December 2014. Physical activity level was assessed using the Korean version of the International Physical Activity Questionnaire Short Form. The presence of impaired LV relaxation was determined based on echocardiographic findings. Physical activity levels were inversely associated with the prevalence of impaired LV relaxation. The multivariable-adjusted odds ratios (95% confidence interval) for impaired LV relaxation comparing minimally active and health-enhancing physically active groups to the inactive group were 0.84 (0.77-0.91) and 0.64 (0.58-0.72), respectively (P for trend < 0.001). These associations were modified by sex (p for interaction <0.001), with the inverse association observed in men, but not in women. This study demonstrated an inverse linear association between physical activity level and impaired LV relaxation in a large sample of middle-aged Koreans independent of potential confounders. Our findings suggest that increasing physical activity may be independently important in reducing the risk of impaired LV relaxation.