Previous studies suggested that genetic, environmental factors and their interactions could affect body fat mass (BFM). However, studies describing these effects were performed at a single time point in a population. In this study, we investigated the interaction between genetic and environmental factors in affecting BFM and implicate the healthcare utilization of lifestyle modifications from a personalized and genomic perspective. We examined how nutritional intake or physical activity changes in the individuals affect BFM concerning the genetic composition. We conducted an observational study including 259 adult participants with single nucleotide polymorphism (SNP) genotyping and longitudinal lifestyle monitoring, including food consumption and physical activities, by following lifestyle modification guidance. The participants' lifelog data on exercise and diet were collected through a wearable device for 3 months. Moreover, we measured anthropometric and serologic markers to monitor their potential changes through lifestyle modification. We examined the influence of genetic composition on body fat reduction induced by lifestyle changes using genetic risk scores (GRSs) of three phenotypes: GRS-carbohydrate (GRS-C), GRS-fat (GRS-F), and GRS-exercise (GRS-E). Our results showed that lifestyle modifications affected BFM more significantly in the high GRS class compared to the low GRS class, indicating the role of genetic factors affecting the efficiency of the lifestyle modification-induced BFM changes. Interestingly, the influence of exercise modification in the low GRS class with active lifestyle change was lower than that in the high GRS class with inactive lifestyle change (P = 0.022), suggesting the implication of genetic factors for efficient body fat control.