We introduce an approach to personalize energy expenditure (EE) estimates in free living. First, we use topic models to discover activity composites from recognized activity primitives and stay regions in daily living data. Subsequently, we determine activity composites that are relevant to contextualize heart rate (HR). Activity composites were ranked and analyzed to optimize the correlation to HR normalization parameters. Finally, individual-specific HR normalization parameters were used to normalize HR. Normalized HR was then included in activity-specific regression models to estimate EE. Our HR normalization minimizes the effect of individual fitness differences from entering in EE regression models. By estimating HR normalization parameters in free living, our approach avoids dedicated individual calibration or laboratory tests. In a combined free-living and laboratory study dataset, including 34 healthy volunteers, we show that HR normalization in 14-day free-living data improves accuracy compared to no normalization and normalization based on activity primitives only ( 29.4% and 19.8 % error reduction against lab reference). Based on acceleration and HR, both recorded from a necklace, and GPS acquired from a smartphone, EE estimation error was reduced by 10.7 % in a leave-one-participant-out analysis.