Physiological responses are essential for health monitoring. However, modeling the complex interactions be- tween them across activity and environmental factors can be challenging. In this paper, we introduce a framework that identifies the state of an individual based on their activity, trains predictive models for their physiological response within these states, and jointly optimizes for the states and the models. We apply this framework to respiratory rate prediction based on heart rate and physical activity, and test it on a dataset of 9 individuals performing various activities of daily life.