Climate drives population dynamics through multiple mechanisms, which can lead to seemingly context-dependent effects of climate on natural populations. For climate-sensitive diseases, such as dengue, chikungunya, and Zika, climate appears to have opposing effects in different contexts. Here we show that a model, parameterized with laboratory measured climate-driven mosquito physiology, captures three key epidemic characteristics across ecologically and culturally distinct settings in Ecuador and Kenya: the number, timing, and duration of outbreaks. The model generates a range of disease dynamics consistent with observed Aedes aegypti abundances and laboratory-confirmed arboviral incidence with variable accuracy (28-85% for vectors, 44-88% for incidence). The model predicted vector dynamics better in sites with a smaller proportion of young children in the population, lower mean temperature, and homes with piped water and made of cement. Models with limited calibration that robustly capture climate-virus relationships can help guide intervention efforts and climate change disease projections.