Road traffic and related noise is a major source of annoyance and impairment to health in urban areas. Many areas exposed to road traffic noise are also exposed to rail and air traffic noise. The resulting annoyance may depend on individual/neighborhood socio-demographic factors. Nevertheless, few studies have taken into account the confounding or modifying factors in the relationship between transportation noise and annoyance due to road traffic. In this study, we address these issues by combining Geographic Information Systems and epidemiologic methods. Street network buffers with a radius of 500 m were defined around the place of residence of the 7290 participants of the RECORD Cohort in Ile-de-France. Estimated outdoor traffic noise levels (road, rail, and air separately) were assessed at each place of residence and in each of these buffers. Higher levels of exposure to noise were documented in low educated neighborhoods. Multilevel logistic regression models documented positive associations between road traffic noise and annoyance due to road traffic, after adjusting for individual/neighborhood socioeconomic conditions. There was no evidence that the association was of different magnitude when noise was measured at the place of residence or in the residential neighborhood. However, the strength of the association between neighborhood noise exposure and annoyance increased when considering a higher percentile in the distribution of noise in each neighborhood. Road traffic noise estimated at the place of residence and road traffic noise in the residential neighborhood (75th percentile) were independently associated with annoyance, when adjusted for each other. Interactions of effects indicated that the relationship between road traffic noise exposure in the residential neighborhood and annoyance was stronger in affluent and high educated neighborhoods. Overall, our findings suggest that it is useful to take into account (i) the exposure to transportation noise in the residential neighborhood rather than only at the residence, (ii) different percentiles of noise exposure in the residential neighborhood, and (iii) the socioeconomic characteristics of the residential neighborhood to explain variations in annoyance due to road traffic in the neighborhood.