Railway induced vibration is an important source of annoyance among residents living in the vicinity of railways. Annoyance increases with vibration magnitude. However, these correlations between the degree of annoyance and vibration exposure are weak. This suggests that railway vibration induced annoyance is governed by more than just vibration level and therefore other factors may provide information to understand the wide variation in annoyance reactions. Factors coming into play when considering an exposure-response relationship between level of railway vibration and annoyance are presented. The factors investigated were: attitudinal, situational and demographic factors. This was achieved using data from field studies comprised of face-to-face interviews and internal vibration measurements (N = 755). It was found that annoyance scores were strongly influenced by two attitudinal factors: Concern of property damage and expectations about future levels of vibration. Type of residential area and age of the respondent were found to have an important effect on annoyance whereas visibility of the railway and time spent at home showed a significant but small influence. These results indicate that future railway vibration policies and regulations focusing on community impact need to consider additional factors for an optimal assessment of railway effects on residential environments.