Urban safety plays an essential role in the quality of citizens' lives and in the sustainable development of cities. In recent years, researchers have attempted to apply machine learning techniques to identify the role of location-specific attributes in the development of urban safety. However, existing studies have mainly relied on limited images (e.g., map images, single- or four-directional images) of areas based on a relatively large geographical unit and have narrowly focused on severe crime rates, which limits their predictive performance and implications for urban safety. In this work, we propose a novel method that predicts "deviance," which includes formal deviant crimes (e.g., murders) and informal deviant behaviors (e.g., loud parties at night). To do this, we first collect a large-scale geo-tagged dataset consisting of incident report data for seven metropolitan cities, along with corresponding sequential images around incident sites obtained from Google Street View. We then design a convolutional neural network that learns spatio-temporal visual attributes of deviant streets. Experimental results show that our framework is able to reliably recognize real-world deviance in various cities. Furthermore, we analyze which visual attribute is important for deviance identification and severity estimation with respect to social science as well as activated feature maps in the neural network. We have released our dataset and source codes on https://github.com/JinhwiPark/DevianceNet/.