Accuracy of a pre-trained sentiment analysis (SA) classification model on tweets related to emergency response and early recovery assessment: the case of 2019 Albanian earthquake

Diana Contreras; Sean Wilkinson; Evangeline Alterman; Javier Hervás

doi:10.1007/s11069-022-05307-w

Accuracy of a pre-trained sentiment analysis (SA) classification model on tweets related to emergency response and early recovery assessment: the case of 2019 Albanian earthquake

Nat Hazards (Dordr). 2022;113(1):403-421. doi: 10.1007/s11069-022-05307-w. Epub 2022 Mar 23.

Authors

Diana Contreras^{1

2}, Sean Wilkinson², Evangeline Alterman³, Javier Hervás⁴

Affiliations

¹ Centre for Resilience and Environmental Change (CHANGING), School of Earth and Environmental Sciences, College of Physical Sciences and Engineering, Cardiff University, Main Building, Park Place, Cardiff, CF10 3AT UK.
² Learning from Earthquakes (LfE), School of Engineering, Faculty of Science, Agriculture and Engineering, Newcastle University, 2nd Floor Drummond Building, Newcastle, NE1 7RU UK.
³ Department Civil Engineering, Faculty Engineering, Auckland University, Private Bag 92019, Auckland, 1142 New Zealand.
⁴ 77 Landmark Place, Churchill Way, Cardiff, CF10 2HS UK.

Abstract

Traditionally, earthquake impact assessments have been made via fieldwork by non-governmental organisations (NGO's) sponsored data collection; however, this approach is time-consuming, expensive and often limited. Recently, social media (SM) has become a valuable tool for quickly collecting large amounts of first-hand data after a disaster and shows great potential for decision-making. Nevertheless, extracting meaningful information from SM is an ongoing area of research. This paper tests the accuracy of the pre-trained sentiment analysis (SA) model developed by the no-code machine learning platform MonkeyLearn using the text data related to the emergency response and early recovery phase of the three major earthquakes that struck Albania on the 26th November 2019. These events caused 51 deaths, 3000 injuries and extensive damage. We obtained 695 tweets with the hashtags: #Albania #AlbanianEarthquake, and #albanianearthquake from the 26th November 2019 to the 3rd February 2020. We used these data to test the accuracy of the pre-trained SA classification model developed by MonkeyLearn to identify polarity in text data. This test explores the feasibility to automate the classification process to extract meaningful information from text data from SM in real-time in the future. We tested the no-code machine learning platform's performance using a confusion matrix. We obtained an overall accuracy (ACC) of 63% and a misclassification rate of 37%. We conclude that the ACC of the unsupervised classification is sufficient for a preliminary assessment, but further research is needed to determine if the accuracy is improved by customising the training model of the machine learning platform.

Keywords: Accuracy (ACC); Earthquakes; Machine learning algorithm; Sentiment analysis (SA); Social media (SM); Twitter.