Storywrangler: A massive exploratorium for sociolinguistic, cultural, socioeconomic, and political timelines using Twitter

Thayer Alshaabi; Jane L Adams; Michael V Arnold; Joshua R Minot; David R Dewhurst; Andrew J Reagan; Christopher M Danforth; Peter Sheridan Dodds

doi:10.1126/sciadv.abe6534

Storywrangler: A massive exploratorium for sociolinguistic, cultural, socioeconomic, and political timelines using Twitter

Sci Adv. 2021 Jul 16;7(29):eabe6534. doi: 10.1126/sciadv.abe6534. Print 2021 Jul.

Authors

Thayer Alshaabi^{1

2

3}, Jane L Adams^{4

2}, Michael V Arnold^{4

2}, Joshua R Minot^{4

2}, David R Dewhurst^{4

2

5}, Andrew J Reagan⁶, Christopher M Danforth^{4

2

3}, Peter Sheridan Dodds^{1

2

3}

Affiliations

¹ Vermont Complex Systems Center, University of Vermont, Burlington, VT 05405, USA. thayer.alshaabi@uvm.edu peter.dodds@uvm.edu.
² Computational Story Lab, University of Vermont, Burlington, VT 05405, USA.
³ Department of Computer Science, University of Vermont, Burlington, VT 05405, USA.
⁴ Vermont Complex Systems Center, University of Vermont, Burlington, VT 05405, USA.
⁵ Charles River Analytics, Cambridge, MA 02138, USA.
⁶ MassMutual Data Science, Boston, MA 02110, USA.

Abstract

In real time, Twitter strongly imprints world events, popular culture, and the day-to-day, recording an ever-growing compendium of language change. Vitally, and absent from many standard corpora such as books and news archives, Twitter also encodes popularity and spreading through retweets. Here, we describe Storywrangler, an ongoing curation of over 100 billion tweets containing 1 trillion 1-grams from 2008 to 2021. For each day, we break tweets into 1-, 2-, and 3-grams across 100+ languages, generating frequencies for words, hashtags, handles, numerals, symbols, and emojis. We make the dataset available through an interactive time series viewer and as downloadable time series and daily distributions. Although Storywrangler leverages Twitter data, our method of tracking dynamic changes in n-grams can be extended to any temporally evolving corpus. Illustrating the instrument's potential, we present example use cases including social amplification, the sociotechnical dynamics of famous individuals, box office success, and social unrest.

Copyright © 2021 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of Science. No claim to original U.S. Government Works. Distributed under a Creative Commons Attribution License 4.0 (CC BY).