(1) Overview

Context

Although Twitter could be understood as a massive corpus of texts in which many distant reading methodologies are deployed, interest in it as a resource for Digital Humanities (DH) projects has not been widespread. Its most popular use to date has been related to the movement “Twitter for scholarly networking”, in which digital humanists analysed how the DH community grew and established their particular networks (; ; ). However, we can highlight relevant DH studies that relate Twitter data to Natural Language Processing (NLP) (; ; ) and the DHNow initiative that has been offering open tools and resources to work with Twitter. Moreover, we note other responses closely related to DH that performed qualitative and quantitative analysis on Twitter (; ).

Digital Narratives of COVID-19 (DHCovid) is a digital humanities project funded by the University of Miami (FL) and developed in collaboration with the National Scientific and Technical Research Council (CONICET, Argentina) that investigates the sociolinguistic and geographical trends and topics in Twitter conversations surrounding the COVID-19 pandemic. Since April 2020, this project has been collecting COVID-19 related tweets in Spanish as well as tweets in English and Spanish in specific geographic locations: Argentina, Colombia, Ecuador, Mexico, Peru, Spain, and South Florida.

(2) Method

Steps

To assemble the Twitter corpus, a PHP programming language script mines the Twitter data streaming through Twitter’s Application Programming Interface (API) and recovers a series of specific tweet identifiers (IDs). Our data mining sampling strategy consists of four main variables: language, keywords, region, and date.

The corpus is available through three repositories:

  1. GitHub. Tweet IDs are stored in a MySQL relational database where they are “hydrated,” that is, all metadata associated with the tweets is recovered, including its body text. Then, an additional script organizes the tweet IDs in the database by day, language, and region, and creates a plaintext file for each combination with a list of corresponding tweet IDs. The script generates these files daily and organizes them into folders, where each directory represents one day. These are uploaded directly to our public GitHub repository (Table 1). The data collection began on April 24th, 2020, and new tweets are automatically uploaded daily, until May 2021.
  2. Project website endpoint. A free access endpoint for query and download of “hydrated” tweets can be accessed from DHCovid website. An additional script queries the database and recovers body text of tweets (see Quality Control section). The access to a tidied and structured Twitter corpus for on-demand querying is one of the most meaningful contributions of our project for data reuse and text mining activities.
  3. Zenodo. A first stable version of the dataset, published on May 13, 2020, was released through Zenodo as a compressed ZIP file containing folders of daily tweets made between April 24th, 2020 and May 12th, 2020. A second and final version will be uploaded by the end of the project in May 2021 with the complete collection of tweet IDs.

Table 1

Organization of plain text datasets in the GitHub repository.


YEAR-MONTH-DAYDAILY FOLDER

dhcovid_YEAR_MONTH_en_fl.txtTweets in English in Florida

dhcovid_YEAR_MONTH_es.txtAll Spanish tweets

dhcovid_YEAR_MONTH_es_ar.txtTweets in Spanish in Argentina

dhcovid_YEAR_MONTH_es_co.txtTweets in Spanish in Colombia

dhcovid_YEAR_MONTH_es_ec.txtTweets in Spanish in Ecuador

dhcovid_YEAR_MONTH_es_es.txtTweets in Spanish in Spain

dhcovid_YEAR_MONTH_es_fl.txtTweets in Spanish in Florida

dhcovid_YEAR_MONTH_es_mx.txtTweets in Spanish in Mexico

dhcovid_YEAR_MONTH_es_pe.txtTweets in Spanish in Peru

Sampling strategy

The recovery of tweets by language and keywords is straightforward: we only query tweets written in Spanish and English that contain one of the words from our user-defined keywords list. We delimited two lists of keywords in English and Spanish related to the COVID-19 pandemic. Consequently, only tweets with one of these words and/or hashtags are selected. The English keywords only apply to Miami area and include terms such as “covid,” “coronavirus,” “pandemic,” “quarantine,” “#stayathome,” “outbreak,” “lockdown,” and “#socialdistancing.” The Spanish keywords include “covid,” “coronavirus,” “pandemia,” “cuarentena,” “confinamiento,” “#quedateencasa,” “desescalada,” and “#distanciamientosocial ().

Our strategy is also shaped by Twitter API policies. First, in its free version, the API did not offer the possibility for querying tweets older than seven days. Second, Twitter allows users to publish georeferenced tweets (with exact location) but retrieving geotagged tweets is complicated due to the absence of a facility for querying by geographic region. A pragmatic approach led us to define “country” as a circle surrounding the area of interest, e.g., “Mexico” is defined as latitude 21.295658, longitude -100.291341, and a radius of 450 miles. Indeed, political and national borders will not always follow our selection criteria, so our area-specific corpus can sometimes contain tweets from a neighbouring country, e.g., a query for Argentina is conflated with parts of Uruguay, and Colombia with parts of Ecuador.

Quality control

The “hydration” of the collected tweet IDs undergoes an additional data tidying process before any body text data is returned to the user. We apply a set of rules to the tweet body text: enforce that all words are in lowercase, remove accents, punctuations, mention of users (@users) to protect privacy, and replace all links with a general “URL” term. While enforcing all text to be in UTF-8 encoding, a particular challenge unique to the Spanish corpus is accents and graphemes, such as the “ñ”, that can be difficult to process and preserve. Most of those cases were resolved through a script by detecting special entity codes and replacing them with the correct character (e.g. &ntilde as ñ). We have also transliterated emojis into its corresponding UTF-8 charset and eliminated them from our experiments as of now. This processing facilitates the application of NLP techniques.

(3) Dataset Description

Format names and versions

.txt

Creation dates

2020-04-24 to 2021-05-31

Dataset creators

Susanna Allés-Torrent: Conceptualization; Funding Acquisition; Project administration; Supervision; Writing

Gimena del Rio Riande: Conceptualization; Writing

Jerry Bonnell: Data curation; Software; Visualization

Dieyun Song: Data curation; Writing

Nidia Hernández: Data curation; Visualization; Writing

Language

Spanish, English (See Table 2).

Table 2

Number of tweets for each month and each region from April 2020 to March 2021. FLeng and FLspa correspond to tweets in English and in Spanish in the Greater Miami area. Spanish column represents all tweets in Spanish.


DATEFLSPAFLENGECUADORPERUCOLOMBIASPAINARGENTINAMEXICOSPANISHTOTAL

2020 Apr1.8k5.7k12.2k12.8k39k47.3k16k93.7k512.2k740.7k

2020 May6.1k22k48.4k56.2k168.1k182.7k68.5k411.1k2.4M3.4M

2020 Jun4.7k18.9k34.7k43.2k149.6k124.7k76.6k319.2k2.1M2.9M

2020 Jul6.5k28.6k34.7k41.4k171.2k127.3k79.8k324.5k2.1M2.9M

2020 Aug4.9k16.4k22.5k32k114.7k116.9k76.3k225.5k1.5M2.1M

2020 Sep4.3k15.6k18.9k25.5k86.8k137.4k86.3k184.5k1.5M2.1M

2020 Oct5.5k23.1k17.6k21.8k85.2k145.6k79.3k205.3k1.5M2.1M

2020 Nov4.6k18.8k18.7k18k74.4k134.8k66.9k188.7k1.3M1.7M

2020 Dec5.2k21.1k20.2k25.7k100.9k116.9k72.6k248.9k1.6M2.2M

2021 Jan5.4k16.4k27k42k125.9k155.5k78k304.4k1.9M2.7M

2021 Feb3.9k13k20.7k39.5k80k112.3k50.3k207.1k1.3M1.9M

2021 Mar4k14.6k21.6k30.8k74k90.3k67.2k170.5k1.3M1.8M

Total56.9k214.3k297.3k388.9k1.3M1.5M817.7k2.9M19.2M26.6M

License

Creative Commons license Attribution 4.0 International (CC BY 4.0).

Repository name

GitHub for the continuously daily updated version. Zenodo for the stable DOI.

Publication date

First published to the GitHub repository on 2020-04-24. Afterwards it was released to Zenodo on 2020-05-13.

Statistics and contents

In Table 2 and Figure 1 we offer the basic statistics of the dataset.

Figure 1 

Overview of number of tweets for each month and each region from April 2020 to March 2021. FLeng and FLspa correspond to tweets in English and in Spanish in the Greater Miami area.

(4) Reuse Potential

The COVID-19 pandemic has motivated a plethora of ambitious digital research focusing on Twitter, including the role and importance of automated Twitter accounts, also known as bots (), the increase of politically radical discourse in social media (), and the general public perception (). Most of the efforts, however, mine and analyze tweets written in English (; ; ). DHCovid bridges the gap by bringing Spanish Twitter narratives into the conversation.

We use a CC BY license to give full reuse of our dataset to the public as text or to apply combined technical and humanistic processing techniques, such as statistics, NLP, topic modelling, and sentiment analysis. Our datasets can be meaningful for teaching, research, and activism resources to be used across language, disciplinary, and professional boundaries. Scholars interested in the human experience of the pandemic, health professionals, policy makers, funding agencies, journalists, and active citizens are all welcome to engage with our project.