Event-Dataset: Temporal information retrieval and text classification dataset

Abstract

Recently, Temporal Information Retrieval (TIR) has grabbed the major attention of the information retrieval community. TIR exploits the temporal dynamics in the information retrieval process and harnesses both textual relevance and temporal relevance to fulfill the temporal information requirements of a user Ur Rehman Khan et al., 2018. The focus time of document is an important temporal aspect which is defined as the time to which the content of the document refers Jatowt et al., 2015; Jatowt et al., 2013; Morbidoni et al., 2018, Khan et al., 2018. To the best of our knowledge, there does not exist any standard benchmark data set (publicly available) that holds the potential to comprehensively evaluate the performance of focus time assessment strategies. Considering these aspects, we have produced the Event-dataset, which is comprised of 35 queries and set of news articles for each query. Such that, $C=\left\{Q_s,D_s\right\},$ where C represents the dataset, $Q_s$ is query set $Q_s=\left\{q_1,q_2,q_3,\dots \dots .,q_{35}\right\}$ and for each $q_i$ there is a set of news articles $q_i=\left\{d_r,d_{nr}\right\}$ . $d_r,d_{nr}$ are sets of relevant documents and non-relevant documents respectively. Each query in the dataset represents a popular event. To annotate these articles into relevant and non-relevant, we have employed a user-study based evaluation method wherein a group of postgraduate students manually annotate the articles into the aforementioned categories. We believe that the generation of such dataset can provide an opportunity for the information retrieval researchers to use it as a benchmark to evaluate focus time assessment methods specifically and information retrieval methods generically.

Keywords: Focus time assessment; Information retrieval; Temporal; Text classification.