Pascal Challenge

Tasks

The corpus will be constituted of 600 annotated and 500 unannotated documents. In order to provide a realistic test scenario the 200 most recent annotated CFP will form the test set. The remaining 400 annotated documents will be divided into four, 100 document, partitions enabling comparative cross-validation experiments to be performed.

The three tasks described below will be evaluated. Each participant can decide to participate in any of tasks, but participation in task 1 is mandatory. Participants will be asked to use the preprocessed form of the corpus, but an optional task (evaluated separately) will also allow algorithms to use a different pre-processor.

TASK1: Full scenario: Learning to annotate implicit relations.

Given 400 annotated documents, learn to extract information. Each algorithm provides results of a four-fold cross-validation experiment using the same document partitions for pre-competitive tests. The main goal of this task is to evaluate the ability of the system to learn how to extract information given a closed world (200 most recent annotated documents). The task will measure the ability to generalize over a limited amount of training material in an environment with a large amount of sparse data.

TASK2: Active learning: Learning to select documents

In this task, the same corpus of 600 documents mentioned above will be used; 400 as training documents and 200 as test documents. Baseline: given fixed subsets of the training corpus of increasing size (e.g. 10, 20, 30, 50, 75, 100, 150, 200), show the learning ability on the full test corpus. Advanced: given an initial number of annotated documents as a seed (e.g. 10), select training subsets of increasing size (e.g. 20, 30, 50, 75, 100, 150, 200) in order to show the algorithm's ability to select the most suitable set of training documents from an unannotated pool.

Each algorithm's results will be plotted on a chart in order to study its learning curve and to allow better understanding the results obtained in TASK1. Moreover, the ability to quickly reach reliable results is an important feature of any adaptive IE system supporting annotation (Ciravegna et al. 2002), so the study of the learning curve will allow to access the suitability of the algorithm for online learning.

TASK3: Enriched Scenario

Same as the full scenario, but the algorithms will be able to use a richer set of information sources. In particular, we will focus on using the unannotated part of the corpus (500 documents). Goal: study how unsupervised or semi-supervised methods can improve the results of supervised approaches. An interesting variant of this task could concern the use of unlimited resources, e.g. the Web