Infectious Diseases Task (ID)

The Infectious Diseases (ID) task is a main task in the BioNLP 2011 Shared Task on Event Extraction.

This tasks focuses on the biomolecular mechanisms of infectious diseases. The primary task broadly follows the task definition and event types of the BioNLP'09 Shared Task, extending it with new entity categories and a new class of events, high-level biological processes. The training and test data are drawn from the primary text content of full-text PMC open access documents selected by domain experts as representative publications on two-component regulatory systems, a prominent class of signaling system ubiquitous in bacteria.

The Infectious Diseases Task does not define explicit subtasks. However, it specifies minimal "core" extraction targets in addition to the "full" task targets. Results for this task will be reported separately for "core" targets and the "full" task and participants can choose to only extract "core" targets. Full task results are considered the primary evaluation for the task.

The ID task is completed. The task received final submissions from seven teams. Thank you for your participation!

The final results are summarized below.

There is also a tool for visualization and comparison of detailed results. [temporarily offline pending correction of CORE task result data in the tool.]

UPDATE (19 October 2011): an error affecting the CORE results was discovered in the evaluation software. Correction of the error resulted in revision of overall CORE task performance results by 0.13-0.49% points for five of the seven participants. Results found below reflect the corrected performance. For details and originally reported (uncorrected) results, please see the detailed page on the correction.

Primary ID task evaluation results

Results for FULL task, primary evaluation criteria

------------------------------------------------------------------------------------

Event Class gold (match) answer (match) recall prec. fscore

------------------------------------------------------------------------------------

FAUST ====[TOTAL]==== 1445 ( 694) 1049 ( 692) 48.03 65.97 55.59

UMass ====[TOTAL]==== 1445 ( 678) 1090 ( 676) 46.92 62.02 53.42

Stanford ====[TOTAL]==== 1445 ( 669) 1194 ( 667) 46.30 55.86 50.63

ConcordU ====[TOTAL]==== 1445 ( 708) 1753 ( 706) 49.00 40.27 44.21

UTurku ====[TOTAL]==== 1445 ( 547) 1121 ( 545) 37.85 48.62 42.57

PNNL ====[TOTAL]==== 1445 ( 401) 764 ( 400) 27.75 52.36 36.27

PredX ====[TOTAL]==== 1445 ( 326) 921 ( 324) 22.56 35.18 27.49

These are the primary evaluation results for the ID task.

We request that participants include these results when reporting on the performance and ranking of their system in the task. (You are free to report any other results also, but these results should be included and identified as primary.)

Additional ID task evaluation results

Results for CORE task, primary evaluation criteria

(PLEASE NOTE: these results were revised on 19 October 2011. See details)

------------------------------------------------------------------------------------

Event Class gold (match) answer (match) recall prec. fscore ------------------------------------------------------------------------------------ FAUST ====[TOTAL]==== 1369 ( 696) 1049 ( 696) 50.84 66.35 57.57 UMass ====[TOTAL]==== 1369 ( 680) 1090 ( 680) 49.67 62.39 55.31 Stanford ====[TOTAL]==== 1369 ( 673) 1194 ( 673) 49.16 56.37 52.52 ConcordU ====[TOTAL]==== 1369 ( 697) 1607 ( 697) 50.91 43.37 46.84 UTurku ====[TOTAL]==== 1369 ( 537) 1076 ( 537) 39.23 49.91 43.93 PNNL ====[TOTAL]==== 1369 ( 402) 764 ( 402) 29.36 52.62 37.69 PredX ====[TOTAL]==== 1369 ( 324) 921 ( 324) 23.67 35.18 28.30

These results are provided to give additional perspective into the performance of systems. Please see above for the primary results.

Task Definition

Entities

The task centers around five classes of entities. Human-created "gold standard" annotations for all of these core entity classes will be provided to participants for both training and test data. Named entity recognition is thus not necessary for participation in the task.

    • Genes and gene products: gene, RNA, and protein name mentions. Following the BioNLP'09 Shared Task convention, these entities are identified with the single "Protein" type.

    • Two-component systems: mentions of the names of two-component regulatory systems, frequently embedding the names of the two Proteins forming the system.

    • Chemicals: mentions of chemical compounds such as "NaCL".

    • Organisms: mentions of organism names or organism specification through specific properties (e.g. "graRS mutant").

    • Regulons/Operons: mentions of names of specific regulons and operons.

Additional arguments in the full task identify the modification sites of proteins and the locations in which core entities are found or move to. Following the BioNLP'09 Shared Task model, these are identified in the data generically as "Entity" annotations. These additional entities are only provided for training data, and systems addressing the full task will need to incorporate detection for these entities.

Events

The primary task targets the nine event types defined in the BioNLP'09 Shared Task and a further category, Process, used to annotate high-level biological processes.

The following table summarizes the events targeted in the task. In the table, "Core entity" refers to any of the five given entity types (Protein, Two-component system, Chemical, Organism, or Regulon/Operon).

The format "Arg(Type)" indicates that an event takes an argument "Arg" which should identify an entity of type "Type": for example, Localization takes one Theme of any of the four core entity types and optionally one additional Site of Entity type. The character "+" indicates that an argument may be repeated: Binding takes as many theme arguments as there are entities stated to bind.

The Process type is unique in that it does not specify mandatory arguments, reflecting that high-level processes such as "virulence" can be referred to in domain texts without explicitly stating the entities they concern. When stated, Process arguments are identified using the a generic "Participant" role.

Data Format

The data for the task will be provided in the BioNLP'09 Shared Task format. In brief, this is a standoff format where the original text of each document is provided in one file (.txt) and the annotations in two files, one (.a1) containing the given entities Protein, Two-component system, Chemical, Organism and Regulon/Operon, and the other the remaining annotations that systems should extract (.a2).

Acknowledgments

The ID task was jointly organized by the University of Tokyo Tsujii Laboratory (Sampo Pyysalo, Tomoko Ohta, Jun'ichi Tsujii), the National Centre for Text Mining (NaCTeM) (Rafal Rak, Sophia Ananiadou) and the Virginia Bioinformatics Institute at Virginia Tech (Dan Sullivan, Chunhong Mao, Chunxia Wang, Bruno Sobral). The annotation was created by Tomoko Ohta (coordinator), Dan Sullivan, Chunhong Mao and Chunxia Wang.