Article Structure
Abstract
The third PASCAL Recognizing Textual Entailment Challenge (RTE-3) contained an optional task that extended the main entailment task by requiring a system to make three-way entailment decisions (entails, contradicts, neither) and to justify its response.
Introduction
The PASCAL Recognizing Textual Entailment (RTE) workshop series (see www.pascal—network .
The Three-way Decision Task
The extended task used the RTE-3 main task test set of entailment pairs as its test set.
J ustifications
The second part of the extended task was for systems to provide explanations of how they reached their conclusions.
Conclusion
The RTE-3 extended task provided an opportunity to examine systems’ abilities to detect contradiction and to provide explanations of their reasoning
Topics
NIST
- The answer key for the three-way decision task was developed at the National Institute of Standards and Technology ( NIST ) using annotators who had experience as TREC and DUC assessors.Page 2, “The Three-way Decision Task”
- NIST assessors annotated all 800 entailment pairs in the test set, with each pair independently annotated by two different assessors.Page 2, “The Three-way Decision Task”
- The three-way answer key was formed by keeping exactly the same set of YES answers as in the two-way key (regardless of the NIST annotations) and having NIST staff adjudicate assessor differences on the remainder.Page 2, “The Three-way Decision Task”
- Main Task NIST Judge 1Page 3, “The Three-way Decision Task”
- Main Task NIST Judge 2 YES UN KN N 0Page 3, “The Three-way Decision Task”
- Recall that NIST judges annotated all 800 entailment pairs in the test set, with each pair independently annotated twice.Page 3, “The Three-way Decision Task”
- For each entailment pair, one of the NIST judges was arbitrarily assigned as the fi rst judge for that pair and the other as the second judge.Page 3, “The Three-way Decision Task”
- The agreement between NIST and RTE annotators is shown in Table 2.Page 3, “The Three-way Decision Task”
- The NIST judges’ answers are given in the columns and the two-way reference answers in the rows.Page 4, “The Three-way Decision Task”
- Agreement is then computed as the percentage of matches when a NIST judge’s NO or UNKNOWN annotation matched a NO two-way reference answer.Page 4, “The Three-way Decision Task”
- The 90% agreement rate between the NIST judges and the two-way answer key probably reflects a somewhat larger amount of disagreement since the test set already had RTE annotators’ disagreements removed.Page 4, “The Three-way Decision Task”
See all papers in Proc. ACL 2008 that mention NIST.
See all papers in Proc. ACL that mention NIST.
Back to top.
