Who, What, When, Where, Why? Comparing Multiple Approaches to the Cross-Lingual 5W Task
Parton, Kristen and McKeown, Kathleen R. and Coyne, Bob and Diab, Mona T. and Grishman, Ralph and Hakkani-Tür, Dilek and Harper, Mary and Ji, Heng and Ma, Wei Yun and Meyers, Adam and Stolbach, Sara and Sun, Ang and Tur, Gokhan and Xu, Wei and Yaman, Sibel

Article Structure

Abstract

Cross-lingual tasks are especially difficult due to the compounding effect of errors in language processing and errors in machine translation (MT).

Introduction

In our increasingly global world, it is ever more likely for a monolingual speaker to require information that is only available in a foreign language document.

Prior Work

The cross-lingual 5W task is closely related to cross-lingual information retrieval and cross-lingual question answering (Wang and Card 2006; Mitamura et al.

The Chinese-English 5W Task

3.1 5W Task Description

SW System

We developed a 5W combination system that was based on five other 5W systems.

Methods 5.1 5W Systems

For the purposes of this evaluation2, we compared the output of 4 systems: English-Function, English-LF, Chinese-align, and the combined system.

Results

Figure 1 shows the cross-lingual performance (on MT) of all the systems for each SW.

Conclusions

In our evaluation of various 5W systems, we discovered several characteristics of the task.

Topics

cross-lingual

Appears in 19 sentences as: Cross-Lingual (2) Cross-lingual (2) cross-lingual (17)
In Who, What, When, Where, Why? Comparing Multiple Approaches to the Cross-Lingual 5W Task
  1. Cross-lingual tasks are especially difficult due to the compounding effect of errors in language processing and errors in machine translation (MT).
    Page 1, “Abstract”
  2. In this paper, we present an error analysis of a new cross-lingual task: the SW task, a sentence-level understanding task which seeks to return the English 5W's (Who, What, When, Where and Why) corresponding to a Chinese sentence.
    Page 1, “Abstract”
  3. The best cross-lingual 5W system was still 19% worse than the best monolingual 5W system, which shows that MT significantly degrades sentence-level understanding.
    Page 1, “Abstract”
  4. Cross-lingual applications address this need by presenting information in the speaker’s language even when it originally appeared in some other language, using machine
    Page 1, “Introduction”
  5. In this paper, we present an evaluation and error analysis of a cross-lingual application that we developed for a government-sponsored evaluation, the 5 W task.
    Page 1, “Introduction”
  6. In this paper, we address the cross-lingual 5 W task: given a source-language sentence, return the 5W’s translated (comprehensibly) into the target language.
    Page 1, “Introduction”
  7. 0 How much does machine translation (MT) degrade the performance of cross-lingual 5W systems, as compared to monolingual performance?
    Page 1, “Introduction”
  8. In this evaluation, we compare several different approaches to the cross-lingual 5W task, two that work on the target language (English) and one that works in the source language (Chinese).
    Page 1, “Introduction”
  9. A central question for many cross-lingual applications is whether to process in the source language and then translate the result, or translate documents first and then process the translation.
    Page 2, “Introduction”
  10. The cross-lingual 5W task is closely related to cross-lingual information retrieval and cross-lingual question answering (Wang and Card 2006; Mitamura et al.
    Page 2, “Prior Work”
  11. In cross-lingual information extraction (Sudo et al.
    Page 2, “Prior Work”

See all papers in Proc. ACL 2009 that mention cross-lingual.

See all papers in Proc. ACL that mention cross-lingual.

Back to top.

MT systems

Appears in 9 sentences as: MT system (3) MT Systems (1) MT systems (5)
In Who, What, When, Where, Why? Comparing Multiple Approaches to the Cross-Lingual 5W Task
  1. In this section, we describe the individual systems that we evaluated, the combination strategy, the parsers that we tuned for the task, and the MT systems .
    Page 3, “SW System”
  2. Finally, Chinese-align used the alignments of three separate MT systems to translate the 5Ws: a phrase-based system, a hierarchical phrase-based system, and a syntax augmented hierarchical phrase-based system.
    Page 4, “SW System”
  3. Since the predicate is essential, it tried to detect when verbs were deleted in MT, and back-off to a different MT system .
    Page 4, “SW System”
  4. 4.4 MT Systems
    Page 4, “SW System”
  5. The MT Combination system used by both of the English 5W systems combined up to nine separate MT systems .
    Page 4, “SW System”
  6. All three annotators were native English speakers who were not system developers for any of the SW systems that were being evaluated (to avoid biased grading, or assigning more blame to the MT system ).
    Page 5, “Methods 5.1 5W Systems”
  7. If the SW system picked an incorrectly translated argument (e. g., “baked a moon” instead of “baked a cake”), then the error would be assigned to the MT system .
    Page 5, “Methods 5.1 5W Systems”
  8. Long-distance phrase movement is a common problem in Chinese-English MT, and many MT systems try to handle it (e. g., Wang et al.
    Page 8, “Results”
  9. Since MT systems are tuned for word-based overlap measures (such as BLEU), verb deletion is penalized equally as, for example, determiner deletion.
    Page 8, “Results”

See all papers in Proc. ACL 2009 that mention MT systems.

See all papers in Proc. ACL that mention MT systems.

Back to top.

word order

Appears in 6 sentences as: Word order (2) word order (4)
In Who, What, When, Where, Why? Comparing Multiple Approaches to the Cross-Lingual 5W Task
  1. Word order mixed up.
    Page 7, “Results”
  2. Garbled word order was chosen for 21-24% of the target-language system Who/W hat errors, but only 9% of the source-language system Who/W hat errors.
    Page 8, “Results”
  3. The source-language word order problems tended to be local, within-phrase errors (e. g., “the dispute over frozen funds” was translated as “the freezing of disputes”).
    Page 8, “Results”
  4. The tar-get-language system word order problems were often long-distance problems.
    Page 8, “Results”
  5. Word order problems in MT also caused attachment ambiguity in When and Where.
    Page 8, “Results”
  6. For example, Chinese-align had fewer problems with word order , and most of those were due to local word-order problems.
    Page 8, “Conclusions”

See all papers in Proc. ACL 2009 that mention word order.

See all papers in Proc. ACL that mention word order.

Back to top.

BLEU

Appears in 4 sentences as: BLEU (3) Bleu (1)
In Who, What, When, Where, Why? Comparing Multiple Approaches to the Cross-Lingual 5W Task
  1. Unlike word- or phrase-overlap measures such as BLEU , the SW evaluation takes into account “concept” or “nugget” translation.
    Page 3, “The Chinese-English 5W Task”
  2. model score and word penalty for a combination of BLEU and TER (2*(1-BLEU) + TER).
    Page 5, “SW System”
  3. Bleu scores on the government supplied test set in December 2008 were 35.2 for formal text, 29.2 for informal text, 33.2 for formal speech, and 27.6 for informal speech.
    Page 5, “SW System”
  4. Since MT systems are tuned for word-based overlap measures (such as BLEU ), verb deletion is penalized equally as, for example, determiner deletion.
    Page 8, “Results”

See all papers in Proc. ACL 2009 that mention BLEU.

See all papers in Proc. ACL that mention BLEU.

Back to top.

machine translation

Appears in 3 sentences as: machine translation (3)
In Who, What, When, Where, Why? Comparing Multiple Approaches to the Cross-Lingual 5W Task
  1. Cross-lingual tasks are especially difficult due to the compounding effect of errors in language processing and errors in machine translation (MT).
    Page 1, “Abstract”
  2. 0 How much does machine translation (MT) degrade the performance of cross-lingual 5W systems, as compared to monolingual performance?
    Page 1, “Introduction”
  3. In this task, both machine translation (MT) and SW extraction must succeed in order to produce correct answers.
    Page 3, “The Chinese-English 5W Task”

See all papers in Proc. ACL 2009 that mention machine translation.

See all papers in Proc. ACL that mention machine translation.

Back to top.

Semantic Role

Appears in 3 sentences as: Semantic Role (1) semantic role (1) semantic roles (1)
In Who, What, When, Where, Why? Comparing Multiple Approaches to the Cross-Lingual 5W Task
  1. To solve this problem, a number of different problems in NLP must be addressed: predicate identification, argument extraction, attachment disambiguation, location and time expression recognition, and (partial) semantic role labeling.
    Page 1, “Introduction”
  2. The 5W task is also closely related to Semantic Role Labeling (SRL), which aims to efficiently and effectively derive semantic information from text.
    Page 2, “Prior Work”
  3. In this task, the 5W’s refer to semantic roles Within a sentence, as defined in Table 1.
    Page 2, “The Chinese-English 5W Task”

See all papers in Proc. ACL 2009 that mention Semantic Role.

See all papers in Proc. ACL that mention Semantic Role.

Back to top.

sentence-level

Appears in 3 sentences as: sentence-level (3)
In Who, What, When, Where, Why? Comparing Multiple Approaches to the Cross-Lingual 5W Task
  1. In this paper, we present an error analysis of a new cross-lingual task: the SW task, a sentence-level understanding task which seeks to return the English 5W's (Who, What, When, Where and Why) corresponding to a Chinese sentence.
    Page 1, “Abstract”
  2. The best cross-lingual 5W system was still 19% worse than the best monolingual 5W system, which shows that MT significantly degrades sentence-level understanding.
    Page 1, “Abstract”
  3. The best cross-lingual 5W system was still 19% worse than the best monolingual 5W system, which shows that MT significantly degrades sentence-level understanding.
    Page 8, “Conclusions”

See all papers in Proc. ACL 2009 that mention sentence-level.

See all papers in Proc. ACL that mention sentence-level.

Back to top.

PDF available from www.aclweb.org.
Send us an email