Name Translation in Statistical Machine Translation - Learning When to Transliterate
Hermjakob, Ulf and Knight, Kevin and Daumé III, Hal

Article Structure

Abstract

We present a method to transliterate names in the framework of end-to-end statistical machine translation.

Introduction

State-of-the-art statistical machine translation (SMT) is bad at translating names that are not very common, particularly across languages with different character sets and sound systems.

Evaluation

In this section we present the evaluation method that we use to measure our system and also discuss challenges in name transliteration evaluation.

Transliterator

This section describes how we transliterate Arabic words or phrases.

Learning what to transliterate

As already mentioned in the introduction, named entity (NE) identification followed by MT is a bad idea.

Integration with SMT

We use the following method to integrate our transliterator into the overall SMT system:

End-to-End results

We applied the NEWA metric (section 2) to both our SMT translations as well as the four human reference translations, using both the original named-entity translation annotation and the re-annotation:

Discussion

We have shown that a state-of-the-art statistical machine translation system can benefit from a dedicated transliteration module to improve the transla-

Topics

SMT system

Appears in 16 sentences as: SMT System (1) SMT system (15) SMT system: (1)
In Name Translation in Statistical Machine Translation - Learning When to Transliterate
  1. The SMT system drops most names in this example.
    Page 1, “Introduction”
  2. The simplest way to integrate name handling into SMT is: (1) run a named-entity identification system on the source sentence, (2) transliterate identified entities with a special-purpose transliteration component, and (3) run the SMT system on the source sentence, as usual, but when looking up phrasal translations for the words identified in step 1, instead use the transliterations from step 2.
    Page 2, “Introduction”
  3. The base SMT system may translate a commonly-occurring name just fine, due to the bitext it was trained on, while the transliteration component can easily supply a worse answer.
    Page 2, “Introduction”
  4. Even if the named-entity identification and transliteration components operate perfectly, adopting their translations means that the SMT system may no longer have access to longer phrases that include the name.
    Page 2, “Introduction”
  5. For example, our base SMT system translates Jug
    Page 2, “Introduction”
  6. We can readily apply it to any base SMT system , and to human translations as well.
    Page 2, “Introduction”
  7. Our goal in augmenting a base SMT system is to increase this percentage.
    Page 2, “Introduction”
  8. 0 We evaluate both the base SMT system and the augmented system in terms of entity translation accuracy and BLEU (Sections 2 and 6).
    Page 2, “Introduction”
  9. We use the following method to integrate our transliterator into the overall SMT system:
    Page 6, “Integration with SMT”
  10. In a tuning step, the Minimim Error Rate Training component of our SMT system iteratively adjusts the set of rule weights, including the weight associated with the transliteration feature, such that the English translations are optimized with respect to a set of known reference translations according to the BLEU translation metric.
    Page 7, “Integration with SMT”
  11. At runtime, the transliterations then compete with the translations generated by the general SMT system .
    Page 7, “Integration with SMT”

See all papers in Proc. ACL 2008 that mention SMT system.

See all papers in Proc. ACL that mention SMT system.

Back to top.

named entities

Appears in 9 sentences as: named entities (6) Named Entity (1) named entity (3)
In Name Translation in Statistical Machine Translation - Learning When to Transliterate
  1. 0 Not all named entities should be transliterated.
    Page 2, “Introduction”
  2. Many named entities require a mix of transliteration and translation.
    Page 2, “Introduction”
  3. We ask: what percentage of source-language named entities are translated correctly?
    Page 2, “Introduction”
  4. General MT metrics such as BLEU, TER, METEOR are not suitable for evaluating named entity translation and transliteration, because they are not focused on named entities (NEs).
    Page 2, “Evaluation”
  5. The general idea of the Named Entity Weak Accuracy (NEWA) metric is to
    Page 3, “Evaluation”
  6. BBN kindly provided us with an annotated Arabic text corpus, in which named entities were marked up with their type (e. g. GPE for Geopolitical Entity) and one or more English translations.
    Page 3, “Evaluation”
  7. As already mentioned in the introduction, named entity (NE) identification followed by MT is a bad idea.
    Page 5, “Learning what to transliterate”
  8. Table 2: Name translation accuracy with respect to BBN and re-annotated Gold Standard on 1730 named entities in 637 sentences.
    Page 7, “End-to-End results”
  9. Improved named entity translation accuracy as measured by the NEWA metric in general, and a reduction in dropped names in particular is clearly valuable to the human reader of machine translated documents as well as for systems using machine translation for further information processing.
    Page 8, “Discussion”

See all papers in Proc. ACL 2008 that mention named entities.

See all papers in Proc. ACL that mention named entities.

Back to top.

BLEU

Appears in 8 sentences as: BLEU (8)
In Name Translation in Statistical Machine Translation - Learning When to Transliterate
  1. First, although names are important to human readers, automatic MT scoring metrics (such as BLEU ) do not encourage researchers to improve name translation in the context of MT.
    Page 1, “Introduction”
  2. A secondary goal is to make sure that our overall translation quality (as measured by BLEU ) does not degrade as a result of the name-handling techniques we introduce.
    Page 2, “Introduction”
  3. 0 We evaluate both the base SMT system and the augmented system in terms of entity translation accuracy and BLEU (Sections 2 and 6).
    Page 2, “Introduction”
  4. General MT metrics such as BLEU , TER, METEOR are not suitable for evaluating named entity translation and transliteration, because they are not focused on named entities (NEs).
    Page 2, “Evaluation”
  5. In a tuning step, the Minimim Error Rate Training component of our SMT system iteratively adjusts the set of rule weights, including the weight associated with the transliteration feature, such that the English translations are optimized with respect to a set of known reference translations according to the BLEU translation metric.
    Page 7, “Integration with SMT”
  6. To make sure our name transliterator does not degrade the overall translation quality, we evaluated our base SMT system with BLEU , as well as our transliteration-augmented SMT system.
    Page 7, “End-to-End results”
  7. The BLEU scores for the two systems were 50.70 and 50.96 respectively.
    Page 8, “End-to-End results”
  8. At the same time, there has been no negative impact on overall quality as measured by BLEU .
    Page 8, “Discussion”

See all papers in Proc. ACL 2008 that mention BLEU.

See all papers in Proc. ACL that mention BLEU.

Back to top.

machine translation

Appears in 6 sentences as: machine translated (1) machine translation (6)
In Name Translation in Statistical Machine Translation - Learning When to Transliterate
  1. We present a method to transliterate names in the framework of end-to-end statistical machine translation .
    Page 1, “Abstract”
  2. State-of-the-art statistical machine translation (SMT) is bad at translating names that are not very common, particularly across languages with different character sets and sound systems.
    Page 1, “Introduction”
  3. This evaluation involves a mixture of entity identification and translation concems—for example, the scoring system asks for coreference determination, which may or may not be of interest for improving machine translation output.
    Page 2, “Introduction”
  4. Finally, here are end-to-end machine translation results for three sentences, with and without the transliteration module, along with a human reference translation.
    Page 8, “End-to-End results”
  5. We have shown that a state-of-the-art statistical machine translation system can benefit from a dedicated transliteration module to improve the transla-
    Page 8, “Discussion”
  6. Improved named entity translation accuracy as measured by the NEWA metric in general, and a reduction in dropped names in particular is clearly valuable to the human reader of machine translated documents as well as for systems using machine translation for further information processing.
    Page 8, “Discussion”

See all papers in Proc. ACL 2008 that mention machine translation.

See all papers in Proc. ACL that mention machine translation.

Back to top.

end-to-end

Appears in 5 sentences as: end-to-end (5)
In Name Translation in Statistical Machine Translation - Learning When to Transliterate
  1. We present a method to transliterate names in the framework of end-to-end statistical machine translation.
    Page 1, “Abstract”
  2. The task of transliterating names (independent of end-to-end MT) has received a significant amount of research, e.g., (Knight and Graehl, 1997; Chen et al., 1998; Al-Onaizan, 2002).
    Page 1, “Introduction”
  3. Most of this work has been disconnected from end-to-end MT, a problem which we address head-on in this paper.
    Page 2, “Introduction”
  4. In the result section of this paper, we will use the NEWA metric to measure and compare the accuracy of NE translations in our end-to-end SMT translations and four human reference translations.
    Page 3, “Evaluation”
  5. Finally, here are end-to-end machine translation results for three sentences, with and without the transliteration module, along with a human reference translation.
    Page 8, “End-to-End results”

See all papers in Proc. ACL 2008 that mention end-to-end.

See all papers in Proc. ACL that mention end-to-end.

Back to top.

Gold Standard

Appears in 5 sentences as: Gold Standard (3) gold standard (2)
In Name Translation in Statistical Machine Translation - Learning When to Transliterate
  1. Based on the issues we found with the BBN annotations, we re-annotated a sub-corpus of 637 sentences of the BBN gold standard .
    Page 3, “Evaluation”
  2. Testing on 10,000 sentences, we achieve precision of 87.4% and a recall of 95 .7% with respect to the automatically marked-up Gold Standard as described in section 4.1.
    Page 6, “Learning what to transliterate”
  3. After adjusting for these deficiencies in the gold standard , we achieve precision of 92.1% and recall of 95.9% in the name tagging task.
    Page 6, “Learning what to transliterate”
  4. Gold Standard BBN GS Re-annotated GS Human 1 87.0% 85.0% Human 2 85.3% 86.9% Human 3 90.4% 91.8% Human 4 86.5% 88.3% SMT System 80.4% 89.7%
    Page 7, “End-to-End results”
  5. Table 2: Name translation accuracy with respect to BBN and re-annotated Gold Standard on 1730 named entities in 637 sentences.
    Page 7, “End-to-End results”

See all papers in Proc. ACL 2008 that mention Gold Standard.

See all papers in Proc. ACL that mention Gold Standard.

Back to top.

bigram

Appears in 4 sentences as: bigram (3) bigrams (1)
In Name Translation in Statistical Machine Translation - Learning When to Transliterate
  1. The same consonant skeleton indexing process is applied to name bigrams (47,700,548 unique with 167,398,054 skeletons) and trigrams (46,543,712 unique with 165,536,451 skeletons).
    Page 4, “Transliterator”
  2. From the stat section we collect statistics as to how often every word, bigram or trigram occurs, and what distribution of name/non-name patterns these ngrams have.
    Page 6, “Learning what to transliterate”
  3. The name distribution bigram
    Page 6, “Learning what to transliterate”
  4. stat corpus bitext, the first word is a marked up as a non-name (”0”) and the second as a name (”1”), which strongly suggests that in such a bigram context, aljzyre better be translated as island or peninsula, and not be transliterated as Al-Jazeera.
    Page 6, “Learning what to transliterate”

See all papers in Proc. ACL 2008 that mention bigram.

See all papers in Proc. ACL that mention bigram.

Back to top.

translation quality

Appears in 4 sentences as: translation quality (4)
In Name Translation in Statistical Machine Translation - Learning When to Transliterate
  1. Typically, translation quality is degraded rather than improved, for the following reasons:
    Page 2, “Introduction”
  2. A secondary goal is to make sure that our overall translation quality (as measured by BLEU) does not degrade as a result of the name-handling techniques we introduce.
    Page 2, “Introduction”
  3. Note that name translation quality varies greatly between human translators, with error rates ranging from 8.2-15.0% (absolute).
    Page 7, “End-to-End results”
  4. To make sure our name transliterator does not degrade the overall translation quality , we evaluated our base SMT system with BLEU, as well as our transliteration-augmented SMT system.
    Page 7, “End-to-End results”

See all papers in Proc. ACL 2008 that mention translation quality.

See all papers in Proc. ACL that mention translation quality.

Back to top.

statistical machine translation

Appears in 3 sentences as: statistical machine translation (3)
In Name Translation in Statistical Machine Translation - Learning When to Transliterate
  1. We present a method to transliterate names in the framework of end-to-end statistical machine translation .
    Page 1, “Abstract”
  2. State-of-the-art statistical machine translation (SMT) is bad at translating names that are not very common, particularly across languages with different character sets and sound systems.
    Page 1, “Introduction”
  3. We have shown that a state-of-the-art statistical machine translation system can benefit from a dedicated transliteration module to improve the transla-
    Page 8, “Discussion”

See all papers in Proc. ACL 2008 that mention statistical machine translation.

See all papers in Proc. ACL that mention statistical machine translation.

Back to top.

PDF available from www.aclweb.org.
Send us an email