Article Structure

Abstract

What do we want to learn from a translation competition and how do we learn it with confidence?

The WMT Translation Competition

Every year, the Workshop on Machine Translation (WMT) conducts a competition between machine translation systems.

A Ranking Problem

For several years, WMT used the following heuristic for ranking the translation systems:

A Problem with Rankings

Suppose four systems participate in a translation competition.

From Rankings to Relative Ability

Ostensibly the purpose of a translation competition is to determine the relative ability of a set of translation systems.

Baselines

Let’s begin then, and create some simple preference models to serve as baselines.

Simple Bayesian Models 6.1 Independent Pairs

Another simple model is the direct estimation of each relative ability P(7r|31, 32) independently.

Item-Response Theoretic (IRT) Models

Let’s revisit (Lopez, 2012)’s objection to the B0-JAR ranking heuristic: “...couldn’t a system still be penalized simply by being compared to [good systems] more frequently than its competitors?” The official WMT 2012 findings (Callison-Burch et al., 2012) echoes this concern in justifying the exclusion of reference translations from the 2012 competition:

Experiments

We organized the competition data as described at the end of Section 4.

Conclusion

WMT has faced a crisis of confidence lately, with researchers raising (real and conjectured) issues with its analytical methodology.