Figure 6: The relative error rate (using round 0 as a baseline) of the best Mechanical Turk user session for each of the four numbers of topics.

The lower the classification error rate , the better the model has captured the structure of the corpus.4

While Null sees no constraints, it serves as an upper baseline for the error rate (lower error being better) but shows the effect of additional inference.

All Full is a lower baseline for the error rate since it both sees the constraints at the beginning and also runs for the maximum number of total iterations.

The experimental results demonstrate that our model is able to significantly outperform the state-of-the-art coherence model by Barzilay and Lapata (2005), reducing the error rate of the previous approach by an average of 29% over three data sets against human upper bounds.

For the combined model, the error rates are significantly reduced in all three data sets.

The average error rate reductions against 100% are 9.57% for the full model and 26.37% for the combined model.

If we compute the average error rate reductions against the human upper bounds (rather than an oracular 100%), the average error rate reduction for the full model is 29% and that for the combined model is 73%.

Table 2: Alignment error rate results for the bidirectional model versus the baseline directional models.

First, we measure alignment error rate (AER), which compares the pro-

The translation model weights were tuned for both the baseline and bidirectional alignments using lattice-based minimum error rate training (Kumar et al., 2009).

First, we can see from the results that several systems appear better when evaluating on a correlation measure like Pearson’s p, while others appear better when analyzing error rate .

Evaluating with a correlative measure yields predictably poor results, but evaluating the error rate indicates that it is comparable to (or better than) the more intelligent BOW metrics.

However, as the perceptron is designed to minimize error rate , this may not reflect an optimal objective when seeking to detect matches.