The two systems differed in acoustic models, confidence scoring model, state tracking method and parameters, number of supported routes (8 vs 40, for D81 and D82 respectively), presence of minor bugs, and user population.

As a baseline, we construct a handcrafted state tracking rule that follows a strategy common in commercial systems: it returns the SLU result with the maximum confidence score , ignoring all other hypotheses.

For example, if the user says “no” to an explicit confirmation or “go back” to an implicit confirmation, they are asked the same question again, which gives an opportunity for a higher confidence score .

Base features consider information about the current turn, including rank of the current SLU result (current hypothesis), the SLU result confidence score (s) in the current N-best list, the difference between the current hypothesis score and the best hypothesis score in the current N-best list, etc.

Those include the number of times an SLU result has been observed before, the number of times an SLU result has been observed before at a specific rank such as rank 1, the sum and average of confidence scores of SLU results across all past recognitions, the number of possible past user negations 0r confirmations 0f the current SLU result etc.

For example, from the current turn, we use the number of distinct SLU results, the entropy of the confidence scores , the best path score of the word confusion network, etc.

For dialog state tracking, most commercial systems use handcrafted heuristics, selecting the SLU result with the highest confidence score , and discarding alternatives.

As described in Section 3.2, the weight of each variable is a linear combination of the language model score, three classifier confidence scores , and three classifier disagreement scores.

Finally, the language model score, classifier confidence scores , and classifier disagreement scores are normalized to take values in [0, 1], based on the H00 2011 development data.

The confidence scores f (s’ , t) of classifiers, where t E E and E is the set of classifiers.

For each article instance in s’, the classifier computes the difference between the maximum confidence score among all possible choices of articles, and the confidence score of the observed article.

Next, to compute whpyg, we collect language model score and confidence scores from the article (ART), preposition (PREP), and noun number (NOUN) classifier, i.e., E = {ART, PREP, NOUN}.

When measuring the gain due to 21131213312 2 1 (change cat to cats), the weight wNoungmluml is likely to be small since A cats will get a low language model score, a low article classifier confidence score, and a low noun number classifier confidence score .

We speculate that it may be helpful to introduce a confidence score for each pattern.

For the second key, we utilize opinion words and opinion patterns with their confidence scores to represent an opinion target.

where conf denotes confidence score estimated by RWR, f req(-) has the same meaning as in Section 3.2.

where T is the opinion target set in which each element is classified as positive during opinion target refinement, s(ti) denotes confidence score exported by the target refining classifier.

The label propagation assigns a confidence score 0 = (01,...,cm) to each node U 2 ul, ..., um, where the score is a real number between —1 and l. A score close to 1 indicates that we are very confident that the node (user) is a chronic critic.

Thus, by minimizing E(c), we assign the confidence scores considering the results of the opinion mining and agreement relationships among the users.

To avoid this problem, Yin and Tan (2011) introduced a neutral fact, which decreases each confidence score by considering the distance from the seeds.