Article Structure
Abstract
We outline the problem of ad hoc rules in treebanks, rules used for specific constructions in one data set and unlikely to be used again.
Introduction and Motivation
When extracting rules from constituency-based treebanks employing flat structures, grammars often limit the set of rules (e.g., Charniak, 1996), due to the large number of rules (Krotov et al., 1998) and “leaky” rules that can lead to mis-analysis (Foth and Menzel, 2006).
Background
2.1 Equivalence classes
Rule dissimilarity and generalizability
3.1 Criteria for rule equivalence
Evaluation
To gauge our success in detecting ad hoc rules, we evaluate the reliability scores in two main ways: 1) whether unreliable rules generalize to new data (section 4.1), and, more importantly, 2) whether the unreliable rules which do generalize are ad hoc in other ways—e.g., erroneous (section 4.2).
Summary and Outlook
We have outlined the problem of ad hoc rules in treebanks—ungeneralizable rules, erroneous rules, rules for ungrammatical text, and rules which are not necessarily consistent with the rest of the annotation scheme.
Topics
bigram
- To do this, we can examine the weakest parts of each rule and compare those across the corpus, to see which anomalous patterns emerge; we do this in the Bigram scoring section below.Page 3, “Rule dissimilarity and generalizability”
- Bigram scoring The other method of detecting ad hoc rules calculates reliability scores by focusing specifically on what the classes do not have in common.Page 4, “Rule dissimilarity and generalizability”
- We abstract to bigrams , including added START and END tags, as longer sequences risk missing generalizations; e. g., unary rules would have no comparable rules.Page 4, “Rule dissimilarity and generalizability”
- Calculate the frequency of each <mother,bigram> pair in a reduced rule: for every reduced rule token with the same pair, add a score of l for that bigram pair.Page 4, “Rule dissimilarity and generalizability”
- Assign the score of the least-frequent bigram as the score of the rule.Page 4, “Rule dissimilarity and generalizability”
- We assign the score of the lowest-scoring bigram because we are interested in anomalous sequences.Page 4, “Rule dissimilarity and generalizability”
- This is in the spirit of Kvéton and Oliva (2002), who define invalid bigrams for POS annotation sequences in order to detect annotation errors..Page 4, “Rule dissimilarity and generalizability”
- As one example, consider (6), where the reduced rule NP —> NP DT NNP is composed of the bigrams START NP, NP DT, DT NNP, and NNP END.Page 4, “Rule dissimilarity and generalizability”
- The disadvantage of the bigram scoring, however, is its missing of the big picture: for example, the erroneous rule NP —> NNP CC NP gets a large score (1905) because each subsequence is quite common.Page 4, “Rule dissimilarity and generalizability”
- The results are shown in figure 1 for the whole daughters scoring method and in figure 2 for the bigram method.Page 5, “Evaluation”
- Figure 2: Bigram ungeneralizability (devo.)Page 5, “Evaluation”
See all papers in Proc. ACL 2008 that mention bigram.
See all papers in Proc. ACL that mention bigram.
Back to top.
treebank
- We outline the problem of ad hoc rules in treebanks , rules used for specific constructions in one data set and unlikely to be used again.Page 1, “Abstract”
- Based on a simple notion of rule equivalence and on the idea of finding rules unlike any others, we develop two methods for detecting ad hoc rules in flat treebanks and show they are successful in detecting such rules.Page 1, “Abstract”
- When extracting rules from constituency-based treebanks employing flat structures, grammars often limit the set of rules (e.g., Charniak, 1996), due to the large number of rules (Krotov et al., 1998) and “leaky” rules that can lead to mis-analysis (Foth and Menzel, 2006).Page 1, “Introduction and Motivation”
- Thus, we need to carefully consider the applicability of rules in a treebank to new text.Page 1, “Introduction and Motivation”
- For example, when ungrammatical or nonstandard text is used, treebanks employ rules to cover it, but do not usually indicate ungrammaticality in the annotation.Page 1, “Introduction and Motivation”
- And these rules are outright damaging if the set of treebank rules is intended to accurately capture the grammar of a language.Page 1, “Introduction and Motivation”
- If a treebank grammar is used (e.g., Metcalf and Boyd,Page 1, “Introduction and Motivation”
- Thus, identifying ad hoc rules can also provide feedback on annotation schemes, an especially important step if one is to use the treebank for specific applications (see, e.g., Vadas and Curran, 2007), or if one is in the process of developing a treebank .Page 2, “Introduction and Motivation”
- Although statistical techniques have been employed to detect anomalous annotation (Ule and Simov, 2004; Eskin, 2000), these methods do not account for linguistically-motivated generalizations across rules, and no full evaluation has been done on a treebank .Page 2, “Introduction and Motivation”
- Treebank (Marcus et al., 1993), six of which are errors.Page 2, “Background”
- For example, in (2), the daughters list RB TO JJ NNS is a daughters list with no correlates in the treebank ; it is erroneous because close to wholesale needs another layer of structure, namely adjective phrase (ADJP) (Bies et al., 1995, p. 179).Page 2, “Background”
See all papers in Proc. ACL 2008 that mention treebank.
See all papers in Proc. ACL that mention treebank.
Back to top.
