Enhancing Performance of Lexicalised Grammars
Dridan, Rebecca and Kordoni, Valia and Nicholson, Jeremy

Article Structure

Abstract

This paper describes how external resources can be used to improve parser performance for heavily lexicalised grammars, looking at both robustness and efficiency.

Introduction

Heavily lexicalised grammars have been used in applications such as machine translation and information extraction because they can produce semantic structures which provide more information than less informed parsers.

Background

In all heavily lexicalised formalisms, such as LTAG, CCG, LPG and HPSG, the lexicon plays a key role in parsing.

Parser Restriction

An exhaustive parser, such as PET, by default produces every parse licensed by the grammar.

Unknown Word Handling

The lexical information available to the parser is what makes the depth of the analysis possible, and the default configuration of the parser uses an all-or-nothing approach, where a parse is not produced if all the lexical information is not available.

Conclusion

The work reported here shows the benefits that can be gained by utilising external resources to annotate parser input in highly lexicalised grammar for-malisms.

Further Work

A number of avenues of future research were suggested by the observations made during this work.

Topics

POS tags

Appears in 48 sentences as: POS tag (9) POS tagged (3) POS tagger (8) POS taggers (1) POS tagging (1) POS tags (32)
In Enhancing Performance of Lexicalised Grammars
  1. In terms of robustness, we try using different types of external data to increase lexical coverage, and find that simple POS tags have the most effect, increasing coverage on unseen data by up to 45%.
    Page 1, “Abstract”
  2. Even using vanilla POS tags we achieve some efficiency gains, but when using detailed lexical types as supertags we manage to halve parsing time with minimal loss of coverage or precision.
    Page 1, “Abstract”
  3. Supertagging is the process of assigning probable ‘supertags’ to words before parsing to restrict parser ambiguity, where a supertag is a tag that includes more specific information than the typical POS tags .
    Page 2, “Background”
  4. In these experiments we look at two methods of restricting the parser, first by using POS tags and then using lexical types.
    Page 2, “Parser Restriction”
  5. We use TreeTagger (Schmid, 1994) to produce POS tags and then open class words are restricted if the POS tagger assigned a tag with a probability over a certain threshold.
    Page 2, “Parser Restriction”
  6. Table 1: Results obtained when restricting the parser lexicon according to the POS tag , where words are restricted according to a threshold of POS probabilities.
    Page 3, “Parser Restriction”
  7. Restricting by lexical types should have the effect of reducing ambiguity further than POS tags can do, since one POS tag could still allow the use of multiple lexical items with compatible lexical types.
    Page 3, “Parser Restriction”
  8. On the other hand, it could be considered more difficult to tag accurately, since there are many more lexical types than POS tags (almost 900 in the ERG) and less training data is available.
    Page 3, “Parser Restriction”
  9. Two models, with and without POS tags as features, were used.
    Page 3, “Parser Restriction”
  10. While POS taggers such as TreeTagger are common, and there some supertaggers are available, notably that of Clark and Curran (2007) for CCG, no standard supertagger exists for HPSG.
    Page 3, “Parser Restriction”
  11. The second used the features from the first, along with POS tags given by TreeTagger for the context tokens.
    Page 3, “Parser Restriction”

See all papers in Proc. ACL 2008 that mention POS tags.

See all papers in Proc. ACL that mention POS tags.

Back to top.

lexicalised

Appears in 10 sentences as: Lexicalised (1) lexicalised (9)
In Enhancing Performance of Lexicalised Grammars
  1. This paper describes how external resources can be used to improve parser performance for heavily lexicalised grammars, looking at both robustness and efficiency.
    Page 1, “Abstract”
  2. Heavily lexicalised grammars have been used in applications such as machine translation and information extraction because they can produce semantic structures which provide more information than less informed parsers.
    Page 1, “Introduction”
  3. pf Lexicalised Grammars
    Page 1, “Introduction”
  4. improving parser performance in these two areas, by annotating the input given to one such deep parser, the PET parser (Callmeier, 2000), which uses lexicalised grammars developed under the HPSG formalism (Pollard and Sag, 1994).
    Page 1, “Introduction”
  5. In all heavily lexicalised formalisms, such as LTAG, CCG, LPG and HPSG, the lexicon plays a key role in parsing.
    Page 1, “Background”
  6. ing a deep parser with a lexicalised grammar are the precision and depth of the analysis produced, but this depth comes from making many fine distinctions which greatly increases the parser search space, making parsing slow.
    Page 2, “Parser Restriction”
  7. Increasing efficiency is important for enabling these heavily lexicalised grammars to bring the benefits of their deep analyses to applications, but simi-
    Page 4, “Parser Restriction”
  8. These results show very clearly one of the potential drawbacks of using a highly lexicalised grammar formalism like HPSG: unknown words are one of the main causes of parse failure, as quantified in Baldwin et al.
    Page 5, “Unknown Word Handling”
  9. The work reported here shows the benefits that can be gained by utilising external resources to annotate parser input in highly lexicalised grammar for-malisms.
    Page 7, “Conclusion”
  10. Even something as simple and readily available (for languages likely to have lexicalised grammars) as a POS tagger can massively increase the parser coverage on unseen text.
    Page 7, “Conclusion”

See all papers in Proc. ACL 2008 that mention lexicalised.

See all papers in Proc. ACL that mention lexicalised.

Back to top.

treebank

Appears in 9 sentences as: treebank (5) treebanking (1) treebanks (3)
In Enhancing Performance of Lexicalised Grammars
  1. In the case study we describe here, the tools, grammars and treebanks we use are taken from work carried out in the DELPH-IN1 collaboration.
    Page 2, “Background”
  2. We also use the PET parser, and the [incr tsdb()] system profiler and treebanking tool (Oepen, 2001) for evaluation.
    Page 2, “Background”
  3. The data set used for these experiments is the jh5 section of the treebank released with the ERG.
    Page 2, “Parser Restriction”
  4. Since a gold standard treebank for our data set was available, it was possible to evaluate the accuracy of the parser.
    Page 2, “Parser Restriction”
  5. Consequently, we developed a Maximum Entropy model for supertagging using the OpenNLP implementation.2 Similarly to Zhang and Kordoni (2006), we took training data from the gold—standard lexical types in the treebank associated with ERG (in our case, the July-07 version).
    Page 3, “Parser Restriction”
  6. We held back the jh5 section of the treebank for testing the Maximum Entropy model.
    Page 3, “Parser Restriction”
  7. Four sets are English text: jh5 described in Section 3; tree consisting of questions from TREC and included in the treebanks released with the ERG; a00 which is taken from the BNC and consists of factsheets and newsletters; and depbank, the 700 sentences of the Briscoe and Carroll version of DepBank (Briscoe and Carroll, 2006) taken from the Wall Street Journal.
    Page 5, “Unknown Word Handling”
  8. The last two data sets are German text: clef700 consisting of German questions taken from the CLEF competition and eiche564 a sample of sentences taken from a treebank parsed with the German HPSG grammar, GG and consisting of transcribed German speech data concerning appointment scheduling from the Verbmobil project.
    Page 5, “Unknown Word Handling”
  9. Since the primary effect of adding POS tags is shown with those data sets for which we do not have gold standard treebanks , evaluating accuracy in this case is more difficult.
    Page 5, “Unknown Word Handling”

See all papers in Proc. ACL 2008 that mention treebank.

See all papers in Proc. ACL that mention treebank.

Back to top.

gold standard

Appears in 8 sentences as: Gold standard (3) gold standard (5)
In Enhancing Performance of Lexicalised Grammars
  1. Since a gold standard treebank for our data set was available, it was possible to evaluate the accuracy of the parser.
    Page 2, “Parser Restriction”
  2. A parse was judged to be correct if it exactly matched the gold standard tree in all aspects, syntactic and semantic.
    Page 3, “Parser Restriction”
  3. Table 2 shows the results achieved by these two models, with the unrestricted results and the gold standard provided for comparison.
    Page 3, “Parser Restriction”
  4. In this experiment, we used gold standard training data, but much less of it (just under 200 000 words) and still achieved a very good precision.
    Page 4, “Parser Restriction”
  5. 80 - - - Gold standard ——®— POS tags
    Page 4, “Parser Restriction”
  6. 95 | l I 90 — —85 — - - - Gold standard —
    Page 4, “Parser Restriction”
  7. Gold standard and unrestricted results shown for comparison.
    Page 4, “Parser Restriction”
  8. Since the primary effect of adding POS tags is shown with those data sets for which we do not have gold standard treebanks, evaluating accuracy in this case is more difficult.
    Page 5, “Unknown Word Handling”

See all papers in Proc. ACL 2008 that mention gold standard.

See all papers in Proc. ACL that mention gold standard.

Back to top.

CCG

Appears in 7 sentences as: CCG (8)
In Enhancing Performance of Lexicalised Grammars
  1. In all heavily lexicalised formalisms, such as LTAG, CCG , LPG and HPSG, the lexicon plays a key role in parsing.
    Page 1, “Background”
  2. Originally described by Bangalore and J oshi (1994) for use in LTAG parsing, it has also been used very successfully for CCG (Clark, 2002).
    Page 2, “Background”
  3. The supertags used in each formalism differ, being elementary trees in LTAG and CCG categories for CCG .
    Page 2, “Background”
  4. Unlike elementary trees and CCG categories, which are predominantly syntactic categories, the HPSG lexical types contain a lot of semantic information, as well as syntactic.
    Page 2, “Background”
  5. This could be considered a form of supertagging as used in LTAG and CCG .
    Page 3, “Parser Restriction”
  6. While POS taggers such as TreeTagger are common, and there some supertaggers are available, notably that of Clark and Curran (2007) for CCG , no standard supertagger exists for HPSG.
    Page 3, “Parser Restriction”
  7. This would be similar to the CCG supertagging mechanism and is likely to give generous speedups at the possible expense of precision, but it would be illuminating to discover how this tradeoff plays out in our setup.
    Page 8, “Further Work”

See all papers in Proc. ACL 2008 that mention CCG.

See all papers in Proc. ACL that mention CCG.

Back to top.

maximum entropy

Appears in 7 sentences as: Maximum Entropy (2) maximum entropy (7)
In Enhancing Performance of Lexicalised Grammars
  1. Consequently, we developed a Maximum Entropy model for supertagging using the OpenNLP implementation.2 Similarly to Zhang and Kordoni (2006), we took training data from the gold—standard lexical types in the treebank associated with ERG (in our case, the July-07 version).
    Page 3, “Parser Restriction”
  2. We held back the jh5 section of the treebank for testing the Maximum Entropy model.
    Page 3, “Parser Restriction”
  3. Again, the lexical items that were to be restricted were controlled by a threshold, in this case the probability given by the maximum entropy model.
    Page 3, “Parser Restriction”
  4. The other difference was the tagging model, maximum entropy versus Hidden Markov Model (HMM).
    Page 4, “Parser Restriction”
  5. We selected maximum entropy because Zhang and Kordoni (2006) had shown that they got better results using a maximum entropy tagger instead of a HMM one when predicting lexical types, albeit for a slightly different purpose.
    Page 4, “Parser Restriction”
  6. The same maximum entropy tagger used in Section 3 was used and each open class word was tagged with its most likely lexical type, as predicted by the maximum entropy model.
    Page 6, “Unknown Word Handling”
  7. Again it is clear that the use of POS tags as features obviously improves the maximum entropy model, since this second model has almost 10% better coverage on our unseen texts.
    Page 6, “Unknown Word Handling”

See all papers in Proc. ACL 2008 that mention maximum entropy.

See all papers in Proc. ACL that mention maximum entropy.

Back to top.

named entity

Appears in 3 sentences as: named entity (3)
In Enhancing Performance of Lexicalised Grammars
  1. Since the parser has the means to accept named entity (NE) information in the input, we also experimented with using generic lexical items generated from NE data.
    Page 6, “Unknown Word Handling”
  2. It is possible that another named entity tagger would give better results, and this may be looked at in future experiments.
    Page 6, “Unknown Word Handling”
  3. While annotating with named entity data or a lexical type supertagger were also found to increase coverage, the POS tagger had the greatest effect with up to 45% coverage increase on unseen text.
    Page 7, “Conclusion”

See all papers in Proc. ACL 2008 that mention named entity.

See all papers in Proc. ACL that mention named entity.

Back to top.

PDF available from www.aclweb.org.
Send us an email