Cross-Domain Dependency Parsing Using a Deep Linguistic Grammar
Zhang, Yi and Wang, Rui

Article Structure

Abstract

Pure statistical parsing systems achieves high in-domain accuracy but performs poorly out-domain.

Introduction

Syntactic dependency parsing is attracting more and more research focus in recent years, partially due to its theory-neutral representation, but also thanks to its wide deployment in various NLP tasks (machine translation, textual entailment recognition, question answering, information extraction, etc.).

Parser Domain Adaptation

In recent years, two statistical dependency parsing systems, MaltParser (Nivre et al., 2007b) and MS TParser (McDonald et al., 2005b), representing different threads of research in data-driven machine learning approaches have obtained high publicity, for their state-of-the-art performances in open competitions such as CoNLL Shared Tasks.

Dependency Parsing with HPSG

In this section, we explore two possible applications of the HPSG parsing onto the syntactic dependency parsing task.

Experiment Results & Error Analyses

To evaluate the performance of our different dependency parsing models, we tested our approaches on several dependency treebanks for English in a similar spirit to the CoNLL 2006-2008 Shared Tasks.

Conclusion & Future Work

Similar to our work, Sagae et al.

Topics

CoNLL

Appears in 20 sentences as: CoNLL (20)
In Cross-Domain Dependency Parsing Using a Deep Linguistic Grammar
  1. In the meantime, successful continuation of CoNLL Shared Tasks since 2006 (Buchholz and Marsi, 2006; Nivre et al., 2007a; Surdeanu et al., 2008) have witnessed how easy it has become to train a statistical syntactic dependency parser provided that there is annotated treebank.
    Page 1, “Introduction”
  2. In recent years, two statistical dependency parsing systems, MaltParser (Nivre et al., 2007b) and MS TParser (McDonald et al., 2005b), representing different threads of research in data-driven machine learning approaches have obtained high publicity, for their state-of-the-art performances in open competitions such as CoNLL Shared Tasks.
    Page 1, “Parser Domain Adaptation”
  3. For these rules, we refer to the conversion of the Penn Treebank into dependency structures used in the CoNLL 2008 Shared Task, and mark the heads of these rules in a way that will arrive at a compatible dependency backbone.
    Page 3, “Dependency Parsing with HPSG”
  4. In combination with the right-branching analysis of coordination in ERG, this leads to the same dependency attachment in the CoNLL syntax.
    Page 3, “Dependency Parsing with HPSG”
  5. the CoNLL shared task dependency structures, minor systematic differences still exist for some phenomena.
    Page 4, “Dependency Parsing with HPSG”
  6. For example, the possessive “’s” is annotated to be governed by its preceding word in CoNLL dependency; while in HP SG, it is treated as the head of a “specifier-head” construction, hence governing the preceding word in the dependency backbone.
    Page 4, “Dependency Parsing with HPSG”
  7. The unlabeled attachment agreement between the HP SG backbone and CoNLL dependency annotation will be shown in Section 4.2.
    Page 4, “Dependency Parsing with HPSG”
  8. 1It is also possible map from HPSG rule names (together with the part-of-speech of head and dependent) to CoNLL dependency labels.
    Page 4, “Dependency Parsing with HPSG”
  9. To evaluate the performance of our different dependency parsing models, we tested our approaches on several dependency treebanks for English in a similar spirit to the CoNLL 2006-2008 Shared Tasks.
    Page 5, “Experiment Results & Error Analyses”
  10. In previous years of CoNLL Shared Tasks, several datasets have been created for the purpose of dependency parser evaluation.
    Page 5, “Experiment Results & Error Analyses”
  11. Our experiments adhere to the CoNLL 2008 dependency syntax (Yamada et al.
    Page 5, “Experiment Results & Error Analyses”

See all papers in Proc. ACL 2009 that mention CoNLL.

See all papers in Proc. ACL that mention CoNLL.

Back to top.

dependency parsing

Appears in 20 sentences as: Dependency Parser (1) dependency parser (6) dependency parsers (1) Dependency Parsing (1) dependency parsing (12)
In Cross-Domain Dependency Parsing Using a Deep Linguistic Grammar
  1. The dependency backbone of an HP SG analysis is used to provide general linguistic insights which, when combined with state-of-the-art statistical dependency parsing models, achieves performance improvements on out-domain testsflL
    Page 1, “Abstract”
  2. Syntactic dependency parsing is attracting more and more research focus in recent years, partially due to its theory-neutral representation, but also thanks to its wide deployment in various NLP tasks (machine translation, textual entailment recognition, question answering, information extraction, etc.).
    Page 1, “Introduction”
  3. In combination with machine learning methods, several statistical dependency parsing models have reached comparable high parsing accuracy (McDonald et al., 2005b; Nivre et al., 2007b).
    Page 1, “Introduction”
  4. In the meantime, successful continuation of CoNLL Shared Tasks since 2006 (Buchholz and Marsi, 2006; Nivre et al., 2007a; Surdeanu et al., 2008) have witnessed how easy it has become to train a statistical syntactic dependency parser provided that there is annotated treebank.
    Page 1, “Introduction”
  5. In recent years, two statistical dependency parsing systems, MaltParser (Nivre et al., 2007b) and MS TParser (McDonald et al., 2005b), representing different threads of research in data-driven machine learning approaches have obtained high publicity, for their state-of-the-art performances in open competitions such as CoNLL Shared Tasks.
    Page 1, “Parser Domain Adaptation”
  6. In addition, most of the previous work have been focusing on constituent-based parsing, while the domain adaptation of the dependency parsing has not been fully explored.
    Page 2, “Parser Domain Adaptation”
  7. Figure 1: Different dependency parsing models and their combinations.
    Page 3, “Parser Domain Adaptation”
  8. In this section, we explore two possible applications of the HPSG parsing onto the syntactic dependency parsing task.
    Page 3, “Dependency Parsing with HPSG”
  9. One is to extract dependency backbone from the HP SG analyses of the sentences and directly convert them into the target representation; the other way is to encode the HP SG outputs as additional features into the existing statistical dependency parsing models.
    Page 3, “Dependency Parsing with HPSG”
  10. Besides directly using the dependency backbone of the HP SG output, we could also use it for building feature-based models of statistical dependency parsers .
    Page 4, “Dependency Parsing with HPSG”
  11. As mentioned before, MS TParser is a graph-based statistical dependency parser , whose leam-ing procedure can be viewed as the assignment of different weights to all kinds of dependency arcs.
    Page 4, “Dependency Parsing with HPSG”

See all papers in Proc. ACL 2009 that mention dependency parsing.

See all papers in Proc. ACL that mention dependency parsing.

Back to top.

Treebank

Appears in 14 sentences as: Treebank (8) treebank (3) treebanks (3)
In Cross-Domain Dependency Parsing Using a Deep Linguistic Grammar
  1. In the meantime, successful continuation of CoNLL Shared Tasks since 2006 (Buchholz and Marsi, 2006; Nivre et al., 2007a; Surdeanu et al., 2008) have witnessed how easy it has become to train a statistical syntactic dependency parser provided that there is annotated treebank .
    Page 1, “Introduction”
  2. the Wall Street Journal (WSJ) sections of the Penn Treebank (Marcus et al., 1993) as training set, tests on BROWN Sections typically result in a 6-8% drop in labeled attachment scores, although the average sentence length is much shorter in BROWN than that in WSJ.
    Page 1, “Introduction”
  3. Note that all grammar rules in ERG are either unary or binary, giving us relatively deep trees when compared with annotations such as Penn Treebank .
    Page 3, “Dependency Parsing with HPSG”
  4. For these rules, we refer to the conversion of the Penn Treebank into dependency structures used in the CoNLL 2008 Shared Task, and mark the heads of these rules in a way that will arrive at a compatible dependency backbone.
    Page 3, “Dependency Parsing with HPSG”
  5. 2More recent study shows that with carefully designed retokenization and preprocessing rules, over 80% sentential coverage can be achieved on the WSJ sections of the Penn Treebank data using the same version of ERG.
    Page 4, “Dependency Parsing with HPSG”
  6. To evaluate the performance of our different dependency parsing models, we tested our approaches on several dependency treebanks for English in a similar spirit to the CoNLL 2006-2008 Shared Tasks.
    Page 5, “Experiment Results & Error Analyses”
  7. Most of them are converted automatically from existing treebanks in various forms.
    Page 5, “Experiment Results & Error Analyses”
  8. The larger part is converted from the Penn Treebank Wall Street Journal Sections #2—#21, and is used for training statistical dependency parsing models; the smaller part, which covers sentences from Section #23, is used for testing.
    Page 5, “Experiment Results & Error Analyses”
  9. Brown This dataset contains a subset of converted sentences from BROWN sections of the Penn Treebank .
    Page 5, “Experiment Results & Error Analyses”
  10. Although the original annotation scheme is similar to the Penn Treebank , the dependency extraction setting is slightly different to the CoNLLWSJ dependencies (e.g.
    Page 5, “Experiment Results & Error Analyses”
  11. To pick the most probable reading from HP SG parsing outputs, we used a dis-criminative parse selection model as described in (Toutanova et al., 2002) trained on the LOGON Treebank (Oepen et al., 2004), which is significantly different from any of the test domain.
    Page 6, “Experiment Results & Error Analyses”

See all papers in Proc. ACL 2009 that mention Treebank.

See all papers in Proc. ACL that mention Treebank.

Back to top.

Shared Task

Appears in 12 sentences as: Shared Task (7) shared task (1) Shared Tasks (4)
In Cross-Domain Dependency Parsing Using a Deep Linguistic Grammar
  1. In the meantime, successful continuation of CoNLL Shared Tasks since 2006 (Buchholz and Marsi, 2006; Nivre et al., 2007a; Surdeanu et al., 2008) have witnessed how easy it has become to train a statistical syntactic dependency parser provided that there is annotated treebank.
    Page 1, “Introduction”
  2. In recent years, two statistical dependency parsing systems, MaltParser (Nivre et al., 2007b) and MS TParser (McDonald et al., 2005b), representing different threads of research in data-driven machine learning approaches have obtained high publicity, for their state-of-the-art performances in open competitions such as CoNLL Shared Tasks .
    Page 1, “Parser Domain Adaptation”
  3. For these rules, we refer to the conversion of the Penn Treebank into dependency structures used in the CoNLL 2008 Shared Task , and mark the heads of these rules in a way that will arrive at a compatible dependency backbone.
    Page 3, “Dependency Parsing with HPSG”
  4. the CoNLL shared task dependency structures, minor systematic differences still exist for some phenomena.
    Page 4, “Dependency Parsing with HPSG”
  5. To evaluate the performance of our different dependency parsing models, we tested our approaches on several dependency treebanks for English in a similar spirit to the CoNLL 2006-2008 Shared Tasks .
    Page 5, “Experiment Results & Error Analyses”
  6. In previous years of CoNLL Shared Tasks , several datasets have been created for the purpose of dependency parser evaluation.
    Page 5, “Experiment Results & Error Analyses”
  7. The same dataset has been used for the domain adaptation track of the CoNLL 2007 Shared Task .
    Page 5, “Experiment Results & Error Analyses”
  8. This is the other datasets used in the domain adaptation track of the CoNLL 2007 Shared Task .
    Page 5, “Experiment Results & Error Analyses”
  9. As have been reported by others, several systematic differences in the original CHILDES annotation scheme has led to the poor system performances on this track of the Shared Task in 2007.
    Page 5, “Experiment Results & Error Analyses”
  10. Admittedly the results on PCHEMTB are lower than the best reported results in CoNLL 2007 Shared Task , we shall note that we are not using any in-domain unlabeled data.
    Page 7, “Experiment Results & Error Analyses”
  11. With the results on BROWN, the performance of our HPSG feature models will rank 2nd on the out-domain test for the CoNLL 2008 Shared Task .
    Page 7, “Experiment Results & Error Analyses”

See all papers in Proc. ACL 2009 that mention Shared Task.

See all papers in Proc. ACL that mention Shared Task.

Back to top.

parsing models

Appears in 8 sentences as: parsing model (2) parsing models (6)
In Cross-Domain Dependency Parsing Using a Deep Linguistic Grammar
  1. The dependency backbone of an HP SG analysis is used to provide general linguistic insights which, when combined with state-of-the-art statistical dependency parsing models , achieves performance improvements on out-domain testsflL
    Page 1, “Abstract”
  2. In combination with machine learning methods, several statistical dependency parsing models have reached comparable high parsing accuracy (McDonald et al., 2005b; Nivre et al., 2007b).
    Page 1, “Introduction”
  3. Granted for the differences between their approaches, both systems heavily rely on machine learning methods to estimate the parsing model from an annotated corpus as training set.
    Page 2, “Parser Domain Adaptation”
  4. Due to the heavy cost of developing high quality large scale syntactically annotated corpora, even for a resource-rich language like English, only very few of them meets the criteria for training a general purpose statistical parsing model .
    Page 2, “Parser Domain Adaptation”
  5. Figure 1: Different dependency parsing models and their combinations.
    Page 3, “Parser Domain Adaptation”
  6. One is to extract dependency backbone from the HP SG analyses of the sentences and directly convert them into the target representation; the other way is to encode the HP SG outputs as additional features into the existing statistical dependency parsing models .
    Page 3, “Dependency Parsing with HPSG”
  7. To evaluate the performance of our different dependency parsing models , we tested our approaches on several dependency treebanks for English in a similar spirit to the CoNLL 2006-2008 Shared Tasks.
    Page 5, “Experiment Results & Error Analyses”
  8. The larger part is converted from the Penn Treebank Wall Street Journal Sections #2—#21, and is used for training statistical dependency parsing models ; the smaller part, which covers sentences from Section #23, is used for testing.
    Page 5, “Experiment Results & Error Analyses”

See all papers in Proc. ACL 2009 that mention parsing models.

See all papers in Proc. ACL that mention parsing models.

Back to top.

domain adaptation

Appears in 7 sentences as: domain adaptation (7)
In Cross-Domain Dependency Parsing Using a Deep Linguistic Grammar
  1. In addition, most of the previous work have been focusing on constituent-based parsing, while the domain adaptation of the dependency parsing has not been fully explored.
    Page 2, “Parser Domain Adaptation”
  2. This not to say that domain adaptation is
    Page 2, “Parser Domain Adaptation”
  3. Since we focus on the domain adaptation issue, we incorporate a less domain dependent language resource (i.e.
    Page 4, “Dependency Parsing with HPSG”
  4. The same dataset has been used for the domain adaptation track of the CoNLL 2007 Shared Task.
    Page 5, “Experiment Results & Error Analyses”
  5. This is the other datasets used in the domain adaptation track of the CoNLL 2007 Shared Task.
    Page 5, “Experiment Results & Error Analyses”
  6. It should be noted that domain adaptation also presents a challenge to the disambiguation model of the HP SG parser.
    Page 6, “Experiment Results & Error Analyses”
  7. Since our work focuses on the domain adaptation , we manually compare the outputs of the original statistical models, the dependency backbone, and the feature-based models on the out-domain data, i.e.
    Page 7, “Experiment Results & Error Analyses”

See all papers in Proc. ACL 2009 that mention domain adaptation.

See all papers in Proc. ACL that mention domain adaptation.

Back to top.

Penn Treebank

Appears in 7 sentences as: Penn Treebank (7)
In Cross-Domain Dependency Parsing Using a Deep Linguistic Grammar
  1. the Wall Street Journal (WSJ) sections of the Penn Treebank (Marcus et al., 1993) as training set, tests on BROWN Sections typically result in a 6-8% drop in labeled attachment scores, although the average sentence length is much shorter in BROWN than that in WSJ.
    Page 1, “Introduction”
  2. Note that all grammar rules in ERG are either unary or binary, giving us relatively deep trees when compared with annotations such as Penn Treebank .
    Page 3, “Dependency Parsing with HPSG”
  3. For these rules, we refer to the conversion of the Penn Treebank into dependency structures used in the CoNLL 2008 Shared Task, and mark the heads of these rules in a way that will arrive at a compatible dependency backbone.
    Page 3, “Dependency Parsing with HPSG”
  4. 2More recent study shows that with carefully designed retokenization and preprocessing rules, over 80% sentential coverage can be achieved on the WSJ sections of the Penn Treebank data using the same version of ERG.
    Page 4, “Dependency Parsing with HPSG”
  5. The larger part is converted from the Penn Treebank Wall Street Journal Sections #2—#21, and is used for training statistical dependency parsing models; the smaller part, which covers sentences from Section #23, is used for testing.
    Page 5, “Experiment Results & Error Analyses”
  6. Brown This dataset contains a subset of converted sentences from BROWN sections of the Penn Treebank .
    Page 5, “Experiment Results & Error Analyses”
  7. Although the original annotation scheme is similar to the Penn Treebank , the dependency extraction setting is slightly different to the CoNLLWSJ dependencies (e.g.
    Page 5, “Experiment Results & Error Analyses”

See all papers in Proc. ACL 2009 that mention Penn Treebank.

See all papers in Proc. ACL that mention Penn Treebank.

Back to top.

in-domain

Appears in 6 sentences as: in-domain (6)
In Cross-Domain Dependency Parsing Using a Deep Linguistic Grammar
  1. Pure statistical parsing systems achieves high in-domain accuracy but performs poorly out-domain.
    Page 1, “Abstract”
  2. With the extra features, we hope that the training of the statistical model will not overfit the in-domain data, but be able to deal with domain independent linguistic phenomena as well.
    Page 5, “Dependency Parsing with HPSG”
  3. With both parsers, we see slight performance drops with both HP SG feature models on in-domain tests (WSJ), compared with the original models.
    Page 7, “Experiment Results & Error Analyses”
  4. When we look at the performance difference between in-domain and out-domain tests for each feature model, we observe that the drop is significantly smaller for the extended models with HP SG features.
    Page 7, “Experiment Results & Error Analyses”
  5. Admittedly the results on PCHEMTB are lower than the best reported results in CoNLL 2007 Shared Task, we shall note that we are not using any in-domain unlabeled data.
    Page 7, “Experiment Results & Error Analyses”
  6. Nevertheless, the drops when compared to in-domain tests are constantly decreased with the help of HP SG analyses features.
    Page 7, “Experiment Results & Error Analyses”

See all papers in Proc. ACL 2009 that mention in-domain.

See all papers in Proc. ACL that mention in-domain.

Back to top.

feature set

Appears in 4 sentences as: Feature Set (1) feature set (3)
In Cross-Domain Dependency Parsing Using a Deep Linguistic Grammar
  1. Therefore, we extend this feature set by adding four more feature categories, which are similar to the original ones, but the dependency relation was replaced by the dependency backbone of the HP S G outputs.
    Page 4, “Dependency Parsing with HPSG”
  2. The extended feature set is shown in Table 1.
    Page 4, “Dependency Parsing with HPSG”
  3. The extended feature set is shown in Table 2 (the new features are listed separately).
    Page 4, “Dependency Parsing with HPSG”
  4. Table 1: The Extra Feature Set for MSTParser.
    Page 5, “Dependency Parsing with HPSG”

See all papers in Proc. ACL 2009 that mention feature set.

See all papers in Proc. ACL that mention feature set.

Back to top.

machine learning

Appears in 4 sentences as: machine learning (4)
In Cross-Domain Dependency Parsing Using a Deep Linguistic Grammar
  1. In combination with machine learning methods, several statistical dependency parsing models have reached comparable high parsing accuracy (McDonald et al., 2005b; Nivre et al., 2007b).
    Page 1, “Introduction”
  2. In recent years, two statistical dependency parsing systems, MaltParser (Nivre et al., 2007b) and MS TParser (McDonald et al., 2005b), representing different threads of research in data-driven machine learning approaches have obtained high publicity, for their state-of-the-art performances in open competitions such as CoNLL Shared Tasks.
    Page 1, “Parser Domain Adaptation”
  3. Granted for the differences between their approaches, both systems heavily rely on machine learning methods to estimate the parsing model from an annotated corpus as training set.
    Page 2, “Parser Domain Adaptation”
  4. Most of these approaches focused on the machine learning perspective instead of the linguistic knowledge embraced in the parsers.
    Page 2, “Parser Domain Adaptation”

See all papers in Proc. ACL 2009 that mention machine learning.

See all papers in Proc. ACL that mention machine learning.

Back to top.

UAS

Appears in 4 sentences as: UAS (4)
In Cross-Domain Dependency Parsing Using a Deep Linguistic Grammar
  1. Table 3 shows the agreement between the HP SG backbone and CoNLL dependency in unlabeled attachment score ( UAS ).
    Page 6, “Experiment Results & Error Analyses”
  2. UAS are reported on all complete test sets, as well as fully parsed subsets (suffixed with “-p”>.
    Page 6, “Experiment Results & Error Analyses”
  3. Most notable is that the dependency backbone achieved over 80% UAS on BROWN, which is close to the performance of state-of-the-art statistical dependency parsing systems trained on WSJ (see Table 5 and Table 4).
    Page 6, “Experiment Results & Error Analyses”
  4. For unlabeled CHILDES* data, only UAS numbers are reported.
    Page 7, “Experiment Results & Error Analyses”

See all papers in Proc. ACL 2009 that mention UAS.

See all papers in Proc. ACL that mention UAS.

Back to top.

dependency relation

Appears in 3 sentences as: dependency relation (3)
In Cross-Domain Dependency Parsing Using a Deep Linguistic Grammar
  1. The dependency backbone extraction works by first identifying the head daughter for each binary grammar rule, and then propagating the head word of the head daughter upwards to their parents, and finally creating a dependency relation , labeled with the HP SG rule name of the parent node, from the head word of the parent to the head word of the non-head daughter.
    Page 3, “Dependency Parsing with HPSG”
  2. Therefore, we extend this feature set by adding four more feature categories, which are similar to the original ones, but the dependency relation was replaced by the dependency backbone of the HP S G outputs.
    Page 4, “Dependency Parsing with HPSG”
  3. unlabeled dependency relation ), fine-grained HP SG features do help the parser to deal with colloquial sentences, such as “What’s wrong with you?”.
    Page 7, “Experiment Results & Error Analyses”

See all papers in Proc. ACL 2009 that mention dependency relation.

See all papers in Proc. ACL that mention dependency relation.

Back to top.

PDF available from www.aclweb.org.
Send us an email