Brutus: A Semantic Role Labeling System Incorporating CCG, CFG, and Dependency Features
Article Structure
Abstract
We describe a semantic role labeling system that makes primary use of CCG-based features.
Introduction
Semantic Role Labeling (SRL) is the process of assigning semantic roles to strings of words in a sentence according to their relationship to the semantic predicates expressed in the sentence.
Combinatory Categorial Grammar
Combinatory Categorial Grammar (Steedman, 2000) is a grammatical framework that describes syntactic structure in terms of the combinatory potential of the lexical (word-level) items.
Potential Advantages to using CCG
There are many potential advantages to using the CCG formalism in SRL.
Identification and Labeling Models
As in previous approaches to SRL, Brutus uses a two-stage pipeline of maximum entropy classifiers.
This is easily read off of the CCG PARG relationships.
(10)
Dependency Parser Features
Finally, several features can be extracted from a dependency representation of the same sentence.
Argument Mapping Model
An innovation in our approach is to use a separate classifier to predict an argument mapping feature.
Enabling Cross-System Comparison
The Brutus system is designed to label headwords of semantic roles, rather than entire constituents.
Results
Using a version of Brutus incorporating only the CCG-based features described above, we achieve better results than a previous CCG based system (Gildea and Hockenmaier, 2003, henceforth G&H).
The Contribution of the New Features
Two features which are less frequently used in SRL research play a major role in the Brutus system: The PARG feature (Gildea and Hockenmaier, 2003) and the argument mapping feature.
Error Analysis
Many of the errors made by the Brutus system can be traced directly to erroneous parses, either in the automatic or treebank parse.
Future Work
As described in the error analysis section, a large number of errors in the system are attributable to errors in the CCG derivation, either in the gold standard or in automatically generated parses.
Acknowledgments
This research was funded by NSF grant IIS-0347799.
Topics
CCG
Appears in 39 sentences as: CCG (41)
In Brutus: A Semantic Role Labeling System Incorporating CCG, CFG, and Dependency Features
- CCG affords ways to augment treepath-based features to overcome these data sparsity issues.Page 1, “Abstract”
- By adding features over CCG word-word dependencies and lexicalized verbal subcategorization frames (“supertags”), we can obtain an F-score that is substantially better than a previous CCG-based SRL system and competitive with the current state of the art.Page 1, “Abstract”
- Brutus uses the CCG parser of (Clark and Curran, 2007, henceforth the C&C parser), Charniak’s parser (Chamiak, 2001) for additional CFG-based features, and MALT parser (Nivre et al., 2007) for dependency features, while (Punyakanok et al., 2008) use results from an ensemble of parses from Charniak’s Parser and a Collins parser (Collins, 2003; Bike], 2004).Page 1, “Introduction”
- We do not employ a similar strategy due to the differing notions of constituency represented in our parsers ( CCG having a much more fluid notion of constituency and the MALT parser using a different approach entirely).Page 1, “Introduction”
- In the following, we briefly introduce the CCG grammatical formalism and motivate its use in SRL (Sections 2—3).Page 1, “Introduction”
- Our main contribution is to demonstrate that CCG — arguably a more expressive and lin—Page 1, “Introduction”
- In particular, using CCG enables us to map semantic roles directly onto verbal categories, an innovation of our approach that leads to performance gains (Section 7).Page 2, “Introduction”
- We conclude with an error analysis (Section 11), which motivates our discussion of future research for computational semantics with CCG (Section 12).Page 2, “Introduction”
- Rather than using standard part-of-speech tags and grammatical rules, CCG encodes much of the combinatory potential of each word by assigning a syntactically informative category.Page 2, “Combinatory Categorial Grammar”
- Further, CCG has the advantage of a transparent interface between the way the words combine and their dependencies with other words.Page 2, “Combinatory Categorial Grammar”
- There are many potential advantages to using the CCG formalism in SRL.Page 2, “Potential Advantages to using CCG”
See all papers in Proc. ACL 2009 that mention CCG.
See all papers in Proc. ACL that mention CCG.
Back to top.
semantic role
Appears in 25 sentences as: Semantic Role (1) semantic role (15) semantic roles (12)
In Brutus: A Semantic Role Labeling System Incorporating CCG, CFG, and Dependency Features
- We describe a semantic role labeling system that makes primary use of CCG-based features.Page 1, “Abstract”
- This analysis also suggests that simultaneous incremental parsing and semantic role labeling may lead to performance gains in both tasks.Page 1, “Abstract”
- Semantic Role Labeling (SRL) is the process of assigning semantic roles to strings of words in a sentence according to their relationship to the semantic predicates expressed in the sentence.Page 1, “Introduction”
- An effective semantic role labeling system must recognize the differences between different configurations:Page 1, “Introduction”
- We use Propbank (Palmer et al., 2005), a corpus of newswire text annotated with verb predicate semantic role information that is widely used in the SRL literature (Marquez et al., 2008).Page 1, “Introduction”
- Rather than describe semantic roles in terms of “agent” or “patient”, Propbank defines semantic roles on a verb-by-verb basis.Page 1, “Introduction”
- During identification, every word in the sentence is labeled either as bearing some (as yet undetermined) semantic role or not .Page 1, “Introduction”
- In particular, using CCG enables us to map semantic roles directly onto verbal categories, an innovation of our approach that leads to performance gains (Section 7).Page 2, “Introduction”
- We will show this to be a valuable tool for semantic role prediction.Page 2, “Combinatory Categorial Grammar”
- An argument mapping is a link between the CCG category and the semantic roles that are likely to go with each of its arguments.Page 2, “Potential Advantages to using CCG”
- some of the non-modifier semantic roles that a verb is likely to express.Page 4, “This is easily read off of the CCG PARG relationships.”
See all papers in Proc. ACL 2009 that mention semantic role.
See all papers in Proc. ACL that mention semantic role.
Back to top.
treebank
Appears in 13 sentences as: Treebank (4) treebank (12)
In Brutus: A Semantic Role Labeling System Incorporating CCG, CFG, and Dependency Features
- The same features are extracted for both treebank and automatic parses.Page 2, “Identification and Labeling Models”
- For gold-standard parses, we remove functional tag and trace information from the Penn Treebank parses before we extract features over them, so as to simulate the conditions of an automatic parse.Page 4, “This is easily read off of the CCG PARG relationships.”
- The Penn Treebank features are as follows:Page 4, “This is easily read off of the CCG PARG relationships.”
- By examining the arguments that the verbal category combines with in the treebank , we can identify the corresponding semantic role for each argument that is marked on the verbal category.Page 5, “Argument Mapping Model”
- | P | R I F G&H (treebank) 67.5% 60.0% 63.5% Brutus ( treebank ) 88.18% 85.00% 86.56%Page 6, “Enabling Cross-System Comparison”
- l P l R | F P. et al (treebank) 86.22% 87.40% 86.81% Brutus ( treebank ) 88.29% 86.39% 87.33%Page 6, “Results”
- Headword ( treebank ) 88.94% 86.98% 87.95%Page 6, “Results”
- Boundary ( treebank ) 88.29% 86.39% 87.33%Page 6, “Results”
- Removing them has a strong effect on accuracy when labeling treebank parses, as shown in our feature ablation results in table 4.Page 6, “The Contribution of the New Features”
- Many of the errors made by the Brutus system can be traced directly to erroneous parses, either in the automatic or treebank parse.Page 6, “Error Analysis”
- However, because in 1956 is erroneously modifying the verb using rather than the verb stopped in the treebank parse, the system trusts the syntactic analysis and places Argl of stopped on using asbestos in 1956.Page 7, “Error Analysis”
See all papers in Proc. ACL 2009 that mention treebank.
See all papers in Proc. ACL that mention treebank.
Back to top.
gold standard
Appears in 6 sentences as: gold standard (6)
In Brutus: A Semantic Role Labeling System Incorporating CCG, CFG, and Dependency Features
- Given these features with gold standard parses, our argument mapping model can predict entire argument mappings with an accuracy rate of 87.96% on the test set, and 87.70% on the development set.Page 5, “Argument Mapping Model”
- The results for gold standard parses are comparable to the winning system of the CoNLL 2005 shared task on semantic role labeling (Punyakanok et al., 2008).Page 6, “Results”
- Table 4: The effects of removing key features from the system on gold standard parses.Page 6, “The Contribution of the New Features”
- The gold standard CCG parse attaches the relative clause to a form of asbestos (figure 8).Page 7, “Error Analysis”
- In the phrase a group of workers exposed to asbestos (figure 10), the gold standard CCG parse attaches the relative clause to workers.Page 7, “Error Analysis”
- As described in the error analysis section, a large number of errors in the system are attributable to errors in the CCG derivation, either in the gold standard or in automatically generated parses.Page 7, “Future Work”
See all papers in Proc. ACL 2009 that mention gold standard.
See all papers in Proc. ACL that mention gold standard.
Back to top.
role labeling
Appears in 5 sentences as: Role Labeling (1) role labeling (4)
In Brutus: A Semantic Role Labeling System Incorporating CCG, CFG, and Dependency Features
- We describe a semantic role labeling system that makes primary use of CCG-based features.Page 1, “Abstract”
- This analysis also suggests that simultaneous incremental parsing and semantic role labeling may lead to performance gains in both tasks.Page 1, “Abstract”
- Semantic Role Labeling (SRL) is the process of assigning semantic roles to strings of words in a sentence according to their relationship to the semantic predicates expressed in the sentence.Page 1, “Introduction”
- An effective semantic role labeling system must recognize the differences between different configurations:Page 1, “Introduction”
- The results for gold standard parses are comparable to the winning system of the CoNLL 2005 shared task on semantic role labeling (Punyakanok et al., 2008).Page 6, “Results”
See all papers in Proc. ACL 2009 that mention role labeling.
See all papers in Proc. ACL that mention role labeling.
Back to top.
semantic role labeling
Appears in 5 sentences as: Semantic Role Labeling (1) semantic role labeling (4)
In Brutus: A Semantic Role Labeling System Incorporating CCG, CFG, and Dependency Features
- We describe a semantic role labeling system that makes primary use of CCG-based features.Page 1, “Abstract”
- This analysis also suggests that simultaneous incremental parsing and semantic role labeling may lead to performance gains in both tasks.Page 1, “Abstract”
- Semantic Role Labeling (SRL) is the process of assigning semantic roles to strings of words in a sentence according to their relationship to the semantic predicates expressed in the sentence.Page 1, “Introduction”
- An effective semantic role labeling system must recognize the differences between different configurations:Page 1, “Introduction”
- The results for gold standard parses are comparable to the winning system of the CoNLL 2005 shared task on semantic role labeling (Punyakanok et al., 2008).Page 6, “Results”
See all papers in Proc. ACL 2009 that mention semantic role labeling.
See all papers in Proc. ACL that mention semantic role labeling.
Back to top.
data sparsity
Appears in 3 sentences as: data sparsity (3)
In Brutus: A Semantic Role Labeling System Incorporating CCG, CFG, and Dependency Features
- Most previously developed systems are CFG-based and make extensive use of a treepath feature, which suffers from data sparsity due to its use of explicit tree configurations.Page 1, “Abstract”
- CCG affords ways to augment treepath-based features to overcome these data sparsity issues.Page 1, “Abstract”
- Because there are a number of different treepaths that correspond to a single relation (figure 2), this approach can suffer from data sparsity .Page 2, “Potential Advantages to using CCG”
See all papers in Proc. ACL 2009 that mention data sparsity.
See all papers in Proc. ACL that mention data sparsity.
Back to top.
development set
Appears in 3 sentences as: development set (3)
In Brutus: A Semantic Role Labeling System Incorporating CCG, CFG, and Dependency Features
- All classifiers were trained to 500 iterations of L-BFGS training — a quasi-Newton method from the numerical optimization literature (Liu and N o-cedal, 1989) — using Zhang Le’s maxent toolkit.2 To prevent overfitting we used Gaussian priors with global variances of l and 5 for the identifier and labeler, respectively.3 The Gaussian priors were determined empirically by testing on the development set .Page 2, “Identification and Labeling Models”
- 4The size of the window was determined experimentally on the development set — we use the same window sizes throughout.Page 2, “Identification and Labeling Models”
- Given these features with gold standard parses, our argument mapping model can predict entire argument mappings with an accuracy rate of 87.96% on the test set, and 87.70% on the development set .Page 5, “Argument Mapping Model”
See all papers in Proc. ACL 2009 that mention development set.
See all papers in Proc. ACL that mention development set.
Back to top.
gold-standard
Appears in 3 sentences as: gold-standard (3)
In Brutus: A Semantic Role Labeling System Incorporating CCG, CFG, and Dependency Features
- For gold-standard parses, we remove functional tag and trace information from the Penn Treebank parses before we extract features over them, so as to simulate the conditions of an automatic parse.Page 4, “This is easily read off of the CCG PARG relationships.”
- Problems with relative clause attachment to genitives are not limited to automatic parses — errors in gold-standard treebank parses cause similar problems when Treebank parses disagree with Propbank annotator intuitions.Page 7, “Error Analysis”
- Figure 8: CCGbank gold-standard parse of a relative clause attachment.Page 7, “Error Analysis”
See all papers in Proc. ACL 2009 that mention gold-standard.
See all papers in Proc. ACL that mention gold-standard.
Back to top.
lexicalized
Appears in 3 sentences as: lexicalist (1) lexicalized (2)
In Brutus: A Semantic Role Labeling System Incorporating CCG, CFG, and Dependency Features
- By adding features over CCG word-word dependencies and lexicalized verbal subcategorization frames (“supertags”), we can obtain an F-score that is substantially better than a previous CCG-based SRL system and competitive with the current state of the art.Page 1, “Abstract”
- Another advantage of a CCG-based approach (and lexicalist approaches in general) is the ability to encode verb-specific argument mappings.Page 2, “Potential Advantages to using CCG”
- We argue that, especially in the heaVily lexicalized CCG framework, headword evaluation is more appropriate, reflecting the emphasis on headword combinatorics in the CCG formalism.Page 6, “Results”
See all papers in Proc. ACL 2009 that mention lexicalized.
See all papers in Proc. ACL that mention lexicalized.
Back to top.
maximum entropy
Appears in 3 sentences as: maximum entropy (3)
In Brutus: A Semantic Role Labeling System Incorporating CCG, CFG, and Dependency Features
- For the identification and labeling steps, we train a maximum entropy classifier (Berger et al., 1996) over sections 02-21 of a version of the CCGbank corpus (Hockenmaier and Steedman, 2007) that has been augmented by projecting the Propbank semantic annotations (Boxwell and White, 2008).Page 1, “Introduction”
- As in previous approaches to SRL, Brutus uses a two-stage pipeline of maximum entropy classifiers.Page 2, “Identification and Labeling Models”
- 6G&H use a generative model with a back-off lattice, whereas we use a maximum entropy classifier.Page 6, “Results”
See all papers in Proc. ACL 2009 that mention maximum entropy.
See all papers in Proc. ACL that mention maximum entropy.
Back to top.
Penn Treebank
Appears in 3 sentences as: Penn Treebank (3)
In Brutus: A Semantic Role Labeling System Incorporating CCG, CFG, and Dependency Features
- For gold-standard parses, we remove functional tag and trace information from the Penn Treebank parses before we extract features over them, so as to simulate the conditions of an automatic parse.Page 4, “This is easily read off of the CCG PARG relationships.”
- The Penn Treebank features are as follows:Page 4, “This is easily read off of the CCG PARG relationships.”
- This particular problem is caused by an annotation error in the original Penn Treebank that was carried through in the conversion to CCGbank.Page 7, “Error Analysis”
See all papers in Proc. ACL 2009 that mention Penn Treebank.
See all papers in Proc. ACL that mention Penn Treebank.
Back to top.
