Robustness and Generalization of Role Sets: PropBank vs. VerbNet
Zapirain, Beñat and Agirre, Eneko and Màrquez, Llu'is

Article Structure

Abstract

This paper presents an empirical study on the robustness and generalization of two alternative role sets for semantic role labeling: PropBank numbered roles and VerbNet thematic roles.

Introduction

Semantic Role Labeling is the problem of analyzing clause predicates in open text by identifying arguments and tagging them with semantic labels indicating the role they play with respect to the verb.

Corpora and Semantic Role Sets

The PropBank corpus is the result of adding a semantic layer to the syntactic structures of Penn Tree-bank 11 (Palmer et al., 2005).

Experimental Setting 3.1 Datasets

The data used in this work is the benchmark corpus provided by the SRL shared task of CoNLL-2005 (Carreras and Marquez, 2005).

On the Generalization of Role Sets

We first seek a basic reference of the comparative performance of the classifier on each role set.

Mapping into VerbNet Thematic Roles

As mentioned in the introduction, the interpretation of PropBank roles depends on the verb, and that

Related Work

As far as we know, there are only two other works performing comparisons of alternative role sets on a common test data.

Conclusion and Future work

In this paper we have presented a study of the performance of a state-of-the-art SRL system trained on two alternative codifications of roles (PropBank and VerbNet) and some particular settings, e.g., in-cluding/excluding verb—specific information in features, labeling of infrequent and unseen verb predicates, and domain shifts.

Topics

CoNLL

Appears in 13 sentences as: CoNLL (11) ‘CoNLL (2) ‘CoNLL’ (1)
In Robustness and Generalization of Role Sets: PropBank vs. VerbNet
  1. Being aware that, in a real scenario, the sense information will not be available, we devised the second setting ( ‘CoNLL’ ), where the hand-annotated verb sense information was discarded.
    Page 4, “On the Generalization of Role Sets”
  2. This is the setting used in the CoNLL 2005 shared task (Carreras and Marquez, 2005).
    Page 4, “On the Generalization of Role Sets”
  3. In the second setting ( ‘CoNLL setting’ row in the same table) the PropBank classifier degrades slightly, but the difference is not statistically significant.
    Page 4, “On the Generalization of Role Sets”
  4. In order to test this hypothesis, we run the CoNLL setting with the 5th constraint disabled (that is, allowing any argument).
    Page 4, “On the Generalization of Role Sets”
  5. The results in the ‘CoNLL setting (no 5th)’ rows of Table 1 show that the drop for PropBank is negligible and not significant, while the drop for VerbNet is more important, and statistically significant.
    Page 4, “On the Generalization of Role Sets”
  6. Table 2 shows these results on the CoNLL setting.
    Page 4, “On the Generalization of Role Sets”
  7. CoNLL setting No verb features PBank VNet PBank VNet corr.
    Page 6, “On the Generalization of Role Sets”
  8. Table 2: Detailed results on the CoNLL setting.
    Page 6, “On the Generalization of Role Sets”
  9. The last two columns refer to the results on the CoNLL setting with no verb features.
    Page 6, “On the Generalization of Role Sets”
  10. A closer look at the detailed role-by-role performances can be done if we compare the F1 rows in the CoNLL setting and in the ‘no verb features’ setting in Table 2.
    Page 6, “On the Generalization of Role Sets”
  11. PropBank to VerbNet (hand) 79.17 :|:0.9 81.77 72.50 VerbNet (SemEval setting) 78.61 :|:0.9 81.28 71.84 PropBank to VerbNet (MF) 77.15 :|:0.9 79.09 71.90 VerbNet (CoNLL setting) 76.99 :|:0.9 79.44 70.88 Test on Brown PropB ank to VerbNet (MF) 64.79 :|:1.0 68.93 55.94 VerbNet ( CoNLL setting) 62.87 :|:1.0 67.07 54.69
    Page 7, “Mapping into VerbNet Thematic Roles”

See all papers in Proc. ACL 2008 that mention CoNLL.

See all papers in Proc. ACL that mention CoNLL.

Back to top.

role labels

Appears in 6 sentences as: Role Labeling (2) role labeling (1) role labels (3)
In Robustness and Generalization of Role Sets: PropBank vs. VerbNet
  1. This paper presents an empirical study on the robustness and generalization of two alternative role sets for semantic role labeling : PropBank numbered roles and VerbNet thematic roles.
    Page 1, “Abstract”
  2. Semantic Role Labeling is the problem of analyzing clause predicates in open text by identifying arguments and tagging them with semantic labels indicating the role they play with respect to the verb.
    Page 1, “Introduction”
  3. Second, assuming that application scenarios would prefer dealing with general thematic role labels , we explore the best way to label a text with thematic roles, namely, by training directly on VerbNet roles or by using the PropBank SRL system and perform a posterior mapping into thematic roles.
    Page 2, “Introduction”
  4. Each verb has a frameset listing its allowed role labels and mapping each numbered role to an English-language description of its semantics.
    Page 2, “Corpora and Semantic Role Sets”
  5. Our basic Semantic Role Labeling system represents the tagging problem as a Maximum Entropy Markov Model (MEMM).
    Page 3, “Experimental Setting 3.1 Datasets”
  6. Assuming that application-based scenarios would prefer dealing with general thematic role labels , we explore the best way to label a text with VerbNet thematic roles, namely, by training directly on VerbNet roles or by using the PropBank SRL system and performing a posterior mapping into thematic roles.
    Page 8, “Conclusion and Future work”

See all papers in Proc. ACL 2008 that mention role labels.

See all papers in Proc. ACL that mention role labels.

Back to top.

statistically significant

Appears in 6 sentences as: statistically significant (7)
In Robustness and Generalization of Role Sets: PropBank vs. VerbNet
  1. In the second setting (‘CoNLL setting’ row in the same table) the PropBank classifier degrades slightly, but the difference is not statistically significant .
    Page 4, “On the Generalization of Role Sets”
  2. The results in the ‘CoNLL setting (no 5th)’ rows of Table 1 show that the drop for PropBank is negligible and not significant, while the drop for VerbNet is more important, and statistically significant .
    Page 4, “On the Generalization of Role Sets”
  3. If we compare these results to those obtained by VerbNet in the SemEval setting (second row of Table 5), they are 0.5 points better, but the difference is not statistically significant .
    Page 7, “Mapping into VerbNet Thematic Roles”
  4. The performance drop compared to the use of the hand-annotated VerbNet class is of 2 points and statistically significant , and 0.2 points above the results obtained using VerbNet directly on the same conditions (fourth row of the same Table).
    Page 7, “Mapping into VerbNet Thematic Roles”
  5. In this case, the difference is larger, 1.9 points, and statistically significant in favor of the mapping approach.
    Page 7, “Mapping into VerbNet Thematic Roles”
  6. While results are similar and not statistically significant in the WSJ test set, when testing on the Brown out—of—domain test set the difference in favor of PropBank plus mapping step is statistically significant .
    Page 8, “Conclusion and Future work”

See all papers in Proc. ACL 2008 that mention statistically significant.

See all papers in Proc. ACL that mention statistically significant.

Back to top.

Semantic Role

Appears in 4 sentences as: Semantic Role (2) semantic role (1) semantic roles (1)
In Robustness and Generalization of Role Sets: PropBank vs. VerbNet
  1. This paper presents an empirical study on the robustness and generalization of two alternative role sets for semantic role labeling: PropBank numbered roles and VerbNet thematic roles.
    Page 1, “Abstract”
  2. Semantic Role Labeling is the problem of analyzing clause predicates in open text by identifying arguments and tagging them with semantic labels indicating the role they play with respect to the verb.
    Page 1, “Introduction”
  3. While Arg0 and Argl are intended to indicate the general roles of Agent and Theme, other argument numbers do not generalize across verbs and do not correspond to general semantic roles .
    Page 1, “Introduction”
  4. Our basic Semantic Role Labeling system represents the tagging problem as a Maximum Entropy Markov Model (MEMM).
    Page 3, “Experimental Setting 3.1 Datasets”

See all papers in Proc. ACL 2008 that mention Semantic Role.

See all papers in Proc. ACL that mention Semantic Role.

Back to top.

fine-grained

Appears in 3 sentences as: fine-grained (3)
In Robustness and Generalization of Role Sets: PropBank vs. VerbNet
  1. In the case of VerbNet, the more fine-grained distinction among roles seems to depend more on the meaning of the predicate.
    Page 6, “On the Generalization of Role Sets”
  2. But if we compare them to the results of the PropBank to VerbNet mapping, where we simply substitute the fine-grained roles by their corresponding groups, we see that they still lag behind (second row in Table 6).
    Page 8, “Mapping into VerbNet Thematic Roles”
  3. We also tried to map the fine-grained VerbNet roles into coarser roles, but it did not yield better results than the mapping from PropBank roles.
    Page 8, “Conclusion and Future work”

See all papers in Proc. ACL 2008 that mention fine-grained.

See all papers in Proc. ACL that mention fine-grained.

Back to top.

Semantic Role Labeling

Appears in 3 sentences as: Semantic Role Labeling (2) semantic role labeling (1)
In Robustness and Generalization of Role Sets: PropBank vs. VerbNet
  1. This paper presents an empirical study on the robustness and generalization of two alternative role sets for semantic role labeling : PropBank numbered roles and VerbNet thematic roles.
    Page 1, “Abstract”
  2. Semantic Role Labeling is the problem of analyzing clause predicates in open text by identifying arguments and tagging them with semantic labels indicating the role they play with respect to the verb.
    Page 1, “Introduction”
  3. Our basic Semantic Role Labeling system represents the tagging problem as a Maximum Entropy Markov Model (MEMM).
    Page 3, “Experimental Setting 3.1 Datasets”

See all papers in Proc. ACL 2008 that mention Semantic Role Labeling.

See all papers in Proc. ACL that mention Semantic Role Labeling.

Back to top.

shared task

Appears in 3 sentences as: shared task (3)
In Robustness and Generalization of Role Sets: PropBank vs. VerbNet
  1. The data used in this work is the benchmark corpus provided by the SRL shared task of CoNLL-2005 (Carreras and Marquez, 2005).
    Page 3, “Experimental Setting 3.1 Datasets”
  2. The system achieves very good performance in the CoNLL-2005 shared task dataset and in the SRL subtask of the SemEval-2007 English lexical sample task (Zapirain et al., 2007).
    Page 3, “Experimental Setting 3.1 Datasets”
  3. This is the setting used in the CoNLL 2005 shared task (Carreras and Marquez, 2005).
    Page 4, “On the Generalization of Role Sets”

See all papers in Proc. ACL 2008 that mention shared task.

See all papers in Proc. ACL that mention shared task.

Back to top.

PDF available from www.aclweb.org.
Send us an email