Validating and Extending Semantic Knowledge Bases using Video Games with a Purpose
Vannella, Daniele and Jurgens, David and Scarfini, Daniele and Toscani, Domenico and Navigli, Roberto

Article Structure

Abstract

Large-scale knowledge bases are important assets in NLP.

Introduction

Large-scale knowledge bases are an essential component of many approaches in Natural Language Processing (NLP).

Related Work

Multiple works have proposed linguistic annotation-based games with a purpose for tasks such as anaphora resolution (Hladka et al., 2009; Poesio et al., 2013), paraphrasing (Chklovski and Gil, 2005), term associations (Artignan et al., 2009; Lafourcade and Joubert, 2010), query expansion (Simko et al., 2011), and word sense disambiguation (Chklovski and Mi-halcea, 2002; Seemakurty et al., 2010; Venhuizen et al., 2013).

Video Game with a Purpose Design

To create video games, our development process focused on a common design philosophy and a common data set.

Game 1: Infection

The first game, Infection, validates the concept-concept relation dataset.

Game 2: The Knowledge Towers

The second game, The Knowledge Towers (TKT), validates the concept-image dataset.

Experiments

Two experiments were performed with Infection and TKT: (1) an evaluation of players’ ability to play accurately and to validate semantic relations and image associations and (2) a comprehensive cost comparison.

Results and Discussion

7.1 Gameplay Analysis

Conclusion

Two video games have been presented for validating and extending knowledge bases.

Topics

knowledge bases

Appears in 16 sentences as: Knowledge base (1) knowledge base (2) knowledge bases (14)
In Validating and Extending Semantic Knowledge Bases using Video Games with a Purpose
  1. Large-scale knowledge bases are important assets in NLP.
    Page 1, “Abstract”
  2. We propose a cost-effective method of validating and extending knowledge bases using video games with a purpose.
    Page 1, “Abstract”
  3. Large-scale knowledge bases are an essential component of many approaches in Natural Language Processing (NLP).
    Page 1, “Introduction”
  4. Semantic knowledge bases such as WordNet (Fellbaum, 1998), YAGO (Suchanek et al., 2007), and BabelNet (Navigli and Ponzetto, 2010) provide ontological structure that enables a wide range of tasks, such as measuring semantic relatedness (Budanitsky and Hirst, 2006) and similarity (Pilehvar et al., 2013), paraphrasing (Kauchak and Barzilay, 2006), and word sense disambiguation (Navigli and Ponzetto, 2012; Moro et al., 2014).
    Page 1, “Introduction”
  5. Furthermore, such knowledge bases are essential for building unsupervised algorithms when training data is sparse or unavailable.
    Page 1, “Introduction”
  6. However, constructing and updating semantic knowledge bases is often limited by the significant time and human resources required.
    Page 1, “Introduction”
  7. Recent approaches have attempted to build or extend these knowledge bases automatically.
    Page 1, “Introduction”
  8. The recent advent of large semistructured resources has enabled the creation of new semantic knowledge bases (Medelyan et al., 2009; Hovy et al., 2013) through automatically merging WordNet and Wikipedia (Suchanek et al., 2007; Navigli and Ponzetto, 2010; Nie-mann and Gurevych, 2011).
    Page 1, “Introduction”
  9. To overcome issues from fully-automatic construction methods, several works have proposed validating or extending knowledge bases using crowdsourcing (Biemann and Nygaard, 2010; Born et al., 2012; Sarasua et al., 2012).
    Page 1, “Introduction”
  10. In this paper, we propose validating and extending semantic knowledge bases using video games with a purpose.
    Page 1, “Introduction”
  11. Last, three two-player games have focused on validating and extending knowledge bases .
    Page 2, “Related Work”

See all papers in Proc. ACL 2014 that mention knowledge bases.

See all papers in Proc. ACL that mention knowledge bases.

Back to top.

gold standard

Appears in 9 sentences as: Gold Standard (1) Gold standard (1) gold standard (9)
In Validating and Extending Semantic Knowledge Bases using Video Games with a Purpose
  1. Second, the game should be playable by a single player, with reinforcement for correct game play coming from gold standard examples.1 We note that gold standard examples may come from both true positive and true negative items.
    Page 3, “Video Game with a Purpose Design”
  2. For both datasets, each negative set NC is constructed as UC/EC\{C} V05, i.e., from the items related in BabelNet to all other concepts in C. By constructing NC directly from the knowledge base, play actions may be validated based on recognition of true negatives, removing the heavy burden for ever manually creating a gold standard test set.
    Page 3, “Video Game with a Purpose Design”
  3. Gold Standard Data To compare the quality of annotation from games and crowdsourcing, a gold standard annotation was produced for a 10% sample of each dataset (cf.
    Page 5, “Experiments”
  4. To measure inter-annotator agreement (IAA) on the gold standard annotations, we calculated Krip-
    Page 5, “Experiments”
  5. Gold standard annotators had high agreement, 0.774, for concept-concept relations.
    Page 6, “Experiments”
  6. This section assesses the annotation quality of both games and of CrowdFlower in terms of (1) the IAA of the participants, measured using Krip-pendorff’s 04, and (2) the percentage agreement of the resulting annotations with the gold standard .
    Page 7, “Results and Discussion”
  7. Therefore, in Table l, we calculate the percentage agreement of the aggregated annotations with the gold standard annotations for approving valid relations (true positives; Col. 5), rejecting invalid relations (true negatives; Col. 6), and for both combined (Col. 7).
    Page 8, “Results and Discussion”
  8. Indeed, despite having lower IAA for images, the free version of TKT provides an absolute 16.3% improvement in gold standard agreement over crowdsourcing.
    Page 8, “Results and Discussion”
  9. Based on agreement with the gold standard (Table 1, Col. 5), the estimated cost for crowdsourcing a correct true positive annotation increases to $0.014 for a concept-image and a $0.048 for concepts-concept annotation.
    Page 9, “Results and Discussion”

See all papers in Proc. ACL 2014 that mention gold standard.

See all papers in Proc. ACL that mention gold standard.

Back to top.

WordNet

Appears in 8 sentences as: WordNet (8)
In Validating and Extending Semantic Knowledge Bases using Video Games with a Purpose
  1. Frequently, such resources are constructed through automatic mergers of complementary resources, such as WordNet and Wikipedia.
    Page 1, “Abstract”
  2. Semantic knowledge bases such as WordNet (Fellbaum, 1998), YAGO (Suchanek et al., 2007), and BabelNet (Navigli and Ponzetto, 2010) provide ontological structure that enables a wide range of tasks, such as measuring semantic relatedness (Budanitsky and Hirst, 2006) and similarity (Pilehvar et al., 2013), paraphrasing (Kauchak and Barzilay, 2006), and word sense disambiguation (Navigli and Ponzetto, 2012; Moro et al., 2014).
    Page 1, “Introduction”
  3. extend WordNet using distributional or structural features to identify novel semantic connections between concepts.
    Page 1, “Introduction”
  4. The recent advent of large semistructured resources has enabled the creation of new semantic knowledge bases (Medelyan et al., 2009; Hovy et al., 2013) through automatically merging WordNet and Wikipedia (Suchanek et al., 2007; Navigli and Ponzetto, 2010; Nie-mann and Gurevych, 2011).
    Page 1, “Introduction”
  5. Rzeniewicz and Szymanski (2013) extend WordNet with commonsense knowledge using a 20 Questions-like game.
    Page 2, “Related Work”
  6. Knowledge base As the reference knowledge base, we chose BabelNet2 (Navigli and Ponzetto, 2010), a large-scale multilingual semantic ontology created by automatically merging WordNet with other collaboratively-constructed resources such as Wikipedia and OmegaWiki.
    Page 3, “Video Game with a Purpose Design”
  7. First, by connecting WordNet synsets to Wikipedia pages, most synsets are associated with a set of pictures; while often noisy, these pictures sometimes illustrate the target concept and are an ideal case for validation.
    Page 3, “Video Game with a Purpose Design”
  8. Second, BabelNet contains the semantic relations from both WordNet and hyperlinks in Wikipedia; these relations are again an ideal case of validation, as not all hyperlinks connect semantically-related pages in Wikipedia.
    Page 3, “Video Game with a Purpose Design”

See all papers in Proc. ACL 2014 that mention WordNet.

See all papers in Proc. ACL that mention WordNet.

Back to top.

synsets

Appears in 5 sentences as: synset (1) synsets (6)
In Validating and Extending Semantic Knowledge Bases using Video Games with a Purpose
  1. First, by connecting WordNet synsets to Wikipedia pages, most synsets are associated with a set of pictures; while often noisy, these pictures sometimes illustrate the target concept and are an ideal case for validation.
    Page 3, “Video Game with a Purpose Design”
  2. Data We created a common set of concepts, 0, used in both games, containing sixty synsets selected from all BabelNet synsets with at least fifty associated images.
    Page 3, “Video Game with a Purpose Design”
  3. Using the same set of synsets , separate datasets were created for the two validation tasks.
    Page 3, “Video Game with a Purpose Design”
  4. For the concept-concept dataset, V0 is the union of VCB, which contains the lemmas of all synsets incident to c in BabelNet, and V0”, which contains novel lemmas derived from statistical associations.
    Page 3, “Video Game with a Purpose Design”
  5. A task begins with a description of a target synset and its textual definition; following, ten annotation questions are shown.
    Page 6, “Experiments”

See all papers in Proc. ACL 2014 that mention synsets.

See all papers in Proc. ACL that mention synsets.

Back to top.

semantic relations

Appears in 4 sentences as: semantic relatedness (1) semantic relations (3)
In Validating and Extending Semantic Knowledge Bases using Video Games with a Purpose
  1. Semantic knowledge bases such as WordNet (Fellbaum, 1998), YAGO (Suchanek et al., 2007), and BabelNet (Navigli and Ponzetto, 2010) provide ontological structure that enables a wide range of tasks, such as measuring semantic relatedness (Budanitsky and Hirst, 2006) and similarity (Pilehvar et al., 2013), paraphrasing (Kauchak and Barzilay, 2006), and word sense disambiguation (Navigli and Ponzetto, 2012; Moro et al., 2014).
    Page 1, “Introduction”
  2. semantic networks, using two games that operate on complementary sources of information: semantic relations and sense-image mappings.
    Page 2, “Introduction”
  3. Second, BabelNet contains the semantic relations from both WordNet and hyperlinks in Wikipedia; these relations are again an ideal case of validation, as not all hyperlinks connect semantically-related pages in Wikipedia.
    Page 3, “Video Game with a Purpose Design”
  4. Two experiments were performed with Infection and TKT: (1) an evaluation of players’ ability to play accurately and to validate semantic relations and image associations and (2) a comprehensive cost comparison.
    Page 5, “Experiments”

See all papers in Proc. ACL 2014 that mention semantic relations.

See all papers in Proc. ACL that mention semantic relations.

Back to top.

PDF available from www.aclweb.org.
Send us an email