Solving Relational Similarity Problems Using the Web as a Corpus
Nakov, Preslav and Hearst, Marti A.

Article Structure

Abstract

We present a simple linguistically-motivated method for characterizing the semantic relations that hold between two nouns.

Introduction

Despite the tremendous amount of work on word similarity (see (Budanitsky and Hirst, 2006) for an overview), there is surprisingly little research on the important related problem of relational similarity —semantic similarity between pairs of words.

Related Work

2.1 Characterizing Semantic Relations

Method

Given a pair of nouns, we try to characterize the semantic relation between them by leveraging the vast size of the Web to build linguistically-motivated lexically-specific features.

Relational Similarity Experiments

4.1 SAT Verbal Analogy

Comparison to Human Judgments

Since in all above tasks the most important features were the verbs, we decided to compare our Web-derived verbs to human-proposed ones for all noun-noun compounds in the Levi-250 dataset.

Conclusions and Future Work

We have presented a simple approach for characterizing the relation between a pair of nouns in terms of linguistically-motivated features which could be useful for many NLP tasks.

Topics

WordNet

Appears in 10 sentences as: (1) WordNet (9) WordNet’s (1)
In Solving Relational Similarity Problems Using the Web as a Corpus
  1. (2005) apply both classic (SVM and decision trees) and novel supervised models (semantic scattering and iterative semantic specialization), using WordNet , word sense disambiguation, and a set of linguistic features.
    Page 2, “Related Work”
  2. Their approach is highly resource intensive (uses WordNet , CoreLex and Moby’s thesaurus), and is quite sensitive to the seed set of verbs: on a collection of 453 examples and 19 relations, they achieved 52.6% accuracy with 84 seed verbs, but only 46.7% with 57 seed verbs.
    Page 2, “Related Work”
  3. where: infll and inflg are inflected variants of nounl and noung generated using the Java WordNet Libraryl; THAT is a complementizer and can be that, which, or who; and * stands for 0 or more (up to 8) instances of Google’s star operator.
    Page 3, “Method”
  4. Finally, we lemmatize the main verb using WordNet’s morphological analyzer Morphy (Fellbaum, 1998).
    Page 3, “Method”
  5. We further experimented with the SemEval’07 task 4 dataset (Girju et al., 2007), where each example consists of a sentence, a target semantic relation, two nominals to be judged on whether they are in that relation, manually annotated WordNet senses, and the Web query used to obtain the sentence:
    Page 6, “Relational Similarity Experiments”
  6. WordNet(el) = "vessel%l:06:OO::", WordNet(e2) = "tool%l:O6:OO::", Content—Container(e2, el) = "true", Query = "contents of the * were a"
    Page 6, “Relational Similarity Experiments”
  7. The SemEval competition defines four types of systems, depending on whether the manually annotated WordNet senses and the Google query are used: A (WordNet=no, Query=no), B (WordNet=yes, Query=no), C (WordNet=no, Query=yes), and D (WordNet=yes, Query=yes).
    Page 6, “Relational Similarity Experiments”
  8. We experimented with types A and C only since we believe that having the manually annotated WordNet sense keys is an unrealistic assumption for a real-world application.
    Page 6, “Relational Similarity Experiments”
  9. 4The best type B system on SemEval achieved 76.3% accuracy using the manually-annotated WordNet senses in context for each example, which constitutes an additional data source, as opposed to an additional resource.
    Page 6, “Relational Similarity Experiments”
  10. The systems that used WordNet as a resource only, i.e., ignoring the manually annotated senses, were classified as type A or C. (Girju et a1., 2007)
    Page 6, “Relational Similarity Experiments”

See all papers in Proc. ACL 2008 that mention WordNet.

See all papers in Proc. ACL that mention WordNet.

Back to top.

statistically significant

Appears in 8 sentences as: statistically significant (8)
In Solving Relational Similarity Problems Using the Web as a Corpus
  1. Our best model 2) + p + c performs a bit better, 71.3% vs. 67.4%, but the difference is not statistically significant .
    Page 4, “Relational Similarity Experiments”
  2. Our best model achieves 40.5% accuracy, which is slightly better than LRA’s 39.8%, but the difference is not statistically significant .
    Page 5, “Relational Similarity Experiments”
  3. However, this time coordinating conjunctions (with prepositions) do help a bit (the difference is not statistically significant ) since SAT verbal analogy questions ask for a broader range of relations, e. g., antonymy, for which coordinating conjunctions like but are helpful.
    Page 5, “Relational Similarity Experiments”
  4. Both results represent a statistically significant improvement over the majority class baseline and over using sentence words only, and a slight improvement over the best type A and type 0 systems on SemEval ’07, which achieved 66% and 67% accuracy, respectively.4
    Page 6, “Relational Similarity Experiments”
  5. As we can see, using prepositions alone yields about 33% accuracy, which is a statistically significant improvement over the maj ority-class baseline.
    Page 7, “Relational Similarity Experiments”
  6. Note however that none of the differences between the different feature combinations involving verbs are statistically significant .
    Page 7, “Relational Similarity Experiments”
  7. We can observe a small l-3% drop in accuracy for all models involving verbs, but it is not statistically significant .
    Page 7, “Relational Similarity Experiments”
  8. As Table 6 shows, we achieved 78.4% accuracy using all verbs (and and 72.3% with the first verb from each worker), which is a statistically significant improve-
    Page 8, “Comparison to Human Judgments”

See all papers in Proc. ACL 2008 that mention statistically significant.

See all papers in Proc. ACL that mention statistically significant.

Back to top.

semantic relations

Appears in 7 sentences as: semantic relation (2) Semantic Relations (1) semantic relations (3) semantic relationship (1)
In Solving Relational Similarity Problems Using the Web as a Corpus
  1. We present a simple linguistically-motivated method for characterizing the semantic relations that hold between two nouns.
    Page 1, “Abstract”
  2. 2.1 Characterizing Semantic Relations
    Page 2, “Related Work”
  3. Turney (2006a) presents an unsupervised algorithm for mining the Web for patterns expressing implicit semantic relations .
    Page 2, “Related Work”
  4. They test their system against both Lauer’s 8 prepositional paraphrases and another set of 21 semantic relations , achieving up to 54% accuracy on the latter.
    Page 2, “Related Work”
  5. Kim and Baldwin (2006) characterized the semantic relationship in a noun-noun compound using the verbs connecting the two nouns by comparing them to predefined seed verbs.
    Page 2, “Related Work”
  6. Given a pair of nouns, we try to characterize the semantic relation between them by leveraging the vast size of the Web to build linguistically-motivated lexically-specific features.
    Page 3, “Method”
  7. We further experimented with the SemEval’07 task 4 dataset (Girju et al., 2007), where each example consists of a sentence, a target semantic relation , two nominals to be judged on whether they are in that relation, manually annotated WordNet senses, and the Web query used to obtain the sentence:
    Page 6, “Relational Similarity Experiments”

See all papers in Proc. ACL 2008 that mention semantic relations.

See all papers in Proc. ACL that mention semantic relations.

Back to top.

manually annotated

Appears in 4 sentences as: manually annotated (4)
In Solving Relational Similarity Problems Using the Web as a Corpus
  1. We further experimented with the SemEval’07 task 4 dataset (Girju et al., 2007), where each example consists of a sentence, a target semantic relation, two nominals to be judged on whether they are in that relation, manually annotated WordNet senses, and the Web query used to obtain the sentence:
    Page 6, “Relational Similarity Experiments”
  2. The SemEval competition defines four types of systems, depending on whether the manually annotated WordNet senses and the Google query are used: A (WordNet=no, Query=no), B (WordNet=yes, Query=no), C (WordNet=no, Query=yes), and D (WordNet=yes, Query=yes).
    Page 6, “Relational Similarity Experiments”
  3. We experimented with types A and C only since we believe that having the manually annotated WordNet sense keys is an unrealistic assumption for a real-world application.
    Page 6, “Relational Similarity Experiments”
  4. The systems that used WordNet as a resource only, i.e., ignoring the manually annotated senses, were classified as type A or C. (Girju et a1., 2007)
    Page 6, “Relational Similarity Experiments”

See all papers in Proc. ACL 2008 that mention manually annotated.

See all papers in Proc. ACL that mention manually annotated.

Back to top.

feature vector

Appears in 3 sentences as: feature vector (2) feature vectors (1)
In Solving Relational Similarity Problems Using the Web as a Corpus
  1. Given a verbal analogy example, we build six feature vectors — one for each of the six word pairs.
    Page 4, “Relational Similarity Experiments”
  2. For the evaluation, we created a feature vector for each head-modifier pair, and we performed a leave-one-out cross-validation: we left one example for testing and we trained on the remaining 599 ones, repeating this procedure 600 times so that each example be used for testing.
    Page 5, “Relational Similarity Experiments”
  3. We calculated the similarity between the feature vector of the testing example and each of the training examples’ vectors.
    Page 5, “Relational Similarity Experiments”

See all papers in Proc. ACL 2008 that mention feature vector.

See all papers in Proc. ACL that mention feature vector.

Back to top.

significant improvement

Appears in 3 sentences as: significant improvement (3)
In Solving Relational Similarity Problems Using the Web as a Corpus
  1. Turney (2006b) achieves 56% accuracy on this dataset, which matches the average human performance of 57%, and represents a significant improvement over the 20% random-guessing baseline.
    Page 4, “Relational Similarity Experiments”
  2. Both results represent a statistically significant improvement over the majority class baseline and over using sentence words only, and a slight improvement over the best type A and type 0 systems on SemEval ’07, which achieved 66% and 67% accuracy, respectively.4
    Page 6, “Relational Similarity Experiments”
  3. As we can see, using prepositions alone yields about 33% accuracy, which is a statistically significant improvement over the maj ority-class baseline.
    Page 7, “Relational Similarity Experiments”

See all papers in Proc. ACL 2008 that mention significant improvement.

See all papers in Proc. ACL that mention significant improvement.

Back to top.