Evaluating Roget's Thesauri
Kennedy, Alistair and Szpakowicz, Stan

Article Structure

Abstract

Roget’s Thesaurus has gone through many revisions since it was first published 150 years ago.

Introduction

Roget’s Thesaurus, first introduced over 150 years ago, has gone through many revisions to reach its current state.

Content comparison of the 1911 and 1987 Thesauri

Although the 1987 and 1911 Thesauri are very similar in structure, there are a few differences, among them, the number of levels and the number of parts-of-speech represented.

Comparison on applications

In this section we consider how the two versions of Roget’s Thesaurus and WordNet perform in three applications — measuring word relatedness, synonym identification, and sentence relatedness.

Conclusion and future work

The 1987 version of Roget’s Thesaurus performed better than the 1911 version on all our tests, but we did not find the differences to be statistically significant.

Topics

WordNet

Appears in 51 sentences as: +WordNet (1) WordNet (50)
In Evaluating Roget's Thesauri
  1. We examine the differences in content between the 1911 and 1987 versions of Roget’s, and we test both versions with each other and WordNet on problems such as synonym identification and word relatedness.
    Page 1, “Abstract”
  2. We also present a novel method for measuring sentence relatedness that can be implemented in either version of Roget’s or in WordNet .
    Page 1, “Abstract”
  3. Although the 1987 version of the Thesaurus is better, we show that the 1911 version performs surprisingly well and that often the differences between the versions of R0-get’s and WordNet are not statistically significant.
    Page 1, “Abstract”
  4. We compare two versions, the 1987 and 1911 editions of the Thesaurus with each other and with WordNet 3.0.
    Page 1, “Introduction”
  5. Roget’s Thesaurus has a unique structure, quite different from WordNet , of which the NLP community has yet to take full advantage.
    Page 1, “Introduction”
  6. In this paper we demonstrate that although the 1911 version of the Thesaurus is very old, it can give results comparable to systems that use WordNet or newer versions of Roget’s Thesaurus.
    Page 1, “Introduction”
  7. For applications that call for an NLP-friendly thesaurus, WordNet has become the defacto standard.
    Page 1, “Introduction”
  8. Although WordNet is a fine resources, we believe that ignoring other thesauri is a serious oversight.
    Page 1, “Introduction”
  9. We also proposed a new method of representing the meaning of sentences or other short texts using either WordNet or Roget’s Thesaurus, and tested it on the data set provided by Li et al.
    Page 1, “Introduction”
  10. Similar experiments were carried out using WordNet in combination with a variety of semantic relatedness functions.
    Page 1, “Introduction”
  11. Roget’s Thesaurus was found generally to outperform WordNet on these problems.
    Page 1, “Introduction”

See all papers in Proc. ACL 2008 that mention WordNet.

See all papers in Proc. ACL that mention WordNet.

Back to top.

synset

Appears in 6 sentences as: synset (6) synsets (1) synset’s (1)
In Evaluating Roget's Thesauri
  1. We consider 10 measures, noted in the table as J&C (Jiang and Conrath, 1997), Resnik (Resnik, 1995), Lin (Lin, 1998), W&P (Wu and Palmer, 1994), L&C (Leacock and Chodorow, 1998), H&SO (Hirst and St—Onge, 1998), Path (counts edges between synsets ), Lesk (Banerjee and Pedersen, 2002), and finally Vector and Vector Pair (Patwardhan, 2003).
    Page 4, “Comparison on applications”
  2. We mean a concept in Roget’s to be either a Class, Section, ..., Semicolon Group, while a concept in WordNet is any synset .
    Page 6, “Comparison on applications”
  3. Likewise, in WordNet if c were a synset, then each Ci would be a hyponym synset of 0.
    Page 6, “Comparison on applications”
  4. Obviously if c is a word sense w,- (a word in either a synset or a Semicolon Group), then there can be no sub-concepts Ci.
    Page 6, “Comparison on applications”
  5. In WordNet, the specificity of a word is 1, its synset — 2, the synset’s hypemym — 3, its hypemym — 4, and so on.
    Page 7, “Comparison on applications”
  6. The maximum for WordNet is 0.8506, where the mean is 3, or the first hypernym synset .
    Page 8, “Comparison on applications”

See all papers in Proc. ACL 2008 that mention synset.

See all papers in Proc. ACL that mention synset.

Back to top.

semantic relatedness

Appears in 5 sentences as: semantic relatedness (5)
In Evaluating Roget's Thesauri
  1. We ran the well-established tasks of determining semantic relatedness of pairs of terms and identifying synonyms (J armasz and Szpakowicz, 2004).
    Page 1, “Introduction”
  2. They propose a method of determining semantic relatedness between pairs of terms.
    Page 1, “Introduction”
  3. Similar experiments were carried out using WordNet in combination with a variety of semantic relatedness functions.
    Page 1, “Introduction”
  4. We compare the results for the 1911 and 1987 Roget’s Thesauri with a variety of WordNet-based semantic relatedness measures — see Table 5.
    Page 4, “Comparison on applications”
  5. Other methods of determining sentence semantic relatedness expand term relatedness functions to
    Page 5, “Comparison on applications”

See all papers in Proc. ACL 2008 that mention semantic relatedness.

See all papers in Proc. ACL that mention semantic relatedness.

Back to top.

statistically significant

Appears in 5 sentences as: statistically significant (5)
In Evaluating Roget's Thesauri
  1. Although the 1987 version of the Thesaurus is better, we show that the 1911 version performs surprisingly well and that often the differences between the versions of R0-get’s and WordNet are not statistically significant .
    Page 1, “Abstract”
  2. Even on the largest set (Finkelstein et al., 2001), however, the differences between Roget’s Thesaurus and the Vector method are not statistically significant at the p < 0.05 level for either thesaurus on a two-tailed test4.
    Page 4, “Comparison on applications”
  3. On the (Miller and Charles, 1991) and (Rubenstein and Goodenough, 1965) data sets the best system did not show a statistically significant improvement over the 1911 or 1987 Roget’s Thesauri, even at p < 0.1 for a two-tailed test.
    Page 4, “Comparison on applications”
  4. Much like (Miller and Charles, 1991), the data set used here is not large enough to determine if any system’s improvement is statistically significant .
    Page 8, “Comparison on applications”
  5. The 1987 version of Roget’s Thesaurus performed better than the 1911 version on all our tests, but we did not find the differences to be statistically significant .
    Page 8, “Conclusion and future work”

See all papers in Proc. ACL 2008 that mention statistically significant.

See all papers in Proc. ACL that mention statistically significant.

Back to top.

part-of-speech

Appears in 4 sentences as: Part-of-speech (1) part-of-speech (2) “part-of-speech” (1)
In Evaluating Roget's Thesauri
  1. Hierarchy 1911 1987 Class 8 8 Section 39 39 Subsection 97 95 Head Group 625 596 Head 1044 990 Part-of-speech 3934 3220 Paragraph 10244 6443 Semicolon Group 43196 59915 Total Words 98924 225 124 Unique Words 59768 100470
    Page 2, “Content comparison of the 1911 and 1987 Thesauri”
  2. The part-of-speech level is a little confusing, since clearly no such grouping contains an exhaustive list of all nouns, all verbs etc.
    Page 2, “Content comparison of the 1911 and 1987 Thesauri”
  3. We will write “POS” to indicate a structure in Roget’s and “part-of-speech” to indicate the word category in general.
    Page 2, “Content comparison of the 1911 and 1987 Thesauri”
  4. We use the OpenNLP toolkit6 for segmentation and part-of-speech tagging.
    Page 6, “Comparison on applications”

See all papers in Proc. ACL 2008 that mention part-of-speech.

See all papers in Proc. ACL that mention part-of-speech.

Back to top.

hypernym

Appears in 3 sentences as: hypernym (3)
In Evaluating Roget's Thesauri
  1. These hypernym relations were also put towards solving analogy questions.
    Page 2, “Introduction”
  2. The maximum for WordNet is 0.8506, where the mean is 3, or the first hypernym synset.
    Page 8, “Comparison on applications”
  3. This suggests that the POS and Head are most important for representing text in Roget’s Thesaurus, while the first hypernym is most important for representing text using WordNet.
    Page 8, “Comparison on applications”

See all papers in Proc. ACL 2008 that mention hypernym.

See all papers in Proc. ACL that mention hypernym.

Back to top.

PDF available from www.aclweb.org.
Send us an email