How to make words with vectors: Phrase generation in distributional semantics
Dinu, Georgiana and Baroni, Marco

Article Structure

Abstract

We introduce the problem of generation in distributional semantics: Given a distributional vector representing some meaning, how can we generate the phrase that best expresses that meaning?

Introduction

Distributional methods for semantics approximate the meaning of linguistic expressions with vectors that summarize the contexts in which they occur in large samples of text.

Related work

To the best of our knowledge, we are the first to explicitly and systematically pursue the generation problem in distributional semantics.

General framework

We start by presenting the familiar synthesis setting, focusing on two-word phrases.

Evaluation setting

In our empirical part, we focus on noun phrase generation.

Noun phrase generation

5.1 One-step decomposition

Noun phrase translation

This section describes preliminary experiments performed in a cross-lingual setting on the task of composing English AN phrases and generating Italian translations.

Generation confidence and generation quality

In Section 3.2 we have defined a search function 3 returning a list of lexical nearest neighbours for a constituent vector produced by decomposition.

Conclusion

In this paper we have outlined a framework for the task of generation with distributional semantic models.

Topics

distributional semantics

Appears in 11 sentences as: distributional semantic (3) Distributional semantics (1) distributional semantics (7)
In How to make words with vectors: Phrase generation in distributional semantics
  1. We introduce the problem of generation in distributional semantics : Given a distributional vector representing some meaning, how can we generate the phrase that best expresses that meaning?
    Page 1, “Abstract”
  2. If distributional semantics is to be considered a proper semantic theory, then it must deal not only with synthesis (going from words to vectors), but also with generation (from vectors to words).
    Page 1, “Introduction”
  3. Distributional semantics assumes a lexicon of atomic expressions (that, for simplicity, we take to be words), each associated to a vector.
    Page 1, “Introduction”
  4. this paper, we introduce a more direct approach to phrase generation, inspired by the work in compositional distributional semantics .
    Page 2, “Introduction”
  5. To the best of our knowledge, we are the first to explicitly and systematically pursue the generation problem in distributional semantics .
    Page 2, “Related work”
  6. They introduce a bidirectional language-to-meaning model for compositional distributional semantics that is similar in spirit to ours.
    Page 2, “Related work”
  7. Compositional distributional semantic systems are often evaluated on phrase and sentence paraphrasing data sets (Bla-coe and Lapata, 2012; Mitchell and Lapata, 2010; Socher et al., 2011; Tumey, 2012).
    Page 5, “Noun phrase generation”
  8. This is a much more challenging task and it paves the way to more realistic applications of distributional semantics in generation scenarios.
    Page 5, “Noun phrase generation”
  9. In this paper we have outlined a framework for the task of generation with distributional semantic models.
    Page 8, “Conclusion”
  10. From a more theoretical point of view, our work fills an important gap in distributional semantics , making it a bidirectional theory of the connection between language and meaning.
    Page 9, “Conclusion”
  11. Some research has already established a connection between neural and distributional semantic vector spaces (Mitchell et al., 2008; Murphy et al., 2012).
    Page 9, “Conclusion”

See all papers in Proc. ACL 2014 that mention distributional semantics.

See all papers in Proc. ACL that mention distributional semantics.

Back to top.

vector representations

Appears in 9 sentences as: vector representations (4) vector representing (4) vectors representing (1)
In How to make words with vectors: Phrase generation in distributional semantics
  1. We introduce the problem of generation in distributional semantics: Given a distributional vector representing some meaning, how can we generate the phrase that best expresses that meaning?
    Page 1, “Abstract”
  2. For example, given the vectors representing red and car, composition derives a vector that approximates the meaning of red car.
    Page 1, “Introduction”
  3. We can, for example, synthesize the vector representing the meaning of a phrase or sentence, and then generate alternative phrases or sentences from this vector to accomplish true paraphrase generation (as opposed to paraphrase detection or ranking of candidate paraphrases).
    Page 1, “Introduction”
  4. Given a vector representing an image, generation can be used to productively construct phrases or sentences that describe the image (as opposed to simply retrieving an existing description from a set of candidates).
    Page 1, “Introduction”
  5. To construct the vector representing a two-word phrase, we must compose the vectors associated to the input words.
    Page 2, “General framework”
  6. where {i and 27 are the vector representations associated to words u and v. fcompR : Rd >< Rd —> Rd (for d the dimensionality of vectors) is a composition function specific to the syntactic relation R holding between the two words.1
    Page 2, “General framework”
  7. Construction of vector spaces We test two types of vector representations .
    Page 4, “Evaluation setting”
  8. (2013a) learns vector representations using a neural network architecture by trying to predict a target word given the words surrounding it.
    Page 4, “Evaluation setting”
  9. (2014) for an extensive comparison of the two types of vector representations .
    Page 4, “Evaluation setting”

See all papers in Proc. ACL 2014 that mention vector representations.

See all papers in Proc. ACL that mention vector representations.

Back to top.

vector spaces

Appears in 7 sentences as: vector space (1) vector spaces (6)
In How to make words with vectors: Phrase generation in distributional semantics
  1. Translation is another potential application of the generation framework: Given a semantic space shared between two or more languages, one can compose a word sequence in one language and generate translations in another, with the shared semantic vector space functioning as interlingua.
    Page 1, “Introduction”
  2. For the rest of this section we describe the construction of the vector spaces and the (de)composition function learning procedure.
    Page 4, “Evaluation setting”
  3. Construction of vector spaces We test two types of vector representations.
    Page 4, “Evaluation setting”
  4. Creation of cross-lingual vector spaces A common semantic space is required in order to map words and phrases across languages.
    Page 6, “Noun phrase translation”
  5. This method is applicable to count-type vector spaces , for which the dimen-
    Page 6, “Noun phrase translation”
  6. Similarly, in the recent work on common vector spaces for the representation of images and text, the current emphasis is on retrieving existing captions (Socher et al., 2014) and not actual generation of image descriptions.
    Page 9, “Conclusion”
  7. Some research has already established a connection between neural and distributional semantic vector spaces (Mitchell et al., 2008; Murphy et al., 2012).
    Page 9, “Conclusion”

See all papers in Proc. ACL 2014 that mention vector spaces.

See all papers in Proc. ACL that mention vector spaces.

Back to top.

cross-lingual

Appears in 5 sentences as: Cross-lingual (1) cross-lingual (4)
In How to make words with vectors: Phrase generation in distributional semantics
  1. We test this in a monolingual scenario (paraphrase generation) as well as in a cross-lingual setting (translation by synthesizing adjective-noun phrase vectors in English and generating the equivalent expressions in Italian).
    Page 1, “Abstract”
  2. The Italian language vectors for the cross-lingual experiments of Section 6 were trained on 1.6 billion tokens from itWaC.5 A word token is a word-form + POS-tag string.
    Page 4, “Evaluation setting”
  3. This section describes preliminary experiments performed in a cross-lingual setting on the task of composing English AN phrases and generating Italian translations.
    Page 6, “Noun phrase translation”
  4. Creation of cross-lingual vector spaces A common semantic space is required in order to map words and phrases across languages.
    Page 6, “Noun phrase translation”
  5. Cross-lingual decomposition training Training proceeds as in the monolingual case, this time concatenating the training data sets and estimating a single (de)-composition function for the two languages in the shared semantic space.
    Page 7, “Noun phrase translation”

See all papers in Proc. ACL 2014 that mention cross-lingual.

See all papers in Proc. ACL that mention cross-lingual.

Back to top.

semantic space

Appears in 5 sentences as: semantic space (5)
In How to make words with vectors: Phrase generation in distributional semantics
  1. Recent work on grounding language in vision shows that it is possible to represent images and linguistic expressions in a common vector-based semantic space (Frome et al., 2013; Socher et al., 2013).
    Page 1, “Introduction”
  2. Translation is another potential application of the generation framework: Given a semantic space shared between two or more languages, one can compose a word sequence in one language and generate translations in another, with the shared semantic vector space functioning as interlingua.
    Page 1, “Introduction”
  3. Creation of cross-lingual vector spaces A common semantic space is required in order to map words and phrases across languages.
    Page 6, “Noun phrase translation”
  4. Cross-lingual decomposition training Training proceeds as in the monolingual case, this time concatenating the training data sets and estimating a single (de)-composition function for the two languages in the shared semantic space .
    Page 7, “Noun phrase translation”
  5. (2012) reconstruct phrase tables based on phrase similarity scores in semantic space .
    Page 9, “Conclusion”

See all papers in Proc. ACL 2014 that mention semantic space.

See all papers in Proc. ACL that mention semantic space.

Back to top.

Recursive

Appears in 4 sentences as: Recursive (2) recursive (2)
In How to make words with vectors: Phrase generation in distributional semantics
  1. longer phrases is handled by recursive extension of the two-word case.
    Page 2, “General framework”
  2. 3.3 Recursive (de)composition
    Page 3, “General framework”
  3. 5.2 Recursive decomposition
    Page 6, “Noun phrase generation”
  4. We continue by testing generation through recursive decomposition on the task of generating noun-preposition-noun (NPN) paraphrases of adjective-nouns (AN) phrases.
    Page 6, “Noun phrase generation”

See all papers in Proc. ACL 2014 that mention Recursive.

See all papers in Proc. ACL that mention Recursive.

Back to top.

gold standard

Appears in 3 sentences as: gold standard (3)
In How to make words with vectors: Phrase generation in distributional semantics
  1. the gold standard (for example vitriol and folk in Table 3).
    Page 6, “Noun phrase generation”
  2. Other than several cases which are acceptable paraphrases but not in the gold standard , phrases related in meaning but not synonymous are the most common error (overcast skies —> skies in sunshine).
    Page 6, “Noun phrase generation”
  3. In many cases (e.g., vicious killer, rough neighborhood) we generate translations that are arguably more natural than those in the gold standard .
    Page 7, “Noun phrase translation”

See all papers in Proc. ACL 2014 that mention gold standard.

See all papers in Proc. ACL that mention gold standard.

Back to top.

PDF available from www.aclweb.org.
Send us an email