Article Structure
Abstract
We propose CMSMs, a novel type of generic compositional models for syntactic and semantic aspects of natural language, based on matrix multiplication.
Introduction
In computational linguistics and information retrieval, Vector Space Models (Salton et al., 1975) and its variations — such as Word Space Models (Schutze, 1993), Hyperspace Analogue to Language (Lund and Burgess, 1996), or Latent Semantic Analysis (Deerwester et al., 1990) — have become a mainstream paradigm for text representation.
Preliminaries
In this section, we recap some aspects of linear algebra to the extent needed for our considerations about CMSMs.
Compositionality and Matrices
The underlying principle of compositional semantics is that the meaning of a sentence (or a word phrase) can be derived from the meaning of its constituent tokens by applying a composition operation.
CMSMs Encode Vector Space Models
In VSMs numerous vector operations have been used to model composition (Widdows, 2008), some of the more advanced ones being related to quantum mechanics.
CMSMs Encode Symbolic Approaches
Now we will elaborate on symbolic approaches to language, i.e., discrete grammar formalisms, and show how they can conveniently be embedded into CMSMs.
Combination of Different Approaches
Another central advantage of the proposed matrix-based models for word meaning is that several matrix models can be easily combined into one.
Related Work
We are not the first to suggest an extension of classical VSMs to matrices.
Conclusion and Future Work
We have introduced a generic model for compositionality in language where matrices are associated with tokens and the matrix representation of a token sequence is obtained by iterated matrix multiplication.
Topics
semantical space
- More formally, the underlying idea can be described as follows: given a mapping [[ - ]] : 2 —> S from a set of tokens (words) 2 into some semantical space S (the elements of which we will simply call “meanings”), we find a semantic composition operation ><2 S* —> S mapping sequences of meanings to meanings such that the meaning of a sequence of tokens 0'10'2 .Page 3, “Compositionality and Matrices”
- the semantical space consists of quadratic matrices, and the composition operator ><1 coincides with matrix multiplication as introduced in Section 2.Page 3, “Compositionality and Matrices”
- This way, abstracting from specific initial mental state vectors, our semantic space S can be seen as a function space of mental transformations represented by matrices, whereby matrix multiplication realizes subsequent execution of those transformations triggered by the input token sequence.Page 3, “Compositionality and Matrices”
- From the perspective of our compositionality framework, those approaches employ a group (or pre-group) (G, -) as semantical space S where the group operation (often written as multiplication) is used as composition operation ><.Page 5, “CMSMs Encode Symbolic Approaches”
See all papers in Proc. ACL 2010 that mention semantical space.
See all papers in Proc. ACL that mention semantical space.
Back to top.
Vector Space
- In computational linguistics and information retrieval, Vector Space Models (Salton et al., 1975) and its variations — such as Word Space Models (Schutze, 1993), Hyperspace Analogue to Language (Lund and Burgess, 1996), or Latent Semantic Analysis (Deerwester et al., 1990) — have become a mainstream paradigm for text representation.Page 1, “Introduction”
- Vector Space Models (VSMs) have been empirically justified by results from cognitive science (Gardenfors, 2000).Page 1, “Introduction”
- A great variety of linguistic models are subsumed by this general idea ranging from purely symbolic approaches (like type systems and cate-gorial grammars) to rather statistical models (like vector space and word space models).Page 3, “Compositionality and Matrices”
- Widdows (2008) proposes a number of more advanced vector operations well-known from quantum mechanics, such as tensor product and convolution, to model composition in vector spaces .Page 8, “Related Work”
See all papers in Proc. ACL 2010 that mention Vector Space.
See all papers in Proc. ACL that mention Vector Space.
Back to top.
word order
- This requires novel modeling paradigms, as most VSMs have been predominantly used for meaning representation of single words and the key problem of common bag-of-words-based VSMs is that word order information and thereby the structure of the language is lost.Page 1, “Introduction”
- Thereby, they realize a multiset (or bag-of-words) semantics that makes them insensitive to structural differences of phrases conveyed through word order .Page 3, “Compositionality and Matrices”
- (2008) use permutations on vectors to account for word order .Page 5, “CMSMs Encode Vector Space Models”
- Among the early attempts to provide more compelling combinatory functions to capture word order information and the non-commutativity of linguistic compositional operation in VSMs is the work of Kintsch (2001) who is using a more sophisticated addition function to model predicate-argument structures in VSMs.Page 8, “Related Work”
See all papers in Proc. ACL 2010 that mention word order.
See all papers in Proc. ACL that mention word order.
Back to top.
