We present a novel technique for learning semantic representations , which extends the distributional hypothesis to multilingual data and joint-space embeddings.

We extend our approach to learn semantic representations at the document level, too.

Most prior work on learning compositional semantic representations employs parse trees on their training data to structure their composition functions (Socher et al., 2012; Hermann and Blunsom, 2013, inter alia).

We utilise this diversity to abstract further from monolingual surface realisations to deeper semantic representations .

Assume two functions f : X —> Rd and g : Y —> Rd, which map sentences from languages cc and 3/ onto distributed semantic representations in Rd.

We present a novel unsupervised technique for learning semantic representations that leverages parallel corpora and employs semantic transfer through compositional representations.

The results on this task, in comparison with a number of strong baselines, further demonstrate the relevance of our approach and the success of our method in learning multilingual semantic representations over a wide range of languages.

Here, we focus on learning semantic representations and investigate how the use of multilingual data can improve learning such representations at the word and higher level.

We describe a multilingual objective function that uses a noise-contrastive update between semantic representations of different languages to learn these word embeddings.

As part of this, we use a compositional vector model (CVM, henceforth) to compute semantic representations of sentences and documents.

Recent advances in functional Magnetic Resonance Imaging (fMRI) offer a significant new approach to studying semantic representations in humans by making it possible to directly observe brain activity while people comprehend words and sentences.

4 Using vector-based models of semantic representation to account for the systematic variances in neural activity

4.1 Lexical Semantic Representation

Table 3 shows the semantic representation for strong and dog.

There have been a variety of approaches from different scientific communities trying to characterize semantic representations .

Recent advances in functional Magnetic Resonance Imaging (fMRI) provide a significant new approach to studying semantic representations in humans by making it possible to directly observe brain activity while people comprehend words and sentences.

Given these early succesess in using fMRI to discriminate categorial information and to model lexical semantic representations of individual words, it is interesting to ask whether a similar approach can be used to study the representation of adjective-noun phrases.

We devise a gold-standard sense- and parse tree-annotated dataset based on the intersection of the Penn Treebank and SemCor, and experiment with different approaches to both semantic representation and disambiguation.

Below, we outline the dataset used in this research and the parser evaluation methodology, explain the methodology used to perform PP attachment, present the different options for semantic representation , and finally detail the disambiguation methods.

The gold-standard sense annotations allow us to perform upper bound evaluation of the relative impact of a given semantic representation on parsing and PP attachment performance, to contrast with the performance in more realistic semantic disambiguation settings.

4.3 Semantic representation

There are three main aspects that we have to consider in this process: (i) the semantic representation , (ii) semantic disambiguation, and (iii) morphology.

The more fine-grained our semantic representation , the higher the average polysemy and the greater the need to distinguish between these senses.

We explore several models for semantic representation , based around WordNet (Fellbaum, 1998).

In experimenting with different semantic representations , we require some strategy to disambiguate the semantic class of polysemous words in context (e. g. determining for each instance of crane whether it refers to an animal or a lifting device).

A corpus-based evaluation with a large-scale grammar shows that our algorithm reduces over 80% of sentences to one or two readings, in negligible runtime, and thus makes it possible to work with semantic representations derived by deep large-scale grammars.

The algorithm presented here makes it possible, for the first time, to derive a single meaningful semantic representation from the syntactic analysis of a deep grammar on a large scale.

In the future, it will be interesting to explore how these semantic representations can be used in applications.

We could then perform such inferences on (cleaner) semantic representations , rather than strings (as they do).

Over the past few years, there has been considerable progress in the ability of manually created large-scale grammars, such as the English Resource Grammar (ERG, Copestake and Flickinger (2000)) or the ParGram grammars (Butt et al., 2002), to parse wide-coverage text and assign it deep semantic representations .

While applications should benefit from these very precise semantic representations, their usefulness is limited by the presence of semantic ambiguity: On the Rondane Treebank (Oepen et al., 2002), the ERG computes an average of several million semantic representations for each sentence, even when the syntactic analysis is fixed.

We follow an underspecification approach to managing ambiguity: Rather than deriving all semantic representations from the syntactic analysis, we work with a single, compact underspecified semantic representation, from which the semantic representations can then be extracted by need.

The idea of deriving a single approximative semantic representation for ambiguous sentences goes back to Hobbs (1983); however, Hobbs only works his algorithm out for a restricted class of quantifiers, and his representations can be weaker than our weakest readings.

The work presented here is related to other approaches that reduce the set of readings of an underspecified semantic representation (USR).

Both of these formalisms can be used to model scope ambiguities compactly by regarding the semantic representations of a sentence as trees.

Though the most likely alignment 6.3- for a fixed semantic representation fizj can be found efficiently using a Viterbi algorithm, computing the most probable pair (éj, fly) is still intractable.

We use a modification of the beam search algorithm, where we keep a set of candidate meanings (partial semantic representations ) and compute an alignment for each of them using a form of the Viterbi algorithm.

We argue that groups of unannotated texts with overlapping and noncontradictory semantics represent a valuable source of information for learning semantic representations .

A simple and efficient inference method recursively induces joint semantic representations for each group and discovers correspondence between lexical entries and latent semantic concepts.

Even though the dependencies are only conveyed via {mj : j 75 the space of possible meanings m is very large even for relatively simple semantic representations , and, therefore, we need to resort to efficient approximations.

However, a major weakness of this algorithm is that decisions about components of the composite semantic representation (e. g., argument values) are made only on the basis of a single text, which first mentions the corresponding aspects, without consulting any future texts k’ > k, and these decisions cannot be revised later.

Alternatively, if such groupings are not available, it may still be easier to give each semantic representation (or a state) to multiple annotators and ask each of them to provide a textual description, instead of annotating texts with semantic expressions.

Unsupervised learning with shared latent semantic representations presents its own challenges, as exact inference requires marginalization over possible assignments of the latent semantic state, consequently, introducing nonlocal statistical dependencies between the decisions about the semantic structure of each text.

Sentence and text alignment has also been considered in the related context of paraphrase extraction (see, e.g., (Dolan et al., 2004; Barzilay and Lee, 2003)) but this prior work did not focus on inducing or learning semantic representations .

In this work we studied the use of weak supervision in the form of noncontradictory relations between documents in learning semantic representations .

However, exact inference for groups of documents with overlapping semantic representation is generally prohibitively expensive, as the shared latent semantics introduces nonlocal dependences between semantic representations of individual documents.

We describe a method of normalizing numerical expressions referring to the same amount in text into a unified semantic representation .

For instance, the context of 319 people in the sentence 319 people face a water shortage is “face” and “water shortage.” In order to extract and aggregate numerical expressions in various documents, we converted the numerical expressions into semantic representations (to be described in Section 4.1), and extracted their context (to be described in Section 4.2).

Numerical Semantic representation Expression Value | Unit ‘ Mod.

The first step for collecting numerical expressions is to recognize when a numerical expression is mentioned and then to normalize it into a semantic representation .

This gives us a semantic representation of each of the 60 words in a 218-dimensional behavioral space.

It is possible that some JNNSE(Brain+Text) dimensions are being used exclusively to fit brain activation data, and not the semantics represented in both brain and corpus data.

This result shows that neural semantic representations can create a latent representation that is faithful to unseen corpus statistics, providing further evidence that the two data sources share a strong common element.

For example, multiple word senses collide in the same vector, and noise from mis-parsed sentences or spam documents can interfere with the final semantic representation .

In this work we focus on the scientific question: Can the inclusion of brain data improve semantic representations learned from corpus data?

One could also use a topic model style formulation to represent this semantic representation task.

The same idea could be applied here: the latent semantic representation generates the observed brain activity and corpus statistics.

For example, models with behavioral data (Sil-berer and Lapata, 2012) and models with visual information (Bruni et al., 2011; Silberer et al., 2013) have both shown to improve semantic representations .

We argue that multilingual parallel data provides a valuable source of indirect supervision for induction of shallow semantic representations .

Although in this work we focused primarily on improving performance for each individual language, crosslingual semantic representation could be extracted by a simple postprocessing step.

5This has been explored before for shallow semantic representations (Lang and Lapata, 2011a; Titov and Klementiev, 201 1).

The goal of this work is to show that parallel data is useful in unsupervised induction of shallow semantic representations .

Though syntactic representations are often predictive of semantic roles (Levin, 1993), the interface between syntactic and semantic representations is far from trivial.

However, most of this research has focused on induction of syntactic structures (Kuhn, 2004; Snyder et al., 2009) or morphologic analysis (Snyder and Barzilay, 2008) and we are not aware of any previous work on induction of semantic representations in the crosslingual setting.

Learning of semantic representations in the context of monolingual weakly-parallel data was studied in Titov and Kozhevnikov (2010) but their setting was semi-supervised and they experimented only on a restricted domain.

Semi-supervised and weakly-supervised techniques have also been explored for other types of semantic representations but these studies again have mostly focused on restricted domains (Kate and Mooney, 2007; Liang et al., 2009; Goldwasser et al., 2011; Liang et al., 2011).

The trained parser produces a full syntactic parse of any sentence, while simultaneously producing logical forms for portions of the sentence that have a semantic representation within the parser’s predicate vocabulary.

We demonstrate our approach by training a parser whose semantic representation contains 130 predicates from the NELL ontology.

Our parser ASP produces a full syntactic parse of any sentence, while simultaneously producing logical forms for sentence spans that have a semantic representation within its predicate vocabulary.

We suggest that a large populated knowledge base should play a key role in syntactic and semantic parsing: in training the parser, in resolving syntactic ambiguities when the trained parser is applied to new text, and in its output semantic representation .

A semantic representation tied to a knowledge base allows for powerful inference operations — such as identifying the possible entity referents of a noun phrase — that cannot be performed with shallower representations (e.g., frame semantics (Baker et al., 1998) or a direct conversion of syntax to logic (B08, 2005)).

Our parser produces a full syntactic parse of every sentence, and furthermore produces logical forms for portions of the sentence that have a semantic representation within the parser’s predicate vocabulary.

This synergy gives our parser a richer semantic representation than previous work, while simultaneously enabling broad coverage.

1, each category is associated with a lambda term of semantic representations , and each combinatory rule is associated with rules for semantic composition.

Since these rules are universal, we can obtain different semantic representations by switching the semantic representations of lexical categories.

coordination and semantic representation in particular.

12) must be used to construct the semantic representation , namely the PAS.

Several options arise to thoroughly represent s. First, we find it useful to consider the semantic representation of the affirmative counterpart: AGENT(the cow, ate), THEME(grass, ate), and INSTRUMENT(With a fork, ate).

Table 2 depicts five different possible semantic representations .

It corresponds to the semantic representation of the affirmative counterpart after applying the pseudo-relation NOT over the focus of the negation.

In this paper, we present a novel way to semantically represent negation using focus detection.

The main contributions are: (l) interpretation of negation using focus detection; (2) focus of negation annotation over all PropBank negated sen-tencesl; (3) feature set to detect the focus of negation; and (4) model to semantically represent negation and reveal its underlying positive meaning.

Despite large typological differences between Wambaya and the languages on which the development of the resource was based, the Grammar Matrix is found to provide a significant jump-start in the creation of the grammar for Wambaya: With less than 5.5 person-weeks of development, the Wambaya grammar was able to assign correct semantic representations to 76% of the sentences in a naturally occurring text.

The core type hierarchy defines the basic feature geometry, the ways that heads combine with arguments and adjuncts, linking types for relating syntactic to semantic arguments, and the constraints required to compositionally build up semantic representations in the format of Minimal Recursion Semantics (Copestake et al., 2005; Flickinger and Bender, 2003).

To relate such discontinuous noun phrases to appropriate semantic representations where ‘having-

The linguistic analyses encoded in the grammar serve to map the surface strings to semantic representations (in Minimal Recursion Semantics (MRS) format (Copestake et al., 2005)).

This section has presented the Matrix-derived grammar of Wambaya, illustrating its semantic representations and analyses and measuring its performance against held-out data.

For example, we could envisage a parser that uses semantic representations to guide its search, e.g., by pruning syntactic analyses that have a low semantic probability.

2009); however, the semantic component of these models is limited to semantic role information, rather than attempting to build a full semantic representation for a sentence.

Importantly, composition models are not defined with a specific semantic space in mind, they could easily be adapted to LSA, or simple co-occurrence vectors, or more sophisticated semantic representations (e.g., Griffiths et al.

LDA is a probabilistic topic model offering an alternative to spatial semantic representations .

Besides, replicating Pynte et al.’s (2008) finding, we were also interested in assessing whether the underlying semantic representation (simple semantic space or LDA) and composition function (additive versus multiplicative) modulate reading times differentially.

We call this approach hypertagging, as it operates at a level “above” the syntax, tagging semantic representations with syntactic lexical categories.

This process involves converting the corpus to reflect more precise analyses, Where feasible, and adding semantic representations to the lexical categories.

We have introduced a novel type of supertagger, which we have dubbed a hypertagger, that assigns CCG category labels to elementary predications in a structured semantic representation with high accuracy at several levels of tagging ambiguity in a fashion reminiscent of (Bangalore and Rambow, 2000).

We have dubbed this approach hypertagging, as it operates at a level “above” the syntax, moving from semantic representations to syntactic categories.

As the effort to engineer a grammar suitable for realization from the CCGbank proceeds in parallel to our work on hypertagging, we expect the hypertagger-seeded realizer to continue to improve, since a more complete and precise extracted grammar should enable more complete realizations to be found, and richer semantic representations should

Finally, we will describe the different types of semantic representation that were used.

We will experiment with the semantic representations used in Agirre et a1.

We experiment with both full 88s and SFs as instances of fine-grained and coarse-grained semantic representation , respectively.

We evaluated the effectiveness of our attribute classifiers by integrating their predictions with traditional text-only models of semantic representation .

(2004)) to learn a joint semantic representation from the textual and visual modalities.

Visual input represents a major source of data from which humans can learn semantic representations of linguistic and nonlinguistic communicative actions (Regier, 1996).

Grounding semantic representations with visual information is an instance of multimodal leam-ing.

On average, each concept was annotated with 19 attributes; approximately 14.5 of these were not part of the semantic representation created by McRae et al.’s (2005) participants for that concept even though they figured in the representations of other concepts.

A head-driven phrase structure grammar (HPSG) parser is used to obtain the deep syntactic information, which includes a fine-grained description of the syntactic property and a semantic representation of a sentence.

The semantic representation of the new phrase is calculated at the same time.

Second, we can identify sub-trees in a parse tree/forest that correspond to basic units of the semantics, namely sub-trees covering a predicate and its arguments, by using the semantic representation given in the signs.

deep syntactic information of an English sentence, which includes a fine-grained description of the syntactic property and a semantic representation of the sentence.

The Logon project2 (Oepen et al., 2007) for Norwegian-English translation integrates in-depth grammatical analysis of Norwegian (using lexical functional grammar, similar to (Riezler and Maxwell, 2006)) with semantic representations in the minimal recursion semantics framework, and fully grammar-based generation for English using HPSG.

Koller and Thater (2006) define semantic equivalence in terms of a rewrite system that specifies under what conditions two quantifiers may exchange their positions without changing the meaning of the semantic representation .

Expressions of natural language itself are (extremely underspecified) descriptions of sets of semantic representations , and so Ebert’s argument applies to NL expressions as well.

In the past few years, a “standard model” of scope underspecification has emerged: A range of formalisms from Underspecified DRT (Reyle, 1993) to dominance graphs (Althaus et al., 2003) have offered mechanisms to specify the “semantic material” of which the semantic representations are built up, plus dominance or outscoping relations between these building blocks.

We can now use regular tree grammars in underspecification by representing the semantic representations as trees and taking an RTG G as an underspecified description of the trees in L(G).

Models that learn semantic representations from both linguistic and perceptual input outperform text-only models in many contexts and better reflect human concept acquisition.

This model learns high quality lexical semantic representations based on the distributional properties of words in text, and has been shown to outperform simple distributional models on applications such as semantic composition and analogical mapping (Mikolov et al., 2013b).

The USP norms have been used in many previous studies to evaluate semantic representations (Andrews et al., 2009; Feng and Lapata, 2010; Silberer and Lapata, 2012; Roller and Schulte im Walde, 2013).

Multi-modal models in which perceptual input is filtered according to our algorithm learn higher-quality semantic representations than previous approaches, resulting in a significant performance improvement of up to 17% in captur-

This system operates over the normalized semantic representations provided by the LinGO English Resource Grammar (ERG; Flickinger, 2000).3 The ERG maps surface strings to meaning representations in the format of Minimal Recursion Semantics (MRS; Copestake et al., 2005).

Our crawling rules operate on semantic representations , but the annotations are with reference to the surface string.

In terms of our operations defined over semantic representations , this is rendered as follows: all arguments of the negated verb are selected by argument crawling, all in-tersective modifiers by label crawling, and functor crawling (Fig.

We present UCCA, a novel multilayered framework for semantic representation that aims to accommodate the semantic distinctions expressed through linguistic utterances.

This paper presented Universal Conceptual Cognitive Annotation (UCCA), a novel framework for semantic representation .

An extensive comparison of UCCA to existing approaches to syntactic and semantic representation , focusing on the major resources available for English, is found in Section 5.

Several annotated corpora offer a joint syntactic and semantic representation .

In tackling this challenging learning problem, we introduce a new semantic representation which highlights a parallel between dependency syntax and efficient evaluation of logical forms.

Our system is based on a new semantic representation , DCS, which offers a simple and expressive alternative to lambda calculus.

A major focus of this work is on our semantic representation , DCS, which offers a new perspective on compositional semantics.

The main technical contribution of this work is a new semantic representation , dependency-based compositional semantics (DCS), which is both simple and expressive (Section 2).

First, it is semantic based in that it takes as input a deep semantic representation rather than e.g., a sentence or a parse tree.

While previous simplification approaches starts from either the input sentence or its parse tree, our model takes as input a deep semantic representation namely, the Discourse Representation Structure (DRS, (Kamp, 1981)) assigned by Boxer (Curran et al., 2007) to the input complex sentence.

By handling deletion using a probabilistic model trained on semantic representations , we can avoid deleting obligatory arguments.

The general form of a semantic representation for a linguistic unit is an ordered tuple of a vector and n E N matrices:1

The form of semantic representations we are using is shown in Table l.2

The semantic representations we propose include a semantic vector for constituents of any semantic type, thus enabling semantic comparison for words of different parts of speech (the case of demolition vs. demolish).

3.2 Semantic Representations

In general, these models specify mechanisms for constructing semantic representations from text corpora based on the distributional hypothesis (Harris, 1970): words that appear in similar linguistic contexts are likely to have related meanings.

Our model uses stacked autoencoders (Bengio et al., 2007) to induce semantic representations integrating visual and textual information.

Other directions of our future work include further exploitation of the new semantic representation .

Since our system uses an off-the-shelf dependency parser, and semantic representations are obtained from simple rule-based conversion from dependency trees, there will be only one (right or wrong) interpretation in face of ambiguous sentences.

Optimistically, we believe DCS can provide a framework of semantic representation with sufficiently wide coverage for real-world texts.

Fortunately, we know the fact that the two phrases should share the same semantic representation if they express the same meaning.

The above equation also indicates that the source-side parameters 65 can be optimized independently as long as the semantic representation pt of the target phrase 75 is given to compute Esem(s|t, 6) with Eq.

For example, as each node in the recursive auto-encoder shares the same weight matrix, the BRAE model would become weak at learning the semantic representations for long sentences with tens of words.

Any combination which contains a semantic representation equivalent to the input at the conclusion of the algorithm is a valid output from a chart generation system.

This is then used by a surface realization module which encodes the enriched semantic representation into natural language.

For instance, a communicative goal of ‘red(d), dog(d)’ (in English, “say anything about a dog which is red.”) would match a sentence with the semantic representation ‘red(subj), dog(subj), cat(obj), chased(subj, obj)’, like “The red dog chased the cat”, for instance.

We demonstrate that our semantic representation achieves state-of-the-art performance in three experiments using semantic similarity at different lexical levels (i.e., sense, word, and text), surpassing the performance of previous similarity measures that are often specifically targeted for each level.

Despite the potential advantages, few approaches to semantic similarity operate at the sense level due to the challenge in sense-tagging text (Navigli, 2009); for example, none of the top four systems in the recent SemEval-2012 task on textual similarity compared semantic representations that incorporated sense information (Agirre et al., 2012).

(2009) used a similar semantic representation of short texts from random walks on WordNet, which was applied to paraphrase recognition and textual entailment.

Semantic representations constructed in this way beat a strong baseline and can be of higher quality than representations directly constructed from corpus data.

A natural extension of our research is to address morpheme composition and morphological induction jointly, trying to model the intuition that good candidate morphemes should have coherent semantic representations .

Our goal is to automatically construct, given distributional representations of stems and affixes, semantic representations for the derived words containing those stems and affixes.

3.1 Tree-based Semantic Representations

In this way, each nonterminal symbol has a semantic representation and an associated parse category.

We have presented a novel technique for surface realisation that treats generation as a sequence labelling task by combining a CRF with tree-based semantic representations .

In the future, we plan to investigate how to adapt smatch to other semantic representations .

Related work on directly measuring the semantic representation includes the method in (Dri-dan and Oepen, 2011), which evaluates semantic parser output directly by comparing semantic substructures, though they require an alignment between sentence spans and semantic substructures.

Following (Langkilde and Knight, 1998) and (Langkilde-Geary, 2002), we refer to this semantic representation as AMR (Abstract Meaning Representation).

The surface realisation algorithm extends the algorithm proposed in (Gardent and Perez—Beltrachini, 2010) and adapts it to work on the SR dependency input rather than on flat semantic representations .

Typically, the input to surface realisation is a structured representation (i.e., a flat semantic representation , a first order logic formula or a dependency tree) rather than a string.

Approaches based on reversible grammars (Carroll et al., 1999) have used the semantic formulae output by parsing to evaluate the coverage and performance of their realiser; similarly, (Gardent et al., 2010) developed a tool called GenSem which traverses the grammar to produce flat semantic representations and thereby provide a benchmark for performance and coverage evaluation.

First, the ability to extract semantic representations from CCG derivations is not dependent on the language class of a CCG.

On the practical side, we have corpora with CCG derivations for each sentence (Hockenmaier and Steedman, 2007), a wide-coverage parser trained on that corpus (Clark and Curran, 2007) and a system for converting CCG derivations into semantic representations (Bos et al., 2004).

Bos’s system for building semantic representations from CCG derivations is only possible due to the categorial nature of CCG.