Let cc represent the feature vector of a candidate relation instance , and y 6 {+1, —1} represent a class label.

After data cleaning, we obtained 4290 positive instances among 48614 candidate relation instances .

In order to concentrate on the classification accuracy for the target relation type, we removed the positive instances of the auxiliary relation types from the test set, although in practice we need to extract these auxiliary relation instances using learned classifiers for these relation types.

We focus on extracting binary relation instances between two relation arguments occurring in the same sentence.

Some example relation instances and their corresponding relation types as defined by ACE can be found in Table 1.

Each pair of entities within a single sentence is considered a candidate relation instance , and the task becomes predicting whether or not each candidate is a true instance of T. We use feature-based logistic regression classifiers.

This time, every pair of entities appearing together in a sentence is considered a potential relation instance , and whenever those entities appear together, features are extracted on the sentence and added to a feature vector for that entity pair.

We refer to individual ordered pairs in this relation as ‘relation instances’ .

We use relations and relation instances from Freebase, a freely available online database of structured semantic data.

This means that 900,000 Freebase relation instances are used in training, and 900,000 are held out.

For human evaluation experiments, all 1.8 million relation instances are used in training.

For all our experiments, we only extract relation instances that do not appear in our training data, i.e., instances that are not already in Freebase.

The NIST Automatic Content Extraction (ACE) RDC 2003 and 2004 corpora, for example, include over 1,000 documents in which pairs of entities have been labeled with 5 to 7 major relation types and 23 to 24 subrelations, totaling 16,771 relation instances .

Thus whereas the supervised training paradigm uses a small labeled corpus of only 17,000 relation instances as training data, our algorithm can use much larger amounts of data: more text, more relations, and more instances.

Table 1 shows examples of relation instances extracted by our system.

Approaches based on WordNet have often only looked at the hypernym (isa) or meronym (part-of) relation (Girju et al., 2003; Snow et al., 2005), while those based on the ACE program (Doddington et al., 2004) have been restricted in their evaluation to a small number of relation instances and corpora of less than a million words.

classifier is trained first to distinguish between relation instances and non-relation instances.

Table 4 simplifies a relation instance as a three tuple <Context, M], M2> where the Context includes the Before, Between and After from the

previous research, we used in experiments the nwire (newswire) and bnews (broadcast news) genres of the data containing 348 documents and 4374 relation instances .

The non-relation instances generated were about 8 times more than the relation instances .

The unbalanced distribution of relation instances and non-relation instances remains as an obstacle for pushing the performance of relation extraction to the next level.

At the lexical level, a relation instance can be seen as a sequence of tokens which form a five tuple <Before, M], Between, M2, After>.

Specifically, we first train a binary classifier to distinguish between relation instances and non-relation instances.

Then rather than using the thresholded output of this binary classifier as training data, we use only the annotated relation instances to train a multi-class classifier for the 7 relation types.

In contrast, the kernel based method does not explicitly extract features; it designs kernel functions over the structured sentence representations (sequence, dependency or parse tree) to capture the similarities between different relation instances (Zelenko et al., 2003; Bunescu and Mooney, 2005a; Bunescu and Mooney, 2005b; Zhao and Grishman, 2005; Zhang et al., 2006; Zhou et al., 2007; Qian et al., 2008).

The assumption is that even if the word soldier may never have been seen in the annotated Employment relation instances , other words which share the same cluster membership with soldier such as president and ambassador may have been observed in the Employment instances.

Relation instance extraction.

Given an input entity and a target relation, we aim at finding a filler value for a relation instance .

For each of the relations to extract, a binary classifier (extractor) decides whether the example is a valid relation instance .

Second, the distant supervision assumption underlying our approach is that for a seed relation instance (entity, relation, value), any textual mention of entity and value expresses the relation.

Under the evaluation metrics proposed by TAC-KBP 2011, if the value of the relation instance is judged as correct, the score for temporal anchoring depends on how well the returned interval matches the one provided in the key.

We assume the input is a relation instance and a set of supporting documents.

For each document and relational instance , we have to select those temporal expressions that are relevant.

Now, the mapping of temporal constraints depends on the temporal link to the time expression identified; also, the semantics of the event have to be considered in order to decide the time period associated to a relation instance .

We will denominate relation instance a triple (entity, relation name, value).

We aim at anchoring relation instances to their temporal validity.

Let us assume that each relation instance is valid during a certain temporal interval, I = [750, if].

Therefore, we chose a content selection representation of a finer granularity than an utterance: we identify relation instances that can both effectively detect the crucial content and incorporate enough syntactic information to facilitate the downstream surface realization.

More specifically, our relation instances are based on information extraction methods that identify a lexical indicator (or trigger) that evokes a relation of interest and then employ syntactic information, often in conjunction with semantic constraints, to find the argument con-stituent(or target phrase) to be extracted.

For example, in the DA cluster of Figure 2, (want, an LCD display with a spinning wheel) and (push-buttons, 0n the outside) are two relation instances .

Given the DA cluster to be summarized, the Content Selection module identifies a set of summary-worthy relation instances represented as indicator-argument pairs (i.e.

In the first step, each relation instance is filled into templates with disparate structures that are learned automatically from the training set (Template F ill-ing).

A statistical ranker then selects one best abstract per relation instance (Statistical Ranking).

In this section, we describe surface realization, which renders the relation instances into natural language abstracts.

Once the templates are learned, the relation instances from Section 4 are filled into the templates to generate an abstract (see Section 5.2).

In order to utilize the structure information of a relation instance , we discuss how soft constraint can be used to capture the local dependency.

is obtained by segmenting every relation instance using the ICTCLAS package, collecting very word produced by ICTCLAS.

The structure information (or dependent information) of relation instance is critical for recognition.

Any relation instance violating these constraints (or below a predefined threshold) will be abandoned.

Aiming at the Chinese inattentive structure, we utilize the soft constraint to capture the local dependency in a relation instance .

At each round of iteration, we gain a recovered matrix and average the F114 scores from Top-5 to Top-all predicted relation instances to measure the performance.

In practical applications, we also concern about the precision on Top-N predicted relation instances .

Table 3: Precision of NFE-13, DRMC-b and DRMC-l on Top-100, Top-200 and Top-500 predicted relation instances .

The relation instances are the triples related to President Barack Obama in the Freebase, and the relation mentions are some sentences describing him in the Wikipedia.

8According to convention, we regard a structured triple r(ei, ej) as a relation instance which is composed of a pair of entities <81, ej >and a relation name 7“ with respect to them.

Not all relation mentions express the corresponding relation instances .

Finally, we can achieve Top-N predicted relation instances via ranking the values of P7173 |pi).

As we are stepping into the big data era, the explosion of unstructured Web texts simulates us to build more powerful models that can automatically extract relation instances from large-scale online natural language corpora without hand-labeled annotation.

(2009) adopted Freebase (Bollacker et al., 2008; Bollacker et al., 2007), a large-scale crowdsourcing knowledge base online which contains billions of relation instances and thousands of relation names, to distantly supervise Wikipedia corpus.

However, there are cases when we may exploit relation extraction in multiple languages and there are corpora with relation instances annotated for more than one language, such as the ACE RDC 2005 English and Chinese corpora.

can be enhanced by relation instances translated from another language (e.g.

This demonstrates that there is some complementariness between relation instances in two languages, particularly when the training data is scarce.

In the held-out evaluation, relation instances discovered from testing articles were automatically compared with those in Freebase.

This let us calculate the precision of each method for the best n relation instances .

For manual evaluation, we picked the top ranked 50 relation instances for the most frequent 15 relations.

A relation instance is a tuple consisting of two entities and relation 7“.

For example, place_of_birth(Michael Jackson, Gary) in Figure 1 is a relation instance .

Relation extraction seeks to extract relation instances from text.

Using Freebase relation instances produces cleaner training data than NELL’s automatically-extracted instances.

Using the relation instances and Wikipedia sentences, we constructed a data set for distantly-supervised relation extraction.

Each system was run on this data set, extracting all logical forms from each sentence that entailed at least one category or relation instance .

Semantics are learned by training the parser to extract knowledge base relation instances from a corpus of unlabeled sentences, in a distantly-supervised training regime.

A knowledge base K (e.g., NELL), containing relation instances Mel, 62) E K.

Distant supervision is provided by the following constraint: every relation instance 7“(€1,€2) E K must be expressed by at least one sentence in 8031,62), the set of sentences that mention both 61 and 62 (Hoffmann et al., 2011).

\II is a deterministic OR constraint that checks whether each logical form entails the relation instance Mel, 62), deterministically setting yr 2 1 if any logical form entails the instance and yr 2 0 otherwise.

The candidate relation instances were generated by considering all pairs of entities that occur in the same sentence.

No-transfer classifier (NT) We only use the few labeled instances of the target relation type together with the negative relation instances to train a binary classifier.

We consider relation extraction as a classification problem, where each pair of entities A and B within a sentence S is a candidate relation instance .

We extract features from a sequence representation and a parse tree representation of each relation instance .

use a subgraph in the relation instance graph (J iang and Zhai, 2007b) that contains only the node presenting the head word of the entity A, labeled with the entity type or entity mention types, to describe a single entity attribute.

In contrast to Open IE, we tune the relation patterns for a domain of interest, using labeled relation instances in source and target domains and unlabeled instances in the target domain.

Finally, new relation instances are extracted using kernel based classifiers, e. g., the SVM classifier.

The feature we used includes characteristics of relation instance , phrase properties and context information (See Section 3 for details).

Relation instances of the same type often share some common characteristics.

diversity in the discovered relation types by restricting the number of times a single word can serve as either an indicator or part of the argument of a relation instance .

First, the model’s generative process encourages coherence in the local features and placement of relation instances .

To incorporate training examples in our model, we simply treat annotated relation instances as observed variables.

For finance, it takes at least 10 annotated documents (corresponding to roughly 130 annotated relation instances ) for the CRF to match the semi-supervised model’s performance.

For earthquake, using even 10 annotated documents (about 71 relation instances ) is not sufficient to match our model’s performance.

We first describe paraphrase acquisition, we then summarize our method for learning surface patterns, and finally describe the use of patterns for extracting relation instances .

To compare the quality of the extraction patterns, and relation instances , we use the method presented by Ravichandran and Hovy (2002) as the baseline.

The intuition is that applying the vague patterns for extracting target relation instances might find some good instances, but will also find many bad ones.

Using distributional similarity avoids the problem of obtaining overly general patterns and the pre-computation of paraphrases means that we can obtain the set of patterns for any relation instantaneously .

While procedurally different, both methods depend heavily on the performance of the syntax parser and require complex syntax tree matching to extract the relation instances .

Many users also contribute to Freebase by annotating relation instances .

One reason is the following: some relation instances should have multiple labels but they have only one label in Freebase.

For automatic evaluation, we use relation instances in Freebase as ground truth, and employ two clustering

They employ a self-leamer to extract relation instances , but no attempt is made to cluster instances into relations.

As in other information extraction systems, the category and relation instances extracted by NELL contain polysemous and synonymous noun phrases.

Both extracting more relation instances and adding new relations to the ontology will improve synonym res-

The main input to ConceptResolver is a set of extracted category and relation instances over noun phrases, like person(:c1) and ceoOf(:c1, :52), produced by running NELL.