Automatic Coupling of Answer Extraction and Information Retrieval
Yao, Xuchen and Van Durme, Benjamin and Clark, Peter

Article Structure

Abstract

Information Retrieval (IR) and Answer Extraction are often designed as isolated or loosely connected components in Question Answering (QA), with repeated over-engineering on IR, and not necessarily performance gain for QA.

Introduction

The overall performance of a Question Answering system is bounded by its Information Retrieval (IR) front end, resulting in research specifically on Information Retrieval for Question Answering (IR4QA) (Greenwood, 2008; Sakai et al., 2010).

Background

Besides Predictive Annotation, our work is closest to structured retrieval, which covers techniques of dependency path mapping (Lin and Pantel, 2001; Cui et al., 2005; Kaisser, 2012), graph matching with Semantic Role Labeling (Shen and Lapata, 2007) and answer type checking (Pinchak et al., 2009), etc.

Method

Table 1 already shows some examples of features associating question types with answer types.

Experiments

We want to measure and compare the performance

Conclusion

We described a method to perform coupled information retrieval with a prior knowledge of the downstream QA system.

Topics

NER

Appears in 15 sentences as: NER (18)
In Automatic Coupling of Answer Extraction and Information Retrieval
  1. This will be our off-the-shelf QA system, which recognizes the association between question type and expected answer types through various features based on e.g., part-of-speech tagging (POS) and named entity recognition ( NER ).
    Page 2, “Introduction”
  2. For instance, line 2 in Table 1 says that if there is a when question, and the current token’s NER label is DATE, then it is likely that this token is tagged as ANS.
    Page 2, “Introduction”
  3. IR can easily make use of this knowledge: for a when question, IR retrieves sentences with tokens labeled as DATE by NER , or POS tagged as CD.
    Page 2, “Introduction”
  4. The only extra processing is to pre-tag and index the text with POS and NER labels.
    Page 2, “Introduction”
  5. Compared to shallow (POS, NER ) structured retrieval, deep structures need more processing power and smoothing, but might also be more precise.
    Page 3, “Background”
  6. We let the trained QA system guide the query formulation when performing coupled retrieval with Indri (Strohman et al., 2005), given a corpus already annotated with POS tags and NER labels.
    Page 3, “Method”
  7. For instance, the NER tagger we used divides location into two categories: GPE (geo locations) and LOC
    Page 3, “Method”
  8. Take the previous where question, besides NER[O]=GPE and NER[O]=LOC, we also found oddly N ER[O]=PERSON an important feature, due to that the NER tool sometimes mistakes PERSON for LOC.
    Page 3, “Method”
  9. Since NER and POS tags are not lexicalized they accumulate many more counts (i.e.
    Page 3, “Method”
  10. NER Types First We found NER labels better indicators of expected answer types than POS tags.
    Page 3, “Method”
  11. In general POS tags are too coarse-grained in answer types than NER labels.
    Page 3, “Method”

See all papers in Proc. ACL 2013 that mention NER.

See all papers in Proc. ACL that mention NER.

Back to top.

POS tags

Appears in 10 sentences as: POS tagged (1) POS tags (9)
In Automatic Coupling of Answer Extraction and Information Retrieval
  1. IR can easily make use of this knowledge: for a when question, IR retrieves sentences with tokens labeled as DATE by NER, or POS tagged as CD.
    Page 2, “Introduction”
  2. Moreover, our approach extends easily beyond fixed answer types such as named entities: we are already using POS tags as a demonstration.
    Page 2, “Introduction”
  3. We let the trained QA system guide the query formulation when performing coupled retrieval with Indri (Strohman et al., 2005), given a corpus already annotated with POS tags and NER labels.
    Page 3, “Method”
  4. Since NER and POS tags are not lexicalized they accumulate many more counts (i.e.
    Page 3, “Method”
  5. NER Types First We found NER labels better indicators of expected answer types than POS tags .
    Page 3, “Method”
  6. In general POS tags are too coarse-grained in answer types than NER labels.
    Page 3, “Method”
  7. POS tags accumulate even more counts than NER labels, thus they need separate down-weighting.
    Page 3, “Method”
  8. If the top-weighted features are based on NER, then we do not include POS tags for that question.
    Page 3, “Method”
  9. Otherwise POS tags are useful, for instance, in answering how questions.
    Page 3, “Method”
  10. But only sentence boundaries, POS tags and NER labels were kept as the annotation of the corpus.
    Page 4, “Experiments”

See all papers in Proc. ACL 2013 that mention POS tags.

See all papers in Proc. ACL that mention POS tags.

Back to top.

named entities

Appears in 7 sentences as: named entities (5) Named Entity (1) named entity (1)
In Automatic Coupling of Answer Extraction and Information Retrieval
  1. Moreover, this approach is more robust against, e.g., entity recognition errors, because answer typing knowledge is learned from how the data was actually labeled, not from how the data was assumed to be labeled (e. g., manual templates usually assume perfect labeling of named entities , but often it is not the case
    Page 1, “Introduction”
  2. This will be our off-the-shelf QA system, which recognizes the association between question type and expected answer types through various features based on e.g., part-of-speech tagging (POS) and named entity recognition (NER).
    Page 2, “Introduction”
  3. Moreover, our approach extends easily beyond fixed answer types such as named entities : we are already using POS tags as a demonstration.
    Page 2, “Introduction”
  4. (2007) proposed indexing text with their semantic roles and named entities .
    Page 2, “Background”
  5. Queries then include constraints of semantic roles and named entities for the predicate and its arguments in the question.
    Page 2, “Background”
  6. 5Ogilvie (2010) showed in chapter 4.3 that keyword and named entities based retrieval actually outperformed SRL—based structured retrieval in MAP for the answer-bearing sentence retrieval task in their setting.
    Page 3, “Method”
  7. sentence-segmented and word-tokenized by NLTK (Bird and Loper, 2004), dependency-parsed by the Stanford Parser (Klein and Manning, 2003), and NER-tagged by the Illinois Named Entity Tagger (Ratinov and Roth, 2009) with an 18-label type set.
    Page 4, “Experiments”

See all papers in Proc. ACL 2013 that mention named entities.

See all papers in Proc. ACL that mention named entities.

Back to top.

unigram

Appears in 6 sentences as: Unigram (1) unigram (10)
In Automatic Coupling of Answer Extraction and Information Retrieval
  1. Unigram QA Model The QA system uses up to trigram features (Table 1 shows examples of unigram and bigram features).
    Page 3, “Method”
  2. We drop this strict constraint (which may need further smoothing) and only use unigram features, not by simply extracting “good” unigram features from the trained model, but by retraining the model with only unigram features.
    Page 4, “Method”
  3. 6This is because the weights of unigram to trigram features in a loglinear CRF model is a balanced consequence for maximization.
    Page 4, “Experiments”
  4. A unigram feature might end up with lower weight because another trigram containing this unigram gets a higher weight.
    Page 4, “Experiments”
  5. Then we would have missed this feature if we only used top unigram features.
    Page 4, “Experiments”
  6. Thus we retrain the model with only unigram features to make sure weights are “assigned properly” among only unigram features.
    Page 4, “Experiments”

See all papers in Proc. ACL 2013 that mention unigram.

See all papers in Proc. ACL that mention unigram.

Back to top.

CRF

Appears in 5 sentences as: CRF (5)
In Automatic Coupling of Answer Extraction and Information Retrieval
  1. The QA system employs a linear chain Conditional Random Field ( CRF ) (Lafferty et al., 2001) and tags each token as either an answer (ANS) or not (0).
    Page 2, “Introduction”
  2. With weights optimized by CRF training (Table 1), we can learn how answer features are correlated with question features.
    Page 2, “Introduction”
  3. These features, whose weights are optimized by the CRF training, directly reflect what the most important answer types associated with each question type are.
    Page 2, “Introduction”
  4. Table I: Learned weights for sampled features with respect to the label of current token (indexed by [0]) in a CRF .
    Page 2, “Introduction”
  5. 6This is because the weights of unigram to trigram features in a loglinear CRF model is a balanced consequence for maximization.
    Page 4, “Experiments”

See all papers in Proc. ACL 2013 that mention CRF.

See all papers in Proc. ACL that mention CRF.

Back to top.

Question Answering

Appears in 3 sentences as: Question Answering (3) question answering: (1)
In Automatic Coupling of Answer Extraction and Information Retrieval
  1. Information Retrieval (IR) and Answer Extraction are often designed as isolated or loosely connected components in Question Answering (QA), with repeated over-engineering on IR, and not necessarily performance gain for QA.
    Page 1, “Abstract”
  2. The overall performance of a Question Answering system is bounded by its Information Retrieval (IR) front end, resulting in research specifically on Information Retrieval for Question Answering (IR4QA) (Greenwood, 2008; Sakai et al., 2010).
    Page 1, “Introduction”
  3. at the three stages of question answering:
    Page 4, “Experiments”

See all papers in Proc. ACL 2013 that mention Question Answering.

See all papers in Proc. ACL that mention Question Answering.

Back to top.

semantic roles

Appears in 3 sentences as: Semantic Role (1) semantic roles (2)
In Automatic Coupling of Answer Extraction and Information Retrieval
  1. Besides Predictive Annotation, our work is closest to structured retrieval, which covers techniques of dependency path mapping (Lin and Pantel, 2001; Cui et al., 2005; Kaisser, 2012), graph matching with Semantic Role Labeling (Shen and Lapata, 2007) and answer type checking (Pinchak et al., 2009), etc.
    Page 2, “Background”
  2. (2007) proposed indexing text with their semantic roles and named entities.
    Page 2, “Background”
  3. Queries then include constraints of semantic roles and named entities for the predicate and its arguments in the question.
    Page 2, “Background”

See all papers in Proc. ACL 2013 that mention semantic roles.

See all papers in Proc. ACL that mention semantic roles.

Back to top.

PDF available from www.aclweb.org.
Send us an email