Discovery of Topically Coherent Sentences for Extractive Summarization
Celikyilmaz, Asli and Hakkani-Tur, Dilek

Article Structure

Abstract

Extractive methods for multi-document summarization are mainly governed by information overlap, coherence, and content constraints.

Introduction

A query-focused multi-document summarization model produces a short-summary text of a set of documents, which are retrieved based on a user’s query.

Multi-Document Summarization Models

Prior research has demonstrated the usefulness of sentence extraction for summarization based on lexical, semantic, and discourse constraints.

Topic Coherence for Summarization

In this section we discuss the main contribution, our two hierarchical mixture models, which improve summary generation performance through the use of tiered topic models.

Two-Tiered Topic Model - TTM

Our base model, the two-tiered topic model (TTM), is inspired by the hierarchical topic model, PAM, proposed by Li and McCallum (2006).

Enriched Two-Tiered Topic Model

Our model can discover words that are related to summary text using posteriors P(6H) and P(6T),

Final Experiments

In this section, we qualitatively compare our models against state-of-the art models and later apply an intrinsic evaluation of generated summaries on topical coherence and informativeness.

Conclusion

We introduce two new models for extracting topically coherent sentences from documents, an important property in extractive multi-document summarization systems.

Topics

topic models

Appears in 14 sentences as: Topic Model (1) topic model (5) topic models (9)
In Discovery of Topically Coherent Sentences for Extractive Summarization
  1. In particular (Haghighi and Vanderwende, 2009; Celikyilmaz and Hakkani-Tur, 2010) build hierarchical topic models to identify salient sentences that contain abstract concepts rather than specific concepts.
    Page 1, “Introduction”
  2. Some of these work (Haghighi and Vanderwende, 2009; Celikyilmaz and Hakkani-Tur, 2010) focus on the discovery of hierarchical concepts from documents (from abstract to specific) using extensions of hierarchal topic models (Blei et al., 2004) and reflect this hierarchy on the sentences.
    Page 2, “Multi-Document Summarization Models”
  3. we utilize the advantages of previous topic models and build an unsupervised generative model that can associate each word in each document with three random variables: a sentence S, a higher-level topic H, and a lower-level topic T, in an analogical way to PAM models (Li and McCallum, 2006), i.e., a directed acyclic graph (DAG) representing mixtures of hierarchical structure, where super-topics are multi-nomials over subtopics at lower levels in the DAG.
    Page 2, “Multi-Document Summarization Models”
  4. In this section we discuss the main contribution, our two hierarchical mixture models, which improve summary generation performance through the use of tiered topic models .
    Page 2, “Topic Coherence for Summarization”
  5. Our base model, the two-tiered topic model (TTM), is inspired by the hierarchical topic model , PAM, proposed by Li and McCallum (2006).
    Page 3, “Two-Tiered Topic Model - TTM”
  6. Figure l: Graphical model depiction of two-tiered topic model (TTM) described in section §4.
    Page 3, “Two-Tiered Topic Model - TTM”
  7. Our two-tiered topic model for salient sentence discovery can be generated for each word in the document (Algorithm 1) as follows: For a word wid in document d, a random variable acid is drawn, which determines if wid is query related, i.e., wid either exists in the query or is related to the queryz.
    Page 3, “Two-Tiered Topic Model - TTM”
  8. Algorithm 1 Two—Tiered Topic Model Generation
    Page 4, “Two-Tiered Topic Model - TTM”
  9. EM algorithms might face problems with local maxima in topic models (Blei et al., 2003) suggesting implementation of approximate methods in which some of the parameters, e.g., 6H, 6T, 2p, and 6, can be integrated out, resulting in standard Dirichlet-multinomial as well as binomial distributions.
    Page 4, “Two-Tiered Topic Model - TTM”
  10. We compare TTM results on synthetic experiments against PAM (Li and McCallum, 2006) a similar topic model that clusters topics in a hierarchical structure, where super-topics are distributions over subtopics.
    Page 5, “Two-Tiered Topic Model - TTM”
  11. The following models are used as benchmark: (i) PYTHY (Toutanova et al., 2007): Utilizes human generated summaries to train a sentence ranking system using a classifier model; (ii) HIERSUM (Haghighi and Vanderwende, 2009): Based on hierarchical topic models .
    Page 8, “Final Experiments”

See all papers in Proc. ACL 2011 that mention topic models.

See all papers in Proc. ACL that mention topic models.

Back to top.

generative model

Appears in 3 sentences as: generative model (2) generative models (1)
In Discovery of Topically Coherent Sentences for Extractive Summarization
  1. In this paper, we introduce a series of new generative models for multiple-documents, based on a discovery of hierarchical topics and their correlations to extract topically coherent sentences.
    Page 1, “Introduction”
  2. we utilize the advantages of previous topic models and build an unsupervised generative model that can associate each word in each document with three random variables: a sentence S, a higher-level topic H, and a lower-level topic T, in an analogical way to PAM models (Li and McCallum, 2006), i.e., a directed acyclic graph (DAG) representing mixtures of hierarchical structure, where super-topics are multi-nomials over subtopics at lower levels in the DAG.
    Page 2, “Multi-Document Summarization Models”
  3. Once the level and conditional path is drawn (see level generation for ETTM above) the rest of the generative model is same as TTM.
    Page 7, “Enriched Two-Tiered Topic Model”

See all papers in Proc. ACL 2011 that mention generative model.

See all papers in Proc. ACL that mention generative model.

Back to top.

generative process

Appears in 3 sentences as: generative process (3)
In Discovery of Topically Coherent Sentences for Extractive Summarization
  1. We identify sentences as meta-variables of document clusters, which the generative process models both sentences and documents using tiered topics.
    Page 3, “Topic Coherence for Summarization”
  2. Thus; we present enriched TTM (ETTM) generative process (Fig.3), which samples words not only from low-level topics but also from high-level topics as well.
    Page 6, “Enriched Two-Tiered Topic Model”
  3. Similar to TTM’s generative process , if wid is related to a given query, then cc 2 1 is deterministic, otherwise cc 6 {0,1} is stochastically determined if wid should be sampled as a background word (2123) or through hierarchical path, i.e., HT pairs.
    Page 6, “Enriched Two-Tiered Topic Model”

See all papers in Proc. ACL 2011 that mention generative process.

See all papers in Proc. ACL that mention generative process.

Back to top.

Gibbs sampling

Appears in 3 sentences as: Gibbs samplers (1) Gibbs sampling (2)
In Discovery of Topically Coherent Sentences for Extractive Summarization
  1. We use Gibbs sampling which allows a combination of estimates from several local maxima of the posterior distribution.
    Page 4, “Two-Tiered Topic Model - TTM”
  2. We obtain DS during Gibbs sampling (in §4.l), which indicates a saliency score of each sentence sj E S,j = LSD:
    Page 5, “Two-Tiered Topic Model - TTM”
  3. For our models, we ran Gibbs samplers for 2000 iterations for each configuration throwing out first 500 samples as burn-in.
    Page 8, “Final Experiments”

See all papers in Proc. ACL 2011 that mention Gibbs sampling.

See all papers in Proc. ACL that mention Gibbs sampling.

Back to top.