The current topic modeling approaches for Information Retrieval do not allow to explicitly model query-oriented latent topics.

We propose a model-based feedback approach that learns Latent Dirichlet Allocation topic models on the top-ranked pseudo-relevant feedback, and we measure the semantic coherence of those topics.

Based on the words used within a document, topic models learn topic level relations by assuming that the document covers a small set of concepts.

This is one of the reasons of the intensive use of topic models (and especially LDA) in current research in Natural Language Processing (NLP) related areas.

From that perspective, topic models seem attractive in the sense that they can provide a descriptive and intuitive representation of concepts.

Instead of viewing 9 as a set of document language models that are likely to contain topical information about the query, we take a probabilistic topic modeling approach.

Figure 1: Semantic coherence of the topic models f( N of feedback documents.

We present a topic model based approach to answer these questions.

Since agreement and disagreement expressions are usually multi-word phrases, we propose to employ a ranking method to identify highly relevant phrases prior to topic modeling .

For quantitative evaluation, topic models are often compared using perplexity.

In our earlier work (Mukherjee and Liu, 2012a), we proposed three topic models to mine contention points, which also extract AD-expressions.

In this paper, we further improve the work by coupling an information retrieval method to rank good candidate phrases with topic modeling in order to discover more accurate AD-expressions.

Topics in most topic models like LDA are usually unigram distributions.

Topic models: Our work is also related to topic modeling and joint modeling of topics and other information as we jointly model several aspects of discussions/debates.

Topic models like pLSA (Hofmann, 1999) and LDA (Blei et al., 2003) have proved to be very successful in mining topics from large text collections.

There have been various extensions to multi-grain (Titov and McDonald, 2008), labeled (Ramage et al., 2009), and sequential (Du et al., 2010) topic models .

In addition to regular text classification, we utilized topic modeling of the entire dataset in various ways.

Topic modeling of the corpora provides interpretable themes that exist in these reports.

A binary topic model was also built as an unsupervised classification approach with the assumption that each topic corresponds to a class.

2.2 Topic Modeling

Topic modeling is an unsupervised learning algorithm that can automatically discover themes of a document collection.

Either sampling methods such as Gibbs Sampling (Griffiths and Steyvers, 2004) or optimization methods such as variational Bayes approximation (Asuncion et al., 2009) can be used to train a topic model based on LDA.

In this study, we developed several topic modeling based classification systems for clinical reports.

Topic modeling is an unsupervised technique that can automatically identify themes from a given set of documents and find topic distributions of each document.

Therefore, topic model output of patient reports could contain very useful clinical information.

For text classification, topic modeling techniques have been utilized in various ways.

In our study, we removed the most frequent and infrequent words to have a manageable vocabulary size but we did not utilize topic model output for this purpose.

Supervised topic models with a logistic likelihood have two issues that potentially limit their practical use: 1) response variables are usually over-weighted by document word counts; and 2) existing variational inference methods make strict mean-field assumptions.

First, we present a general framework of Bayesian logistic supervised topic models with a regularization parameter to better balance response variables and words.

Second, to solve the intractable posterior inference problem of the generalized Bayesian logistic supervised topic models , we present a simple Gibbs sampling algorithm by exploring the ideas of data augmentation (Tanner and Wong, 1987; van Dyk and Meng, 2001; Holmes and Held, 2006).

More specifically, we extend Polson’s method for Bayesian logistic regression (Polson et al., 2012) to the generalized logistic supervised topic models , which are much more challeng-

We now present the generalized Bayesian logistic supervised topic models .

A logistic supervised topic model consists of two parts — an LDA model (Blei et al., 2003) for describing the words W = {wd}dD=1, where Wd 2 {wdnfigl denote the words within document d, and a logistic classifier for considering the supervising signal y = {yd}dD=1.

Logistic classifier: To consider binary supervising information, a logistic supervised topic model (e.g., sLDA) builds a logistic classifier using the topic representations as input features

It can efficiently handle semantic ambiguity by extending standard topic models with two new features.

Since MTR provides a mixture of properties adapted from earlier models, we present performance benchmarks on tag clustering using: (i) LDA; (ii) Hidden Markov Topic Model HMTM (Gruber et al., 2005); and, (iii) w-LDA (Petterson et al., 2010) that uses word features as priors in LDA.

Each topic model uses Gibbs sampling for inference and parameter learning.

For fair comparison, each benchmark topic model is provided with prior information on word-semantic tag distributions based on the labeled training data, hence, each K latent topic is assigned to one of K semantic tags at the beginning of Gibbs sampling.

Our first contribution is a new probabilistic topic model , Markov Topic Regression (MTR), which uses rich features to capture the degree of association between words and semantic tags.

Standard topic models , such as Latent Dirichlet Allocation (LDA) (Blei et al., 2003), use a bag-of-words approach, which disregards word order and clusters words together that appear in a similar global context.

Recent topic models consider word sequence information in documents (Griffiths et al., 2005; Moon et al., 2010).

Thus, we build a semantically rich topic model , MTR, using word context features as side information.

Our approach employs a monolingual LDA topic model to derive a similarity measure between the test conversation and the set of training conversations, which is used to bias translation choices towards the current context.

We use the DARPA TransTac English-Iraqi parallel two-way spoken dialogue collection to train both translation and LDA topic models .

We use the English side of these conversations for training LDA topic models .

4.1 Topic modeling with LDA

The full conversation history is available for training the topic models and estimating topic distributions in the training set.

We use Mallet (McCallum, 2002) for training topic models and inferring topic distributions.

We begin by building a monolingual latent Dirichlet allocation (LDA) topic model on the training conversations (each conversation corresponds to a “document” in the LDA paradigm).

To avoid the need for hard decisions about domain membership, some have used topic modeling to improve SMT performance, e.g., using latent semantic analysis (Tam et al., 2007) or ‘biTAM’ (Zhao and Xing, 2006).

(2012), who both describe adaptation techniques where monolingual LDA topic models are used to obtain a topic distribution over the training data, followed by dynamic adaptation of the phrase table based on the inferred topic of the test document.

While our proposed approach also employs monolingual LDA topic models , it deviates from the above methods in the following important ways.

Our unsupervised model brings together familiar components in natural language processing (like parsers and topic models ) with contextual political information—temporal and dyad dependence—to infer latent event classes.

This is an example of individual lexical features outperforming a topic model for predictive task, because the topic model’s dimension reduction obscures important indicators from individual words.

Similarly, Gerrish and Blei (2011) found that word-based regression outperformed a customized topic model when predicting Congressional bill passage, and Eisen-

13 In the latter, a problem-specific topic model did best.

We use syntactic preprocessing and a logistic normal topic model , including latent temporal smoothing on the political context prior.

Thus the language model is very similar to a topic model’s generation of token topics and wordtypes.

This simple logistic normal prior is, in terms of topic models , analogous to the asymmetric Dirichlet prior version of LDA in Wallach et al.

We separated this corpus into three non-overlapping sets: a training set of 500 programs for parameter estimation in topic modeling and LE, a development set of 133 programs for empirical tuning and a test set of 400 programs for performance evaluation.

When evaluating the effects of different size of the training set, the number of latent topics in topic modeling process was set to 64.

When evaluating the effects of different number of latent topics in topic modeling computation, we fixed the size of the training set to 500 news programs and changed the number of latent topics from 16 to 256.

To deal with these problems, some topic model techniques which provide conceptual level matching have been introduced to text and story segmentation task (Hearst, 1997).

Using topic models to discover subtypes of businesses, a domain-specific sentiment lexicon, and a number of new techniques for increasing precision in sentiment aspect extraction yields attributes that give a rich representation of the restaurant domain.

‘Top-level’ repeatedly queries the user’s top-level category preferences, ‘Subtopic’ additionally uses our topic modeling subcategories, and ‘All’ uses these plus the aspects extracted from reviews.

Using these topic models , we assign a business

2We use the Topic Modeling Toolkit implementation: http://n1p.stanford.edu/software/tmt

The framework makes use of techniques from topic modeling and sentiment-based aspect extraction to identify fine-grained attributes for each business.

generate the bilingual topic model (0” from the

Preiss (2012) transformed the source language topical model to the target language and classified probability distribution of topics in the same language, whose shortcoming is that the effect of model translation seriously hampers the comparable corpora quality.

(2009) adapted monolingual topic model to bilingual topic model in which the documents of a concept unit in different languages were assumed to share identical topic distribution.

Bilingual topic model is widely adopted to mine translation equivalents from multi-language documents (Mimno et al., 2009;Ivaneta1,2011)

The larger question threads data is employed for feature learning, such as translation model, and topic model training.

To reduce the false negatives of word mismatch in vector model, we also use the topic models to extend matching to semantic topic level.

The topic model , such as Latent Dirichlet Allocation (LDA) (Blei et al., 2003), considers a collection of documents with K latent topics, where K is much smaller than the number of words.

While most works on context-insensitive predicate inference rules, such as DIRT (Lin and Pantel, 2001), are based on word-level similarity measures, almost all prior models addressing context-sensitive predicate inference rules are based on topic models (except for (Pantel et al., 2007), which was outperformed by later models).

In addition, (Dinu and Lapata, 2010a) adapted the predicate inference topic model from (Dinu and Lapata, 2010b) to compute lexical similarity in context.

This way, we calculate similarity over vectors in the original word space, while biasing them towards the given context via a topic model .

Our model follows the general DIRT scheme while extending it to handle context-sensitive scoring of rule applications, addressing the scenario dealt by the context-sensitive topic models .

Since RSP falls into the unsupervised distributional approach, we compare it with previous similarity-based methods and unsupervised generative topic model 3.

C) Seaghdha (2010) applies topic models for the SP induction with three variations: LDA, Rooth-LDA, and Dual-LDA; Ritter et al.

We use the Mat-lab Topic Modeling Toolbox4 for the inference of latent topics.

Recently, more sophisticated methods are innovated for SP based on topic models , where the latent variables (topics) take the place of semantic classes and distributional clusterings (Seaghdha, 2010; Ritter et al., 2010).

To address the term ambiguity detection problem, we employ a model that combines data from language models, ontologies, and topic modeling .

To do so, we utilized the popular Latent Dirichlet Allocation (LDA (Blei et al., 2003)) topic modeling method.

Following standard procedure, stopwords and infrequent words were removed before topic modeling was performed.

We also attempted using topic modeling approach to detect target candidates.

As shown in Table9 (K is the number of predefined topics), PLSA is not quite effective mainly because traditional topic modeling approaches do not perform well on short texts from social media.

For comparison we also attempted topic modeling approach to detect target candidates, as shown in section 5.3.

Firstly, we would like to include as many features as possible to represent the source/target words in our framework besides simple bag-of-words context similarity (for example, left-context, right-context, and other general-purpose features based on topic models , etc.).

Similarly, we can add other features based on topic models , orthography (Haghighi et al., 2008), temporal (Klementiev et al., 2012), etc.

We note that the new sampling framework is easily extensible to many additional feature types (for example, monolingual topic model features, etc.)