We design a segmentation model to optimally partition a sentence into lineal constituents, which can be used to define distributional contexts that are less noisy, semantically more interpretable, and linguistically dis-ambiguated.

While existing work has focused on the classification task of categorizing a phrasal constituent as a MWE or a non-MWE, the general ideas of most of these works are in line with our current framework, and the feature-set for our motif segmentation model is designed to subsume most of these ideas.

3.1 Linear segmentation model

The segmentation model forms the core of the framework.

This study investigates on building a better Chinese word segmentation model for statistical machine translation.

It aims at leveraging word boundary information, automatically learned by bilingual character-based alignments, to induce a preferable segmentation model .

4.2 Various Segmentation Models

The practice in state-of-the-art MT systems is that Chinese sentences are tokenized by a monolingual supervised word segmentation model trained on the hand-annotated treebank data, e.g., Chinese treebank

In recent years, a number of works (Xu et al., 2005; Chang et al., 2008; Ma and Way, 2009; Xi et al., 2012) attempted to build segmentation models for SMT based on bilingual unsegmented data, instead of monolingual segmented data.

We propose leveraging the bilingual knowledge to form learning constraints that guide a supervised segmentation model toward a better solution for SMT.

An intuitive manner is to directly leverage the induced boundary distributions as label constraints to regularize segmentation model learning, based on a constrained learning algorithm.

(2008) enhanced a CRFs segmentation model in MT tasks by tuning the word granularity and improving the segmentation consistence.

(2008) produced a better segmentation model for SMT by concatenating various corpora regardless of their different specifications.

(2011) used the words learned from “chars-to-word” alignments to train a maximum entropy segmentation model .

Inspired by experimental psychological findings suggesting that function words play a special role in word learning, we make a simple modification to an Adaptor Grammar based Bayesian word segmentation model to allow it to learn sequences of monosyllabic “function words” at the beginnings and endings of collocations of (possibly multisyllabic) words.

(1996) and Brent (1999) our word segmentation models identify word boundaries from unsegmented sequences of phonemes corresponding to utterances, effectively performing unsupervised learning of a lexicon.

a word segmentation model should segment this as ju want tu si 69 buk, which is the IPA representation of “you want to see the book”.

Section 2 describes the specific word segmentation models studied in this paper, and the way we extended them to capture certain properties of function words.

Perhaps the simplest word segmentation model is the unigram model, where utterances are modeled as sequences of words, and where each word is a sequence of segments (Brent, 1999; Goldwater et al., 2009).

The next two subsections review the Adaptor Grammar word segmentation models presented in Johnson (2008) and Johnson and Goldwater (2009): section 2.1 reviews how phonotac-tic syllable-structure constraints can be expressed with Adaptor Grammars, while section 2.2 reviews how phrase-like units called “collocations” capture inter-word dependencies.

(2009) point out the detrimental effect that inter-word dependencies can have on word segmentation models that assume that the words of an utterance are independently generated.

M monolingual segmentation model

13 bilingual segmentation model

segmentation model M or B.