Article Structure
Abstract
A number of results in the study of real-time sentence comprehension have been explained by computational models as resulting from the rational use of probabilistic linguistic information.
Introduction
The language processing tasks of reading, listening, and even speaking are remarkably difficult.
Models of eye movements in reading
The two most successful models of eye movements in reading are E—Z Reader (Reichle, Pollatsek, Fisher, & Rayner, 1998; Reichle et al., 2006) and SWIFT (Engbert, Longtin, & Kliegl, 2002; Engbert et al., 2005).
Explaining between-word regressions
In this paper, we use our model to provide a novel explanation for between-word regressive saccades.
Reading as Bayesian inference
At its core, the framework we are proposing is one of reading as Bayesian inference.
Simulation 1
With the description of our model in place, we next proceed to describe the first simulation in which we used the model to test the hypothesis that making regressions is a rational way to cope with confidence in previous regions falling.
Simulation 2
In Simulation 2, we perform a more direct test of the idea that making regressions is a rational response to the problem of confidence falling about previous regions using optimization techniques.
Conclusion
In this paper, we presented a model that performs Bayesian inference on the identity of a sentence, combining a language model with noisy information about letter identities from a realistic visual input model.
Topics
language model
- Unfortunately, however, the Mr. Chips model simplifies the problem of reading in a number of ways: First, it uses a unigram model as its language model , and thus fails to use any information in the linguistic context to help with word identification.Page 2, “Models of eye movements in reading”
- Specifically, our model identifies the words in a sentence by performing Bayesian inference combining noisy input from a realistic visual model with a language model that takes context into account.Page 2, “Models of eye movements in reading”
- This simple example just illustrates the point that if a reader is combining noisy visual information with a language model , then confidence in previous regions will sometimes fall.Page 3, “Explaining between-word regressions”
- Specifically, the model begins reading with a prior distribution over possible identities of a sentence given by its language model .Page 3, “Reading as Bayesian inference”
- model’s prior distribution over the identity of the sentence given the language model is updated to a posterior distribution taking into account both the language model and the visual input obtained thus far.Page 4, “Reading as Bayesian inference”
- Given the visual input and a language model, inferences about the identity of the sentence w can be made by standard Bayesian inference, where the prior is given by the language model and the likelihood is a function of the total visual input obtained from the first to the ith timestep Ii ,Page 6, “Reading as Bayesian inference”
- This model can be efficiently and simply implemented using weighted finite-state automata (wFSAs; Mohri, 1997) as follows: First, we begin with a wFSA representation of the language model , where each arc emits a single character (or is an epsilon-transition emitting nothing).Page 6, “Reading as Bayesian inference”
- 5.1.2 Language modelPage 7, “Simulation 1”
- Our reader’s language model was an unsmoothed bigram model created using a vocabulary set con-Page 7, “Simulation 1”
- Specifically, we constructed the model’s initial belief state (i.e., the distribution over sentences given by its language model ) by directly translating the bigram model into a wFSA in the log semiring.Page 7, “Simulation 1”
- 6.1.3 Language modelPage 8, “Simulation 2”
See all papers in Proc. ACL 2010 that mention language model.
See all papers in Proc. ACL that mention language model.
Back to top.
bigram
- Our reader’s language model was an unsmoothed bigram model created using a vocabulary set con-Page 7, “Simulation 1”
- From this vocabulary, we constructed a bigram model using the counts from every bigram in the BNC for which both words were in vocabulary (about 222,000 bigrams ).Page 7, “Simulation 1”
- Specifically, we constructed the model’s initial belief state (i.e., the distribution over sentences given by its language model) by directly translating the bigram model into a wFSA in the log semiring.Page 7, “Simulation 1”
- To ensure that our results did not depend on smoothing, we only tested the model on sentences in which every bigram occurred in the BNC.Page 7, “Simulation 1”
- Thus, we made single-word changes to 25 more of the sentences (mostly changing proper names and rare nouns) to produce a total of 33 sentences to read, for which every bigram did occur in the BNC.Page 7, “Simulation 1”
- Instead, we begin with the same set of bigrams used in Sim.Page 8, “Simulation 2”
- 1 — i.e., those that contain two in-vocabulary words — and trim this set by removing rare bigrams that occur less than 200 times in the BNC (except that we do not trim any bigrams that occur in our test corpus).Page 8, “Simulation 2”
- This reduces our set of bigrams to about 19,000.Page 8, “Simulation 2”
See all papers in Proc. ACL 2010 that mention bigram.
See all papers in Proc. ACL that mention bigram.
Back to top.
