Article Structure
Abstract
Hierarchical HMM (HHMM) parsers make promising cognitive models: while they use a bounded model of working memory and pursue incremental hypotheses in parallel, they still achieve parsing accuracies competitive with chart-based techniques.
Introduction
Since the introduction of a parser-based calculation for surprisal by Hale (2001), statistical techniques have been become common as models of reading difficulty and linguistic complexity.
Parsing Model
This section describes an incremental parser in which surprisal and entropy reduction are simple calculations (Section 2.1).
Evaluation
Surprisal, entropy reduction, and embedding difference from the HHMM parser were evaluated against a full array of factors (Table l) on a corpus of word-by-word reading times using a linear mixed-effects model.
Results
Using the full list of factors in Table l, fixed-effect coefficients were estimated in Table 2.
Discussion
As with previous work on large-scale parser-derived complexity metrics, the linear mixed-effect models suggest that sentence-level factors are effective predictors for reading difficulty — in these evaluations, better than commonly-used lexical and near-neighbor predictors (Pollatsek et al., 2006; Engbert et al., 2005).
Conclusion
The task at hand was to determine whether the HHMM could consistently be considered a plausible psycholinguistic model, producing viable complexity metrics while maintaining other characteristics such as bounded memory usage.
Topics
language model
- Ideally, a psychologically-plausible language model would produce a surprisal that would correlate better with linguistic complexity.Page 1, “Introduction”
- Therefore, the specification of how to encode a syntactic language model is of utmost importance to the quality of the metric.Page 1, “Introduction”
- The purpose of this paper is to determine whether the language model defined by the HHMM parser can also predict reading times —it would be strange if a psychologically plausible model did not also produce Viable complexity metrics.Page 2, “Introduction”
- The rest of this paper is organized as follows: Section 2 defines the language model of the HHMM parser, including definitions of the three complexity metrics.Page 2, “Introduction”
- Both of these metrics fall out naturally from the time-series representation of the language model .Page 3, “Parsing Model”
- With the understanding of what operations need to occur, a formal definition of the language model is in order.Page 5, “Parsing Model”
- This is particularly true when the sentence structure is defined in a language model that is psycholinguistically plausible (here, bounded-memory right-corner form).Page 8, “Discussion”
- This accords with an understated result of Boston et al.’s eye-tracking study (2008a): a richer language model predicts eye movements during reading better than an oversimplified one.Page 8, “Discussion”
- Frank (2009) similarly reports improvements in the reading-time predictiveness of unlexi-calized surprisal when using a language model that is more plausible than PCFGs.Page 8, “Discussion”
- However, we see that this language model performs well despite its lack of lexicalization.Page 9, “Discussion”
See all papers in Proc. ACL 2010 that mention language model.
See all papers in Proc. ACL that mention language model.
Back to top.
probability models
- This go is factored and taken to be a deterministic constant, and is therefore unimportant as a probability model .Page 3, “Parsing Model”
- This leads us to a specification of the reduce and shift probability models .Page 5, “Parsing Model”
- These models can be thought of as picking out a ftd first, finding the matching case, then applying the probability models that matches.Page 5, “Parsing Model”
- A final note: the notation |5@(- | has been used to indicate probability models that are empirical, trained directly from frequency counts of right-comer transformed trees in a large corpus.Page 6, “Parsing Model”
See all papers in Proc. ACL 2010 that mention probability models.
See all papers in Proc. ACL that mention probability models.
Back to top.
statistically significant
- We report factors as statistically significant contributors to reading time if the absolute value of the t-value is greater than 2.Page 7, “Evaluation”
- The first data column shows the regression on all data; the second and third columns divide the data into open and closed classes, because an evaluation (not reported in detail here) showed statistically significant interactions between word class and 3 of the predictors.Page 7, “Results”
- Out of the non-parser-based metrics, word order and bigram probability are statistically significant regardless of the data subset; though reciprocal length and unigram frequency do not reach significance here, likelihood ratio tests (not shown) confirm that they contribute to the model as a whole.Page 7, “Results”
- It can be seen that nearly all the slopes have been estimated with signs as expected, with the exception of reciprocal length (which is not statistically significant ).Page 7, “Results”
See all papers in Proc. ACL 2010 that mention statistically significant.
See all papers in Proc. ACL that mention statistically significant.
Back to top.
lexicalization
- More generally, the parser used in these evaluations differs from other reported parsers in that it is not lexicalized .Page 9, “Discussion”
- However, we see that this language model performs well despite its lack of lexicalization .Page 9, “Discussion”
- This indicates that lexicalization is not a requisite part of syntactic parser performance with respect to predicting linguistic complexity, corroborating the evidence of Demberg and Keller’s (2008) ‘unlexicalized’ (POS-generating, not word-generating) parser.Page 9, “Discussion”
See all papers in Proc. ACL 2010 that mention lexicalization.
See all papers in Proc. ACL that mention lexicalization.
Back to top.
