Automatic detection of deception in child-produced speech using syntactic complexity features
Yancheva, Maria and Rudzicz, Frank

Article Structure

Abstract

It is important that the testimony of children be admissible in court, especially given allegations of abuse.

Introduction

The challenge of disambiguating between truth and deception is critical in determining the admissibility of court testimony.

Related Work

Research in the detection of deception in adult speech has included analyses of verbal and nonverbal cues such as behavioral changes, facial expression, speech dysfluencies, and cognitive complexity (DePaulo et al., 2003).

Data

The data used in this study were obtained from Lyon et al.

Methods

Since the data consist of speech produced by 4- to 7-year-old children, the predictive features must depend on the level of syntactic competence of this age group.

Results

We evaluate five classifiers: logistic regression (LR), a multilayer perceptron (MLP), nai've Bayes (NB), a random forest (RF), and a support vector machine (SVM).

Discussion and future work

This paper evaluates automatic estimation of truthfulness in the utterances of children using a novel set of lexical-syntactic features across five types of classifiers.

Topics

SVM

Appears in 11 sentences as: SVM (12)
In Automatic detection of deception in child-produced speech using syntactic complexity features
  1. Two classifiers, Nai've Bayes (NB) and a support vector machine (SVM), were applied on the tokenized and stemmed statements to obtain best classification accuracies of 70% (abortion topic, NB), 67.4% (death penalty topic, NB), and 77% (friend description, SVM ), where the baseline was taken to be 50%.
    Page 2, “Related Work”
  2. The authors note this as well by demonstrating significantly lower results of 59.8% for NB and 57.8% for SVM when cross-topic classification is performed by training each classifier on two topics and testing on the third.
    Page 2, “Related Work”
  3. We evaluate five classifiers: logistic regression (LR), a multilayer perceptron (MLP), nai've Bayes (NB), a random forest (RF), and a support vector machine ( SVM ).
    Page 4, “Results”
  4. The SVM is a parametric binary classifier that provides highly nonlinear decision boundaries given particular kernels.
    Page 5, “Results”
  5. The SVM classifier
    Page 5, “Results”
  6. SVM 0.5954 Polynomial, E = 3, C = l
    Page 6, “Results”
  7. LR 0.6136 0.5333 0.5957* 0.4886 MLP 0.6136 0.5583 0.6170T 0.5909* NB 0.6136* 0.5250 0.5426 0.5682 RF 0.6364T 0.6333* 0.6383T 0.6591T SVM 0.6591 0.5583 0.6064 0.6250*
    Page 6, “Results”
  8. Here, both SVM and RF achieve 83.8% cross-validation accuracy in identifying deception in the speech of 7-year-old subjects.
    Page 6, “Results”
  9. LR 0.7500T 0.5417 0.6667T 0.7297T MLP 0.8333l 0.6250T 0.6154 0.7838T NB 0.6667T 0.4583 0.4103 0.7297* RF 0.8333l 0.5625 0.7179T 0.8378T SVM 0.9167* 0.6250T 0.6154* 0.8378T
    Page 6, “Results”
  10. RF MLP RF MLP RF SVM
    Page 8, “Discussion and future work”
  11. While past research has used logistic regression as a binary classifier (Newman et al., 2003), our experiments show that the best-performing classifiers allow for highly nonlinear class boundaries; SVM and RF models achieve between 62.5% and 91.7% accuracy across age groups — a significant improvement over the baselines of LR and NB, as well as over previous results.
    Page 8, “Discussion and future work”

See all papers in Proc. ACL 2013 that mention SVM.

See all papers in Proc. ACL that mention SVM.

Back to top.

binary classification

Appears in 9 sentences as: Binary classification (3) binary classification (4) binary classifier (2)
In Automatic detection of deception in child-produced speech using syntactic complexity features
  1. Further, the use of binary classification schemes in previous work does not account for partial truths often encountered in real-life scenarios.
    Page 2, “Related Work”
  2. The SVM is a parametric binary classifier that provides highly nonlinear decision boundaries given particular kernels.
    Page 5, “Results”
  3. 5.1 Binary classification across all data
    Page 5, “Results”
  4. 5.2 Binary classification by age group
    Page 5, “Results”
  5. Table 3: Cross-validation accuracy of binary classification performed on entire dataset of 346 transcriptions.
    Page 6, “Results”
  6. Table 4: Cross-validation accuracy of binary classification partitioned by age.
    Page 6, “Results”
  7. 5.3 Binary classification by age group, on verbose transcriptions
    Page 6, “Results”
  8. Table 5: Cross-validation accuracy of binary classification performed on transcriptions with above average word count (136 transcriptions), by age group.
    Page 6, “Results”
  9. While past research has used logistic regression as a binary classifier (Newman et al., 2003), our experiments show that the best-performing classifiers allow for highly nonlinear class boundaries; SVM and RF models achieve between 62.5% and 91.7% accuracy across age groups — a significant improvement over the baselines of LR and NB, as well as over previous results.
    Page 8, “Discussion and future work”

See all papers in Proc. ACL 2013 that mention binary classification.

See all papers in Proc. ACL that mention binary classification.

Back to top.

statistically significant

Appears in 8 sentences as: Statistically significant (1) statistically significant (7)
In Automatic detection of deception in child-produced speech using syntactic complexity features
  1. Despite statistically significant predictors of deception such as shorter talking time, fewer semantic details, and less coherent statements, DePaulo et al.
    Page 1, “Related Work”
  2. and revealed that verbal cues based on lexical categories extracted using the LIWC tool show statistically significant , though small, differences between truth- and lie-tellers.
    Page 2, “Related Work”
  3. Statistically significant results are in bold.
    Page 5, “Results”
  4. performs best, with 59.5% cross-validation accuracy, which is a statistically significant improvement over the baselines of LR (75(4) 2 22.25, p < .0001), and NB (25(4) 2 16.19,]?
    Page 5, “Results”
  5. In comparison with classification accuracy on pooled data, a paired t-test shows statistically significant improvement across all age groups using RF, 75(3) = 1037,]?
    Page 6, “Results”
  6. The classifiers showing statistically significant incremental improvement are marked: *p < .05, lp < .001 (paired t-test, d.f.
    Page 6, “Results”
  7. The classifiers showing statistically significant incremental improvement are marked: *p < .05, Tp < .001 (paired t-test, d.f.
    Page 6, “Results”
  8. A one-factor ANOVA with 7' as the independent variable with 8 levels, and cross-validation accuracy as the dependent variable, confirms that the effect of the threshold is statistically significant (F7,40 = 220.69, p < .0001) with 7' = 4 being the most conservative setting.
    Page 6, “Results”

See all papers in Proc. ACL 2013 that mention statistically significant.

See all papers in Proc. ACL that mention statistically significant.

Back to top.

feature set

Appears in 7 sentences as: Feature Set (1) feature set (5) feature sets (1)
In Automatic detection of deception in child-produced speech using syntactic complexity features
  1. Descriptions of the data (section 3) and feature sets (section 4) precede experimental results (section 5) and the concluding discussion (section 6).
    Page 3, “Related Work”
  2. l.00‘_ Feature Set ; —LIWC 3‘ ‘ —S ntactic «3 .90“.
    Page 7, “Discussion and future work”
  3. Figure 3: Effect of feature set choice on cross-validation accuracy.
    Page 7, “Discussion and future work”
  4. 2012; Almela et al., 2012; Fomaciari and Poesio, 2012), our results suggest that the set of syntactic features presented here perform significantly better than the LIWC feature set on our data, and across seven out of the eight experiments based on age groups and verbosity of transcriptions.
    Page 7, “Discussion and future work”
  5. In forward selection, features are greedily added one-at-a-time (given an initially empty feature set ) until the cross-validation error stops decreasing with the addition of new features (Deng, 1998).
    Page 7, “Discussion and future work”
  6. Table 6: Best 10-fold cross-validation accuracies 2 LIWC-based feature set .
    Page 8, “Discussion and future work”
  7. fication results across all classifiers suggest that accuracies are significantly higher given forward selection (,u = 0.58, 0 = 0.02) relative to the original feature set (it 2 056,0 2 0.03); 75(5) 2 —2.28, p < .05 while the results given the mRMR features are not significantly different.
    Page 8, “Discussion and future work”

See all papers in Proc. ACL 2013 that mention feature set.

See all papers in Proc. ACL that mention feature set.

Back to top.

support vector

Appears in 5 sentences as: support vector (5)
In Automatic detection of deception in child-produced speech using syntactic complexity features
  1. In this work, we evaluate various parameterizations of five classifiers (including support vector machines, neural networks, and random forests) in deciphering truth from lies given transcripts of interviews with 198 victims of abuse between the ages of 4 and 7.
    Page 1, “Abstract”
  2. Our results show that sentence length, the mean number of clauses per utterance, and the Stajner—Mitkov measure of complexity are highly informative syntactic features, that classification accuracy varies greatly by the age of the speaker, and that accuracy up to 91.7% can be achieved by support vector machines given a sufficient amount of data.
    Page 1, “Abstract”
  3. Two classifiers, Nai've Bayes (NB) and a support vector machine (SVM), were applied on the tokenized and stemmed statements to obtain best classification accuracies of 70% (abortion topic, NB), 67.4% (death penalty topic, NB), and 77% (friend description, SVM), where the baseline was taken to be 50%.
    Page 2, “Related Work”
  4. (2006) combined two independent systems — an acoustic Gaussian mixture model based on Mel cepstral features, and a prosodic support vector machine based on features such as pitch, energy, and duration — and achieved an accuracy of 64.4% on a test subset of the Columbia-SRI—Colorado (CSC) corpus of deceptive and non-deceptive speech (Hirschberg et al., 2005).
    Page 2, “Related Work”
  5. We evaluate five classifiers: logistic regression (LR), a multilayer perceptron (MLP), nai've Bayes (NB), a random forest (RF), and a support vector machine (SVM).
    Page 4, “Results”

See all papers in Proc. ACL 2013 that mention support vector.

See all papers in Proc. ACL that mention support vector.

Back to top.

logistic regression

Appears in 4 sentences as: logistic regression (4)
In Automatic detection of deception in child-produced speech using syntactic complexity features
  1. These features were obtained with the Linguistic Inquiry and Word Count (LIWC) tool and used in a logistic regression classifier which achieved, on average, 61% accuracy on test data.
    Page 2, “Related Work”
  2. We evaluate five classifiers: logistic regression (LR), a multilayer perceptron (MLP), nai've Bayes (NB), a random forest (RF), and a support vector machine (SVM).
    Page 4, “Results”
  3. Here, na‘1've Bayes, which assumes conditional independence of the features, and logistic regression , which has a linear decision boundary, are baselines.
    Page 4, “Results”
  4. While past research has used logistic regression as a binary classifier (Newman et al., 2003), our experiments show that the best-performing classifiers allow for highly nonlinear class boundaries; SVM and RF models achieve between 62.5% and 91.7% accuracy across age groups — a significant improvement over the baselines of LR and NB, as well as over previous results.
    Page 8, “Discussion and future work”

See all papers in Proc. ACL 2013 that mention logistic regression.

See all papers in Proc. ACL that mention logistic regression.

Back to top.

significant improvement

Appears in 3 sentences as: significant improvement (3)
In Automatic detection of deception in child-produced speech using syntactic complexity features
  1. performs best, with 59.5% cross-validation accuracy, which is a statistically significant improvement over the baselines of LR (75(4) 2 22.25, p < .0001), and NB (25(4) 2 16.19,]?
    Page 5, “Results”
  2. In comparison with classification accuracy on pooled data, a paired t-test shows statistically significant improvement across all age groups using RF, 75(3) = 1037,]?
    Page 6, “Results”
  3. While past research has used logistic regression as a binary classifier (Newman et al., 2003), our experiments show that the best-performing classifiers allow for highly nonlinear class boundaries; SVM and RF models achieve between 62.5% and 91.7% accuracy across age groups — a significant improvement over the baselines of LR and NB, as well as over previous results.
    Page 8, “Discussion and future work”

See all papers in Proc. ACL 2013 that mention significant improvement.

See all papers in Proc. ACL that mention significant improvement.

Back to top.

PDF available from www.aclweb.org.
Send us an email