Article Structure
Abstract
The effort required for a human annotator to detect sentiment is not uniform for all texts, irrespective of his/her expertise.
Introduction
The effort required by a human annotator to detect sentiment is not uniform for all texts.
Understanding Sentiment Annotation Complexity
The process of sentiment annotation consists of two sub-processes: comprehension (where the annotator understands the content) and sentiment judgment (where the annotator identifies the sentiment).
Creation of dataset annotated with SAC
We wish to predict sentiment annotation complexity of the text using a supervised technique.
Predictive Framework for SAC
The previous section shows how gold labels for SAC can be obtained using eye-tracking experiments.
Discussion
Our proposed metric measures complexity of sentiment annotation, as perceived by human annotators.
Conclusion & Future Work
We presented a metric called Sentiment Annotation Complexity (SAC), a metric in SA research that has been unexplored until now.
Topics
human annotator
- The effort required for a human annotator to detect sentiment is not uniform for all texts, irrespective of his/her expertise.Page 1, “Abstract”
- As for training data, since any direct judgment of complexity by a human annotator is fraught with subjectivity, we rely on cognitive evidence from eye-tracking.Page 1, “Abstract”
- We also study the correlation between a human annotator’s perception of complexity and a machine’s confidence in polarity determination.Page 1, “Abstract”
- The effort required by a human annotator to detect sentiment is not uniform for all texts.Page 1, “Introduction”
- Our proposed metric measures complexity of sentiment annotation, as perceived by human annotators .Page 5, “Discussion”
See all papers in Proc. ACL 2014 that mention human annotator.
See all papers in Proc. ACL that mention human annotator.
Back to top.
confidence score
- In other words, the goal is to show that the confidence scores of a sentiment classifier are negatively correlated with SAC.Page 5, “Discussion”
- The confidence score of a classifier8 for given text t is computed as follows:Page 5, “Discussion”
- Table 3 presents the accuracy of the classifiers along with the correlations between the confidence score and observed SAC values.Page 5, “Discussion”
- Finally, we observe a negative correlation between the classifier confidence scores and a SAC, as expected.Page 5, “Conclusion & Future Work”
See all papers in Proc. ACL 2014 that mention confidence score.
See all papers in Proc. ACL that mention confidence score.
Back to top.
MaxEnt
- We use three sentiment classification techniques: Na‘1've Bayes, MaxEnt and SVM with un-igrams, bigrams and trigrams as features.Page 5, “Discussion”
- MaxEnt (Movie) -0.29 (72.17) MaxEnt (Twitter) -0.26 (71.68) SVM (Movie) -().24 (66.27) SVM (Twitter) -().19 (73.15)Page 5, “Discussion”
- MaxEnt has the highest negative correlation of -().29 and -().26.Page 5, “Discussion”
See all papers in Proc. ACL 2014 that mention MaxEnt.
See all papers in Proc. ACL that mention MaxEnt.
Back to top.
SVM
- We use three sentiment classification techniques: Na‘1've Bayes, MaxEnt and SVM with un-igrams, bigrams and trigrams as features.Page 5, “Discussion”
- 7http://scikit-learn.org/stable/ 8In case of SVM , the probability of predicted class is computed as given in Platt (1999).Page 5, “Discussion”
- MaxEnt (Movie) -0.29 (72.17) MaxEnt (Twitter) -0.26 (71.68) SVM (Movie) -().24 (66.27) SVM (Twitter) -().19 (73.15)Page 5, “Discussion”
See all papers in Proc. ACL 2014 that mention SVM.
See all papers in Proc. ACL that mention SVM.
Back to top.
