Sentiment analysis methods extract the attitude of a text via systematic algorithms. To evaluate the validity of common sentiment analysis methods, we use polychoric correlation to compare computer-mediated methods and human-rated analogues. Our main topics of interest are the internal consistency of the raters' scores, the level of consensus among raters, and how well raters' scores correlate with those given by sentiment analysis methods for randomly collected Twitter data.
Our analysis found that there is good validity for methods that measure negative and positive sentiments in short texts, both in terms of inter-rater consistency and when comparing raters to computer-mediated sentiment analysis methods. The more complex sentiment pair anger and joy had lower levels of inter-rater consistency. Rater-computer consistency for anger and joy was the weakest among the methods tested, raising questions about the construct validity for measuring expressions of anger and joy in short texts.
Level of Degree
Mathematics & Statistics
First Committee Member (Chair)
Fletcher G. W. Christensen
Second Committee Member
Erik Barry Erhardt
Third Committee Member
Sentiment, Sentiment Analysis, Polychoric, Polychoric Correlation
Kasper, Kelli N.. "Assessing the Validity of Sentiment Analysis Measures through Polychoric Correlation." (2020). https://digitalrepository.unm.edu/math_etds/174