What Causes Wrong Sentiment Classifications of Game Reviews

Sentiment analysis is a popular technique to identify the sentiment of a piece of text. Although several techniques have been proposed, the performance of current sentiment analysis techniques are still far from acceptable and the causes of wrong classifications are not clear. In this paper, we study how sentiment analysis performs on game reviews. We report the results of a large scale study on the performance of widely-used sentiment analysis classifiers on game reviews. Then, we investigate the root causes for misclassifications and quantify the impact of each cause on the overall performance. We study three existing classifiers: Stanford CoreNLP, NLTK, and SentiStrength. Our results show that most classifiers do not perform well on game reviews, with the best one being NLTK (with an AUC of 0.70). We also identified four main causes for wrong classifications, such as reviews that point out advantages and disadvantages of the game, which might confuse the classifier. The identified causes are not trivial to be resolved and our suggestion to game developers is to prioritize the causes with higher impact on the sentiment classification performance. Finally, we show that training sentiment classifiers on reviews that are stratified by the game genre is effective.