Sentiment Analysis: Harder Than It Looks

“Sentiment analysis” sounds like a solved classification problem, and the demos support that impression. The difficulty is not in the model. It is that the label is underspecified, the phenomena that break it are structural, and the accuracy you see on a public dataset does not transfer to your inbox.

Three tasks under one name

Before choosing a method, notice which of these you actually want, because they have different labels and different ceilings.

• Document polarity. Is this review positive or negative? The task Pang, Lee and Vaithyanathan defined in Thumbs up? (EMNLP 2002) using movie reviews, and the one Stanford Sentiment Treebank (Socher et al., EMNLP 2013) refined with phrase-level annotations. Easiest of the three, and rarely the one a business needs.
• Aspect-based sentiment. What does this text say about the battery, the shipping and the price, separately? Defined by SemEval-2014 Task 4. This is almost always what a product team means, and it is a different problem: the answer is a set of pairs, not a label.
• Emotion or intent. Is the writer angry, is this churn risk, does this need escalation? These are business categories wearing a sentiment costume, and they have no public dataset because they are yours. Treat them as ordinary classification with your own labels.

Most disappointment comes from picking a tool built for the first while wanting the second or third.

Negation, and why counting fails

A bag-of-words model sees the room was not clean as the terms room, not, clean — and if your pipeline removed stop words, just room and clean. The strongest positive feature in the sentence survives; the word that reverses it does not. This is the most consequential interaction in the whole preprocessing debate and it is why the standard stop list containing not is not a trivia point.

Three fixes exist, in ascending order of cost. Bigrams capture not_clean as one feature and handle the common cases — Wang and Manning’s Baselines and Bigrams (ACL 2012) is the reference for how strong a simple bigram model with Naive Bayes features remains on this task. Explicit negation scoping tags every token between a negator and the next punctuation, which VADER (Hutto and Gilbert, ICWSM 2014) does as part of a rule-based system alongside intensifiers, capitalisation and emoji. And a contextual model reads the sentence, which is what makes transformers straightforwardly better here.

Note what the rule-based option buys: VADER is a lexicon plus a handful of grammatical rules, runs in microseconds with no model to load, and handles the specific constructions of short social text it was built for. On tweets and chat messages at high volume it is a completely reasonable production choice, and it is free.

Sarcasm is a labelling problem

“The model cannot detect sarcasm” is usually stated as a modelling limitation. The research suggests it is upstream of that. Wallace et al., Humans Require Context to Infer Ironic Intent (ACL 2014), found that human annotators frequently could not identify irony from the text alone and requested surrounding context before they would commit — and that the cases annotators needed context for were the same cases classifiers failed on.

SemEval-2018 Task 3 made irony detection its own shared task, which is itself the point: it is hard enough to warrant separate treatment rather than being folded into polarity. The practical consequence for a product is not “buy a better model”. It is that if your annotators disagree on a class of examples, no model will be reliable on that class, and your evaluation set cannot measure it either. Measure inter-annotator agreement before you measure a model.

Domain shift

Sentiment vocabulary is domain-specific in ways that look absurd once you see them. Unpredictable is praise for a plot and a defect in a car. Small is positive for a phone and negative for a hotel room. Sick and insane invert in some registers. A classifier fitted on movie reviews carries all of these the wrong way, and the domain-adaptation literature for sentiment goes back to Blitzer, Dredze and Pereira’s multi-domain Amazon review work (ACL 2007) for exactly this reason.

This is also why the accuracy number on a public leaderboard is close to useless as a prediction for you, and why an off-the-shelf sentiment API is a bigger gamble than it looks: you are inheriting someone else’s domain assumptions with no way to inspect them.

A protocol for your own data

Nobody can tell you what accuracy you will get, so here is the cheapest honest way to find out. It takes an afternoon and it is worth more than any benchmark.

• Write the label definition first, in one page. What counts as negative? Is a factual complaint negative? A question? A neutral status update? Most disagreement is definitional.
• Sample 200 documents at random from real traffic — not the interesting ones, random ones. Skewing the sample towards hard cases makes every later number meaningless.
• Have two people label them independently and compute agreement. If two humans agree only 80% of the time, 80% is your ceiling and any model reporting higher is fitting your annotator, not the task.
• Run every candidate against that set — a lexicon, a bigram linear model, a small fine-tuned encoder, a prompted model — and compare per-class recall rather than accuracy, because the class you care about is usually the rare one. The reasoning is in why accuracy lies under imbalance.
• Read every error. Two hundred documents makes this possible, and the errors cluster: negation, sarcasm, mixed sentiment in one document, or a definitional disagreement you can fix in the label guide rather than in a model.