Classifying Speaker Emotion From Non-Verbal Vocal Cues

“Fine.” The word is neutral. Whether it means agreement, resignation or fury is entirely in the pitch contour, the intensity and the timing, and none of that survives transcription.

Two channels carrying different information

Speech carries affect in two largely separable ways, and mixing them up is the most common design error in this area.

The lexical channel is what a transcript preserves: word choice, negation, intensifiers, topic. Text sentiment analysis works on this, and it is genuinely informative — people say what is wrong. It is also where sarcasm lives as an unresolvable ambiguity, because the words are the opposite of the meaning by construction.

The paralinguistic channel is everything transcription discards: how high the voice is and how much it moves, how loud, how fast, where the pauses fall, how breathy or creaky or tense the phonation is, and the non-verbal vocalisations — sighs, laughs, sharp intakes of breath — that most transcription systems drop entirely.

The useful generalisation is that the two channels are good at different halves of the problem. Prosody separates arousal — activated from calm — very well, because loudness, pitch range and speech rate all rise with activation and they rise together. Prosody separates valence — pleasant from unpleasant — poorly, because hot anger and elation look remarkably similar on all of those measures. The words separate valence far better than they separate arousal. A system predicting both from audio alone is doing the easy half well and the hard half badly, and its aggregate accuracy hides which is which.

The parameters that carry affect

The field converged on a standard descriptor set rather than everyone inventing their own. The extended Geneva Minimalistic Acoustic Parameter Set — eGeMAPS, from Eyben and colleagues, published in IEEE Transactions on Affective Computing in 2016 and available as an author preprint — specifies 88 parameters, chosen for theoretical grounding and reproducibility rather than for raw accuracy, and implemented in the openSMILE toolkit. Its groups are worth knowing even if you never use the set directly, because they are the vocabulary:

• Frequency-related. Fundamental frequency F0 and its statistics, jitter (cycle-to-cycle variation in period), formant frequencies and bandwidths. F0 mean and range are the workhorses of arousal.
• Energy and amplitude. Loudness and its dynamics, shimmer (cycle-to-cycle variation in amplitude), harmonics-to-noise ratio. HNR falls as phonation becomes breathy or harsh.
• Spectral. Alpha ratio, Hammarberg index, spectral slope, and harmonic differences such as H1-H2 and H1-A3. These describe voice quality — the difference between a tense voice and a relaxed one at the same pitch and loudness.
• Temporal. Rate of loudness peaks, the lengths of voiced and unvoiced regions, continuous voiced regions per second. This is where speech rate and pausing enter.

Modern systems mostly do not hand these features to a classifier. They fine-tune a self-supervised speech encoder on labelled emotion data and outperform the descriptor set. But eGeMAPS remains the right thing to compute alongside, for two reasons: it is interpretable, so you can say which acoustic property drove a decision, and it is a fixed reference, so a drop in performance can be traced to a shift in the input distribution rather than guessed at.

One preprocessing point does most of the work in practice. Absolute F0 is dominated by who is speaking, not how they feel — the difference between two speakers is far larger than the difference between one speaker calm and agitated. Without per-speaker normalisation, a model learns to identify speakers and inherits their base rates of labelled emotion. Normalise F0 and loudness against a per-speaker baseline drawn from neutral speech before anything else.

Categories, dimensions, and which to predict

Two label schemes are in use. Categorical schemes assign one of a small set of discrete emotions — anger, happiness, sadness, neutral is a common four. Dimensional schemes place an utterance in a continuous space, usually valence, arousal and dominance, each on a bounded scale.

Dimensional labelling is the better target for most engineering purposes, because it degrades sensibly. A categorical model forced to choose between anger and happiness on a high-arousal utterance is either right or wrong; a dimensional model reports high arousal and uncertain valence, which is both true and actionable. Categorical schemes also embed a theoretical commitment to a fixed emotion inventory that is genuinely contested across cultures, and the inventory silently varies between corpora, so a model trained on one set of categories cannot be evaluated on another without a mapping somebody invented.

Whichever you use, inter-annotator agreement is the ceiling. Human raters disagree substantially about emotion in speech, so the labels are aggregates of disagreement, and a model reported as more accurate than the annotators agree with each other is measuring the aggregation procedure. Report agreement alongside accuracy or the accuracy figure has no scale.

The validity problem nobody has solved

The corpora that made this field possible are largely acted. In IEMOCAP and RAVDESS, among others, performers produce utterances in specified emotional states, which gives clean audio, balanced classes and unambiguous labels. It also gives portrayals rather than experiences.

Acted emotion is more extreme, more prototypical and more categorically separable than the spontaneous kind. A model trained on it learns the performance convention, and the standard evidence for this is that cross-corpus performance collapses: a model that scores well within one acted corpus loses a large fraction of that when evaluated on another corpus, let alone on spontaneous speech. Any reported figure that comes from a within-corpus split should be read as an upper bound that will not survive contact with real recordings.

Underneath that sits a deeper issue. There is no ground truth for another person’s internal state, only a label somebody assigned from the outside. The measurable object is a vocal expression, and the relationship between expression and felt emotion is variable across individuals, situations and cultures. That is a limitation of the construct, not of the model, and no amount of training data addresses it.

Where this can and cannot be used

The defensible uses share a shape: aggregate rather than individual, advisory rather than determinative, and evaluated against an outcome you can observe.

• Aggregate trend on a call population. Arousal averaged over thousands of support calls, tracked over weeks, is a reasonable signal about a process. It does not require being right about any individual call.
• Retrieval and triage. Surfacing the most high-arousal segments of a long recording for a human to listen to is useful even at moderate precision, because a person makes the actual judgement.
• Not individual assessment. Inferring a person’s state to make a decision about them — hiring, discipline, credit, clinical assessment — asks the model for a level of individual validity the evidence does not support, and the construct problem above means better models will not fix it.
• Not a health signal without clinical validation. Voice quality does change with some conditions, and that is an active research area. Any such use is a clinical claim requiring prospective validation and, in most jurisdictions, regulatory review. The same standard applies to respiratory sound classification, where the modelling is more tractable and the approval burden is no lighter.

The honest framing for almost every product here is that the system measures vocal arousal, that this correlates with but is not identical to emotion, and that a human remains in the loop for anything consequential. A system described that way survives scrutiny. One described as detecting how a customer feels does not.