Wake Words and Always-On Audio: False Accepts and the Privacy Design

A wake word detector is a model that runs continuously and is wrong in two directions: it wakes when nobody spoke, and it fails to wake when somebody did. The design problem is that these two errors are counted in different units, traded against each other by one threshold, and only one of them is visible to the user.

The cascade, and why there is more than one model

Nobody runs a good wake-word model continuously. The energy arithmetic forbids it: a model accurate enough to have a tolerable false-accept rate is too expensive to run ten times a second on battery. So the detector is a cascade, and each stage exists to keep the next one asleep.

The structure is the same duty-cycle argument that governs everything on microcontroller-class hardware: the cheap stage does not exist to be accurate, it exists so that the expensive stage is idle 99.9% of the time.

False accepts per hour, derived

Model evaluation reports a per-inference false-positive rate. Users experience false accepts per hour. The conversion is where most published wake-word claims become uninterpretable, and it is one multiplication:

inferences_per_hour = inference_rate_hz × 3600
false_accepts_per_hour = inferences_per_hour × p_false_positive

Worked, a stage-1 detector evaluating a 1-second window every 100 ms:

  inference_rate      10 Hz
  inferences_per_hour 10 × 3600 = 36,000

  p_fp = 1e-3  →  36 false accepts per hour   (unusable)
  p_fp = 1e-5  →  0.36 per hour               (about 9 a day)
  p_fp = 1e-7  →  0.0036 per hour             (about 1 a fortnight)

Two things fall out of this that are worth stating plainly. First, a per-inference false-positive rate that sounds excellent — one in a thousand — is thirty-six spurious wakes an hour, which is a broken product. An always-on detector needs error rates several orders of magnitude below what a normally-invoked classifier would need, purely because of the multiplier.

Second, the inference rate is a lever on the false-accept rate and people forget it is. Halving how often you evaluate halves false accepts per hour at no cost in model quality — at the price of latency, because the phrase may now sit unnoticed for longer. Overlap between windows is what keeps that latency acceptable, and choosing the stride is a real design decision rather than a default to accept.

Choosing the operating point

One threshold on the model’s score moves both errors in opposite directions. Sweeping it traces a curve, and the useful form plots false accepts per hour against the false-reject rate — the proportion of genuine utterances missed. Two facts about that curve decide the product:

• The two errors are not symmetric in cost. A missed wake is a small annoyance the user recovers from immediately by repeating themselves. A false accept in a quiet room is unsettling, and a false accept that sends audio somewhere is a privacy incident. Set the stage-2 threshold accordingly and let stage 1 be permissive.
• The curve is different for every population. Accents, background noise profiles, room acoustics, and the presence of a television all move it. An operating point tuned on clean recordings from your own team will not survive contact with a kitchen.

A frequently-underused option: make the threshold adaptive to context. Tighten it when the device is in a noisy environment or when a media player is running on the same device — the audio you are about to mistake for a wake word is being emitted by you.

The ring buffer is the privacy design

For a wake word to work, the audio containing it must already have been captured — you cannot start recording after you detect a phrase that has finished. So there is always a rolling buffer of recent audio in memory, and that buffer is the honest centre of the privacy question.

ring buffer size = seconds × sample_rate × bytes_per_sample × channels

  1 s at 16 kHz, 16-bit mono = 1 × 16,000 × 2 × 1 = 32,000 bytes
  2 s pre-roll                                    = 64,000 bytes

Design rules follow directly from what that buffer is:

1. Keep it as short as the detector needs. The pre-roll exists to give the verifier the beginning of the phrase, not to record the room. One to two seconds is the working range; more than that is capturing conversation you have no reason to hold.
2. Never persist it. The buffer is memory, overwritten continuously. Writing it to disk — even to a temporary file, even for debugging — converts a transient into a record, and records get backed up, synced and subpoenaed.
3. Draw the network boundary at stage 2, not stage 1. If stage-1 firings are streamed to a server for verification, your false-accept rate is now the rate at which room audio leaves the device, and that rate is the deliberately permissive one.
4. Make the indicator honest and unconditional. The indicator that audio is being captured should be driven by the audio path itself, not by application logic that can be wrong. A user who discovers the indicator was accurate but the explanation was not has still lost trust.
5. Give a real off switch. Not a setting that stops the feature responding while the microphone stays open — a switch that releases the audio input.

Doing all of this locally is the strongest position available, and it is the case where local processing wins on grounds other than cost without any argument being needed.

Measuring your own rates

Two corpora, measured separately, and neither substitutes for the other.

Negatives, for false accepts

Many hours of audio that contains no wake word but does contain everything else: conversation, television, music, kitchen noise, a podcast in which somebody says a phonetically similar phrase. Run the full cascade over it at your production inference rate and count firings. Report firings per hour with the corpus described in the same sentence — a number without its corpus is not a measurement.

Positives, for false rejects

Recordings of the phrase from a genuinely diverse set of speakers, at a range of distances, at a range of background noise levels, in rooms with different reverberation. Report the miss rate broken down by those factors rather than pooled. A pooled 3% miss rate hiding a 20% miss rate for one accent group is the failure mode that gets discovered publicly.

Re-run both on every model change, every threshold change, and every audio pipeline change — a different gain setting or a new noise suppressor upstream moves the operating point without touching the model at all.

What to tell users, and what not to claim

Three claims are commonly made and only two of them can be true at once.

• “The device is not recording.” False as usually meant. It is continuously capturing into a buffer; it is not retaining. The accurate sentence is the second one, and saying it plainly earns more trust than the first does.
• “Nothing leaves the device unless you say the wake word.” True only if verification is fully local. If a fired detection is sent for server-side confirmation, audio leaves on false accepts too — which is exactly when the user did not say anything.
• “It never makes mistakes.” No detector has a zero false-accept rate at a usable false-reject rate. Publishing your measured rate and its corpus is a stronger position than an absolute claim that a single user recording can falsify.