Handling Schema Drift in a Sensor Data Pipeline

A firmware release changes a pressure field from kilopascals to hectopascals. The field name is unchanged, the type is unchanged, the values are still positive floats in a believable range. Every schema validator passes and every downstream number is wrong by a factor of ten.

Three kinds of drift, one of them dangerous

“Schema drift” covers changes with very different consequences, and separating them is the first useful step.

• Structural additive. A new field appears. Almost always harmless: consumers that do not know about it ignore it, and formats designed for evolution such as Avro and Protobuf handle it by construction. This is the case everyone designs for.
• Structural breaking. A field is removed, renamed, or changes type. Loud and immediate: parsers throw, jobs fail, someone is paged. Painful but self-announcing, and self-announcing failures are cheap.
• Semantic. The field keeps its name and its type and changes meaning — different units, different reference frame, different sign convention, a different sampling rate behind an unchanged field, a status code whose value 3 used to mean “warming up” and now means “fault”. Nothing fails. The data is quietly wrong for however long it takes someone to notice, and the historical record is corrupted for that entire period.

The third is the one this page is about, and it is endemic to sensor systems specifically, because device firmware is written by a different team on a different release cycle from the pipeline, and a units change looks locally like a tidy-up.

The units change, worked through

A weather station reports barometric pressure. Firmware 2.3 sends kilopascals; firmware 2.4, shipped to a third of the fleet on a Tuesday, sends hectopascals.

firmware 2.3   {"device":"ws-114","press":101.3,"temp":18.4}
firmware 2.4   {"device":"ws-114","press":1013.0,"temp":18.4}

JSON schema  : both valid, "press" is a number
type check   : both float
null check   : both present
range check  : if the bound is 0 < press < 2000, both pass
freshness    : unaffected

Downstream, a pressure-trend feature computed over the transition:
  before  101.3, 101.2, 101.4, 101.3
  after   1013.0, 1012.0, 1014.0
  the trend feature now reports a 900-unit step change
  the anomaly model fires for every device that updated
  the forecast model, which was fitted on kPa, produces nonsense

Two things about this are typical. The alert that fires is an anomaly alert on the data, not a pipeline error, so the first hypothesis is a real weather event or a sensor fault, and the investigation goes down the wrong path for a day. And because only part of the fleet updated, the aggregate across devices is a mixture of two units, which makes fleet-level statistics wrong in a way that is much harder to spot than a clean break would be.

Detecting a semantic change

Since the value stays valid, detection has to come from distribution and from metadata, not from validation.

Distributional checks per device and per version

Track summary statistics — median, interquartile range, and the fraction outside a tight expected band — per field, grouped by firmware version, and compare each new interval against the trailing baseline. A factor-of-ten shift in the median is enormous compared with natural variation and trips immediately, provided the baseline is grouped by version so a partial rollout does not average away. This is the sensor-specific instance of the general practice in data quality checks.

Physical plausibility, not just range

Range bounds set wide enough never to false-alarm are wide enough to admit the wrong units. Tighter physical constraints catch far more: sea-level barometric pressure on Earth stays within a few percent of 101 kPa, so a bound of 87 to 109 kPa is defensible from meteorology and rejects hectopascals instantly. Cross-field constraints are stronger still — dew point cannot exceed air temperature, power should approximate voltage times current, a total must equal the sum of its parts — because they are violated by unit changes in any single field.

Change-point detection on the fleet

A step change occurring simultaneously across many devices is almost never physical. Running a simple change-point test on the fleet median of each field, and correlating detected change points against the firmware deployment log, turns this class of bug into a report that names the release. It is the single highest-value check to build, because it catches semantic drift of any kind rather than of one anticipated kind.

The contract that prevents it

Detection is a mitigation. The prevention is to make the meaning explicit in the message so that a change to it is a change to the schema.

{
  "device": "ws-114",
  "fw": "2.4.1",
  "schema": 3,
  "readings": {
    "press": {"v": 1013.0, "u": "hPa"},
    "temp":  {"v": 18.4,   "u": "Cel"}
  }
}

Carrying an explicit unit per field means the pipeline converts rather than assumes, and an unrecognised unit string is a loud failure instead of a silent one. Using a controlled vocabulary matters here — UCUM, the Unified Code for Units of Measure, gives canonical strings so that Cel, degC and C do not become three independent bugs. The per-message overhead is real but small next to what it prevents, and it compresses to nearly nothing given how repetitive it is, as sensor stream compression covers.

The rest of the contract is organisational and equally necessary. A schema version integer that the pipeline can branch on. A registry where the current schema for each version lives, so producers and consumers refer to one artefact rather than to two copies of an understanding. And a compatibility rule enforced at registration: new versions may add optional fields, and any change to an existing field’s meaning requires a new field name. That last rule is the one that actually stops this class of bug, and it costs a firmware author one line of thought.

Handling drift you cannot prevent

• Normalise at the edge of the pipeline, once. A single ingest-time normalisation layer that maps every known version to one internal representation keeps version handling out of every downstream job. The alternative — each consumer doing its own conversion — guarantees that one of them is missed.
• Keep the raw message. When a conversion turns out to be wrong, the only recovery is to re-derive from what the device actually sent. Storing raw alongside normalised roughly doubles volume for a stream that compresses extremely well, and it is regularly the difference between a reprocessing job and permanent data loss.
• Fail loudly on unknown versions. A pipeline that silently passes through a schema version it does not recognise is choosing the worst option. Quarantine those messages and alert; unrecognised is a state you can recover from, mis-parsed is not.
• Record the firmware version as a column. If it is not in the data, no downstream analysis can condition on it and no change-point correlation can be run. It costs a few bytes and it is the field every incident investigation reaches for first.
• Backfill deliberately, with a marker. When you fix historical data, record which rows were corrected and by what rule. Models trained across a silently corrected boundary behave in ways nobody can explain a year later.