Backups and Restore for AI Data

Half the data in an AI system is derived and should be rebuilt rather than restored. The half that is not derived has one specific trap: a logical dump of a table with an HNSW index does not contain the index, it contains the statement that recreates it — so your restore time silently includes a multi-hour index build nobody put in the plan.

Back up, rebuild, or accept the loss

Start by classifying every store, because backing up derived data is expensive and backing up nothing is worse.

The row that people get wrong is embeddings. They are derived, so the instinct is to skip them. But the derivation costs money per token and depends on a third party continuing to serve a specific model version — and changing embedding models mid-restore means your restored index is not comparable to anything you had before. Back them up.

The second row hides a dependency worth auditing separately. Calling something derived is a claim that you can derive it again, and that claim has three preconditions people rarely check: the extractor version is pinned and still installable, the chunking parameters are recorded rather than living in a constant somebody has since changed, and the embedding model is still served by the provider under the same identifier. Break any one and the data was never derived; it was unique and you were not backing it up.

-- The audit, as a query. Every distinct value here is a dependency
-- you are asserting you can reproduce. If any of them names a model
-- or a library version you can no longer obtain, that data is not
-- derived and belongs in the backup.
SELECT extractor, count(*) FROM document_versions GROUP BY 1;
SELECT chunker,   count(*) FROM chunks             GROUP BY 1;
SELECT model,     count(*) FROM chunk_embeddings   GROUP BY 1;

Providers deprecate embedding models on a timescale of a year or two, and a deprecated model is not merely inconvenient — the vectors it produced cannot be regenerated, cannot be extended to new documents, and cannot be mixed with a successor’s output. That makes an embedding table a hostage to somebody else’s roadmap, which is a reason to back it up and also a reason to keep the source text so that a forced migration is possible at all.

The pg_dump trap

pg_dump produces a logical backup: CREATE TABLE, data, then CREATE INDEX. It is portable across Postgres versions, restorable table by table, and small. It also has two properties that are specific to vector tables and that will surprise you.

The index is not in the dump. Only the statement that builds it. Restore a table of fifty million vectors and you get the rows quickly and then wait hours for CREATE INDEX … USING hnsw to finish, during which search does not work. Your recovery time objective must include that build, calculated with the scaling law from choosing and tuning a pgvector index.

The default format is text, and vectors in text are large. A float32 written as decimal text is around ten to twelve bytes rather than four, so a plain-text dump of an embedding table can be two to three times the size of the data. Use the custom or directory format, which compresses and allows parallel restore:

# Directory format, 8 parallel workers, compressed.
pg_dump -Fd -j 8 -Z 6 -f /backup/app-2026-08-04 app

# Restore, also parallel. The index builds happen here and dominate.
pg_restore -d app -j 8 /backup/app-2026-08-04

# Restore data first and defer the indexes so the application can
# start on exact search while the HNSW build runs:
pg_restore -d app -j 8 --section=pre-data --section=data /backup/app-2026-08-04
pg_restore -d app -j 4 --section=post-data /backup/app-2026-08-04

That last pair of commands is the trick worth remembering. Loading pre-data and data gets you a working, correct, slow database — exact search over an unindexed table is the plan from the pgvector guide and it returns the right answers. Post-data adds the indexes afterwards. Your service is degraded rather than down for the several hours the build takes, which is usually the difference between an incident and a catastrophe.

Physical backups and point-in-time recovery

A physical backup copies the data directory, index pages and all. It restores at the speed of your disks with no index rebuild, which for a vector database is the decisive advantage.

# Base backup, streamed and compressed.
pg_basebackup -D /backup/base-2026-08-04 -Ft -z -X stream -P

# With continuous WAL archiving configured in postgresql.conf:
#   archive_mode = on
#   archive_command = 'test ! -f /wal/%f && cp %p /wal/%f'
# you can then recover to any moment covered by the archive:
#   restore_command = 'cp /wal/%f %p'
#   recovery_target_time = '2026-08-04 14:22:00+00'

Point-in-time recovery is the only defence against the failure that actually happens, which is not a disk dying — it is a migration or a delete that destroyed data at a known moment. A nightly dump gives you “yesterday”. WAL archiving gives you “the second before the bad statement ran”, and the difference is a day of customer work.

The trade-off against logical backups is real: a physical backup is tied to the Postgres major version and the platform, cannot restore one table, and is much larger — it includes your indexes, which the arithmetic in storing embeddings shows is about half your total bytes. Most teams running vectors at scale end up with both: physical for recovery, logical for portability and for extracting a single table.

Object storage and weights

Buckets are durable, which is not the same as backed up. Durability protects against hardware failure. It does nothing about a delete, an overwrite, or a credential that leaked.

• Versioning on, on every bucket holding source data. An overwrite becomes a new version and the old one is recoverable. Add a lifecycle rule expiring noncurrent versions after 30 to 90 days, or versioning becomes an unbounded bill.
• Object lock for anything with a retention obligation. Compliance mode prevents deletion by anybody, including the account root, for the retention period. This is the only real defence against a compromised credential deleting your history — and it is also irreversible, so set the period deliberately.
• Replicate weights to a second region. Not for durability but for availability: a regional outage during a scale-up is precisely when you need to pull weights and precisely when you cannot.
• Record the checksum in the database. A restored object that is silently truncated is worse than a missing one. The sha256 column from object storage for documents and weights is what makes verification a query.

The restore drill

An untested backup is a hypothesis. The drill is the experiment, and it should be scheduled — quarterly is a reasonable cadence — and timed, because the number you need is not “did it work” but “how long did it take”.

1. Provision a scratch instance from nothing, the way you would in an incident. If this step requires a person who is on holiday, that is the finding.
2. Restore the most recent backup without consulting anyone who wrote the backup script. Start a timer.
3. Record the time to first correct query — data restored, indexes still building — and separately the time to full performance. These are two different numbers and both belong in your runbook.
4. Verify content, not just row counts. Counts match when a column restored as nulls.

   -- Row counts and null checks
   SELECT count(*) AS rows,
          count(*) FILTER (WHERE embedding IS NULL) AS null_vectors
   FROM chunk_embeddings;
   
   -- Content fingerprint: same on both instances, or something is wrong.
   SELECT md5(string_agg(content_sha256::text, ',' ORDER BY id))
   FROM chunks;
   
   -- And a real retrieval, compared against production for the same query.
   SELECT chunk_id FROM chunk_embeddings
   ORDER BY embedding <=> $known_probe LIMIT 10;

5. Write down what broke and fix the runbook, not just the instance. Every drill finds something: an extension that was not installed, a role that did not exist, a search_path assumption, an object store credential the restored app did not have.
6. Destroy the scratch instance. A half-restored copy of production data left running is a data breach with a delay on it.

Putting numbers on RPO and RTO

Recovery point objective is how much data you accept losing; recovery time objective is how long you accept being down. Both are decisions, not facts, and both should be written down per store because they differ enormously.

Store               RPO                      RTO
------------------  -----------------------  -------------------------
Postgres (rows)     seconds, via WAL         hours, incl. index rebuild
Vector index        n/a — derived            = index build time
Object store        zero, versioned          minutes
Fine-tuned weights  = last training run      minutes if replicated
Cache               total loss acceptable    zero — it refills
Queue               zero — jobs are user     minutes; needs AOF
                    work                     persistence

The line that usually forces a design change is the second one. If your RTO is one hour and your HNSW rebuild is four, no backup strategy fixes it — you need either a physical backup that ships the index, or a standby replica that already has one. Deriving that build time from your own row count before the incident is the point of doing this arithmetic at all.

Write these numbers into the runbook next to the commands, not into a strategy document. The person restoring at three in the morning needs to know that the index build takes four hours and that the two-command split in the section above gets search working at minute twenty — because the alternative is that they watch a progress bar for an hour before someone thinks to ask whether the service could be running degraded in the meantime. A recovery objective nobody can act on during the recovery is a number for an audit, not a plan.