Disaster Recovery — Restore Drill

Owner: Security + SRE. Cadence: quarterly. Required evidence for SOC 2 CC7.5 + A1.3.

This is the procedure for a Managed-tier restore drill — proving that the encrypted off-cluster backups recorded by src/control-plane/services/backup-mirror.js can actually be turned back into a queryable database within the published RTO. It pairs with the incident response plan (which covers the escalation path when a drill fails) and the audit-trail architecture (which covers the forensic evidence captured along the way).

Definitions

RPO (Recovery Point Objective) — the maximum acceptable data loss, measured as the age of the most recent successful backup at the moment a drill is initiated. Backups run nightly; RPO target ≤ 24 hours.
RTO (Recovery Time Objective) — the maximum acceptable elapsed time between a restore being requested and a read-only Postgres being available for inspection. RTO target ≤ 60 minutes for one tenant; the procedure is per-tenant and embarrassingly parallel across tenants.
Drill — the per-quarter exercise that runs the procedure end-to-end against a real backup, measures both numbers, and logs the result under docs/compliance/dr-drills/<YYYY-Qn>.md.

Scope

In scope for the drill:

One tenant's most recent successful backup (chosen at random from the last 30 days).
Manifest fetch, restore-token mint, cipher-byte download, KMS decrypt (when key_source: "kms"), AES-256-GCM authenticate + decrypt, untar into a scratch PVC, read-only Postgres bring-up, ledger sanity check, tear-down.
Wall-clock measurement of each phase.

Out of scope:

Cutting traffic over to the restored environment (that's a fail-over exercise, not a restore drill).
Restoring more than one tenant in a single drill.
Re-encrypting and re-uploading after the drill — the scratch namespace + its bytes are deleted at the end.

Procedure

The first four phases are automatable via the npm run dr:drill helper (see scripts/dr-drill.js). The remaining phases require a real Kubernetes namespace.

Pick a tenant. From the operator workstation, list tenants with a configured backup_target. Pick one at random whose most recent backup is < 30 days old. Record the slug + plan in the drill log.
Phase A — manifest fetch. GET /cp/backups (owner session). Confirm the most recent row exists and that key_fingerprint matches what the org currently has configured. Capture created_at → this is the observed RPO for the drill.
Phase B — restore-token mint. POST /cp/backups/:id/restore-token. Confirm the token expires_at is in the future. Start the RTO clock here.
Phase C — cipher-byte download. Pull the cipher bytes from the configured transport (s3 / gcs / azure / stub). Time + record bytes transferred. Failure here typically means the bucket policy drifted or the transport credentials rolled.
Phase D — decrypt.
- key_source: "kms" — call the configured KMS provider's Decrypt on the wrapped DEK. Failure here is the realistic failure mode in production: customer-side KMS IAM drift or MeterBox IAM role drift. Treat any KMS Decrypt failure as Sev-1 — see the incident response plan — because a live restore would also fail.
- key_source: "static" — confirm the in-cluster secret matches the stored key_fingerprint.
- In both modes: AES-256-GCM authenticate + decrypt the blob using the IV
  - auth tag from the manifest.
Phase E — untar. Untar the plaintext into a scratch PVC mounted in dr-drill-<yyyymmdd> namespace. Confirm the file tree shape matches the bundled-Postgres layout (base/, pg_wal/, postgresql.conf, …).
Phase F — read-only Postgres + ledger sanity. Start a postgres:16 pod with the scratch PVC mounted at /var/lib/postgresql/data. Once ready, pick a recent ledger_entries row from the live cluster, query the same (customer_id, timestamp) in the restored database, confirm the cost + token totals match. Stop the RTO clock when the row matches. This is the measured RTO.
Tear down. Delete the namespace + PVC. Confirm no residual snapshot bytes remain outside the customer's bucket.
Log the drill under docs/compliance/dr-drills/<YYYY-Qn>.md using the template in template.md. The structured output from npm run dr:drill covers phases A–D verbatim; phases E–H are filled by the operator running the destination side.

Success criteria

A drill is passing when all of the following hold:

Observed RPO ≤ 24 hours.
Measured RTO ≤ 60 minutes.
KMS Decrypt succeeded (when key_source: "kms").
AES-GCM authenticate succeeded — no integrity mismatch.
Ledger row in the restored database matches the live cluster.
No Sev-1 follow-up was opened during the drill.

A drill is failing when any of the above does not hold. A failed drill opens an incident under incident-response.md at the severity matched to what would happen if this were a real outage — usually Sev-1.

Cadence + evidence

One drill per quarter, run by SRE with a Security observer.
Drill log committed within 5 business days. The log IS the SOC 2 evidence for CC7.5 (tested RPO/RTO restore drill) and A1.3 (restore drill with measured RPO/RTO).
The most recent four logs (one year) are surfaced to the auditor.