Audit trail architecture — decision record

Owner: Eng. Review cadence: revisit if the two-log split or the chain design needs to change. Last revisited: 2026-06-17 — the data-plane log gained the WORM hash-chain (see History below).

TL;DR

MeterBox ships two audit logs, by design:

Log	Where it runs	What it captures	Tamper-evidence	Read API
`/cp/audit` (control-plane)	central cluster Postgres `cp_audit_events`	tenant lifecycle: signup, login, MFA, invite, member role, org-delete, password reset, verify-email	WORM hash-chain (per-org SHA-256, `verifyChainForOrg`)	`GET /cp/audit`, `GET /cp/audit/verify`
`/v1/audit` (data-plane)	per-(org, env) pod's env-scoped storage	billing-ops: plan / customer / credit mutations with before-after diff	WORM hash-chain (env-scoped SHA-256, `verifyChain`)	`GET /v1/audit`, `GET /v1/audit/verify`, `GET /v1/audit/export`

The two logs deliberately use the same audit_events table name in different storage planes. They never collide (the data-plane store is env-scoped via envArrayStore); the shared name signals they're the same kind of record at different planes.

Why two logs

The two planes are physically separate processes in production:

The control plane owns identity/tenancy state (cp_* tables) in a central cluster Postgres shared across orgs.
A data plane runs in its own per-(org, env) pod with its own storage bundle (the in-cluster Postgres for that pod) — see the "customer data stays in cluster" rule in the project memory and README → Roadmap → P3 scale & on-prem.

A single shared audit log would force one of two architecturally lossy choices:

Data-plane writes directly to the central cp_audit_events — couples planes and gives every per-org pod the central Postgres creds. Breaks the "each pod sees only its own data" boundary the on-prem / sovereign tier is built on.
Data-plane HTTP-POSTs each mutation to a control-plane endpoint — adds a hot-path network dependency to every plan/credit/customer mutation and an extra availability requirement (mutations can't proceed if the control plane is unreachable).

Both options trade architecture cleanliness for the optics of "one log." Splitting is the cheaper choice: each log naturally lives where its data lives, and the dashboard can render both — /cp/audit for tenancy events, /v1/audit for billing-ops, no cross-plane writes in the critical path.

Both logs are hash-chained (WORM)

Control plane.

cp_audit_events is the compliance anchor the on-prem ladder leans on (SOC2 CC2.1 — see SOC 2 evidence map, and a hard requirement for HIPAA + FedRAMP). The per-org SHA-256 chain (prev_hash + hash over canonical content) makes any after-the-fact edit detectable by GET /cp/audit/verify, which walks the chain and reports the first broken_at row. Implementation: src/control-plane/services/audit.js (computeHash, verifyChainForOrg). Retention pruning is tolerated — the first surviving hashed row after a prune is a "post-prune genesis".

Data plane.

The billing-ops log (plan price changes, credit grants/revocations, customer status transitions) is chained the same way, with one simplification: a data-plane pod is a single (org, env), so the entire audit_events store is one chain — no per-org partition. verifyChain walks it and reports the first broken_at; GET /v1/audit/verify exposes it (admin) and the SPA Audit Log page shows an integrity badge. Implementation: src/billing-audit.js (computeHash, verifyChain). Same tolerances as the control plane (pre-chain legacy rows counted-not-chained; post-prune genesis).

Concurrent appends are safe. Both chains serialize the read-tail-through- insert under a per-chain in-process lock (src/lib/async-mutex.js) — keyed by env for the data-plane log, by org for the control-plane log — and generate the row's id + timestamp inside the lock, so insert order matches the chronological order verify walks. Two genuinely-parallel admin writes therefore can't fork the chain. The lock is process-local, which is sufficient: the data plane is one process per (org, env) pod and the control plane is a single writer per org. A multi-replica-per-env deployment would need a distributed lock (see async-mutex.js).

One known limit, shared by both chains:

Forward-only. Rows written before the chain shipped stay unhashed (counted as legacy_count); the chain protects everything written since.

History

The data-plane chain was initially deferred — v1 shipped append-only with no chain, on the rationale that the control-plane chain satisfied the anchor cert and billing-ops investigations didn't strictly need tamper-evidence. It was brought forward to close the SOC 1 ICFR "ledger/audit immutability" + SOC 2 CC7 evidence gap: billing-ops mutations directly affect customers' financial statements (metering → revenue, credits → revenue adjustments, plan price → invoiced totals), so an auditor asking for tamper-evidence on that surface is expected. The implementation ports the control-plane's proven computeHash / chain-verify rather than inventing a new scheme. Off-host NDJSON export (GET /v1/audit/export, oldest-first, optional since/until) shipped alongside, reaching parity with GET /cp/audit/export so the billing log can be shipped to a SIEM for retention + off-host tamper-resistance.

Cross-references

Roadmap entry that closed this scope: P3 · Platform & ops — billing-ops audit trail (now removed from the roadmap since shipped).
Data-plane service: src/billing-audit.js
Data-plane route: src/routes/audit.js
Control-plane service: src/control-plane/services/audit.js
Instrumented mutation surfaces: src/routes/catalog.js (plan create/update/publish/delete), src/routes/customers.js (customer create/plan-change/cancel/archive/unarchive/erase, credit grant/revoke).
Compliance evidence index: SOC 2 evidence map → CC2.1