MeterBox — Managed (sovereign) tier operator runbook

Owner: MeterBox SRE. Review cadence: whenever a Managed-tier capability changes (KMS provider, audit transport, helm chart, break-glass model).

This document is the operator-facing companion to docs/on-prem.md. The on-prem doc tells the customer what each tier inherits; this runbook tells the SRE operating a Managed (sovereign) cluster (or the customer's security team reading along) how the three Managed-tier promises are actually run:

1. Single-tenant cluster ops
2. Audit-log streaming to customer SIEM
3. VPC peering + break-glass network access

The Managed tier is Self-Managed run by MeterBox SRE in the customer's sovereign region — same helm chart, same images, same APIs. What's different is who holds keys, where the cluster lives, and how operator access is audited. Everything below assumes that posture.

1. Single-tenant cluster ops

1.1 Region selection + bootstrap

The customer picks one region from the supported sovereign clouds (AWS GovCloud, GCP Assured Workloads, Azure Government, EU-only AWS, or on-prem with the customer's compute). MeterBox SRE provisions one EKS / GKE / AKS / k3s cluster per customer — single-tenant, never shared.

Bootstrap order:

1. Provision the empty cluster under the customer's billing account (the customer's cloud bill, not MeterBox's). Cluster name pattern: bp-<customer-slug>-<region>.

2. Mirror the images from ghcr.io/<owner>/meterbox/{control-plane, data-plane}:<tag> into the customer's image registry (ECR, GAR, ACR). GHCR images are private — pull secrets from the MeterBox ops keychain are used for the mirror step only; the cluster pulls from the customer registry afterward.

3. Install the helm chart with the Managed-tier values overlay:

   helm install meterbox ./deploy/helm/meterbox \
     --namespace meterbox --create-namespace \
     -f deploy/helm/meterbox/values.yaml \
     -f managed-tier-overlay.yaml
   

   The overlay sets security.networkPolicies.enabled=true, controlPlane.replicas=2, dataPlane.warmPool.enabled=true, and the image-registry overrides.

4. Wire KMS for backup-mirror keys. Set the per-org backup target to key_source: "kms" with the customer's KMS provider + key ARN (see src/control-plane/services/backup-mirror.js). The plaintext data-key never lives on disk; MeterBox only sees the wrapped DEK.

5. Bootstrap an initial owner account via the standard signup flow, transfer ownership to the customer's named operator, then have SRE step down to admin role (or remove entirely once the customer has multiple owners).

6. Hand over the customer-portal token-mint endpoint (POST /v1/customers/:id/tokens) so the customer can wire their own end-customer JWT minting if they plan to embed <CustomerPortal> (see sdks/customer-portal/).

1.2 On-call rotation

MeterBox SRE runs a 24×7 follow-the-sun rotation per Managed tenant — shared with general SRE for paging, but separate alert routing per cluster so one tenant's spike doesn't drown another's signal. Severities:

Runbooks per alert live in the MeterBox SRE wiki (out of scope for this document); links land in the alert payload itself.

1.3 Upgrade cadence

• Helm chart upgrades: monthly release train. Each release ships a tagged helm chart + matching image tag pair. Managed-tier upgrades are scheduled with the customer; we publish a release-notes diff 7 days ahead.
• Image-only patches: rolled out same-day for sev-1 security fixes, with customer-facing notification but no scheduling. The chart image.tag field is the only knob.
• Rollback: every release is reversible to the previous tag. Helm --atomic is on by default; failed upgrades restore in-place.
• Pre-upgrade checklist (SRE):
  1. Confirm a fresh encrypted backup landed within the last 24h (see § 1.4).
  2. Confirm warm pool has ≥1 standby data-plane slot in the target region.
  3. Drain non-essential cron jobs (sweepers, billing-close) for the upgrade window.

1.4 Backup + restore drills

Encrypted off-cluster backups run nightly via the sweeper (src/control-plane/services/backup-mirror-sweeper.js). Managed-tier customers run with key_source: "kms", so each snapshot carries a wrapped DEK + KMS key id on its manifest row. The plaintext DEK only exists in process memory long enough to encrypt one tarball.

Quarterly restore drill (mandatory, evidence for SOC 2 + auditors):

Full procedure: compliance/dr-restore-drill.md. Phases A–D (manifest → restore-token → cipher-byte download → decrypt) are automated by npm run dr:drill, which prints a structured report you can paste straight into the drill log template at compliance/dr-drills/template.md. Phases E–F (untar into a scratch PVC + bring up a read-only Postgres + verify a ledger_entries row matches the live cluster) are the operator's portion and run in a dr-drill-<yyyymmdd> namespace. Tear the namespace down after.

Commit the completed drill log under docs/compliance/dr-drills/<YYYY-Qn>.md within 5 business days. The committed log is the SOC 2 evidence (CC7.5 + A1.3).

If KMS Decrypt fails during a drill, treat as Sev-1 — the customer's KMS policy or MeterBox's IAM role has drifted and live restore would also fail.

1.5 Key rotation

The Managed tier has four key materials and three rotation cadences:

When METERBOX_RBAC_STRICT=true is on (Managed tier default — see § 1.6), admin-key rotation can be staged: keep the matrix tight on destructive actions during the rotation window so a leaked old key has the smallest possible blast radius.

1.6 Hardening posture (defaults)

Managed-tier overlay sets these defaults that don't ship by default in Self-Managed:

# managed-tier-overlay.yaml
controlPlane:
  env:
    METERBOX_RBAC_STRICT: "true"
    METERBOX_ORG_ID: "<customer-org-id>"   # pinned at bootstrap
    BACKUP_MIRROR_ENABLED: "true"
security:
  networkPolicies:
    enabled: true
  podSecurityStandard: restricted
dataPlane:
  warmPool:
    enabled: true
    minReady: 1

1.7 Incident response break-glass

SRE access to the cluster is never via long-lived kubeconfig handed to laptops. The pattern (§ 3 for the network shape):

1. SRE opens a ticket in the customer's ITSM (ServiceNow / Jira) describing the issue + the action they need to take.
2. Customer (or pre-approved automation) provisions a 4-hour-TTL session in their identity broker (AWS Identity Center, GCP CIEM, Azure PIM).
3. The session opens an audited SSH session to the in-region jumphost; from there kubectl against the cluster API.
4. Every command runs through the customer's audited shell (auditd or equivalent), which streams to the customer's SIEM (see § 2).
5. Session expires automatically; no standing access.

No exceptions — even Sev-1 mid-incident sessions go through the same path. If the broker is unreachable, the page escalates to the customer's CISO line rather than bypassing it.

2. Audit-log streaming to customer SIEM

The control-plane audit log carries every privileged action — logins, role changes, MFA events, backup-target mutations, RBAC-matrix edits, customer archival, credit grants/revokes, GDPR exports — with actor, IP, user-agent, and outcome. The chain is hash-evidenced (docs/compliance/audit-trail-architecture.md) so SIEM downstream can re-verify integrity if it wants.

2.1 Two export shapes

There is no streaming WebSocket / SSE push from MeterBox. The cron pattern keeps SIEM transports decoupled from the control-plane process and means a SIEM outage doesn't backpressure billing operations.

2.2 Pull endpoint contract

GET /cp/audit/export?since=<iso>&until=<iso> returns application/x-ndjson, oldest-first, one event per line. Owner/admin only. Window defaults to last 24h if no params supplied. Pagination is implicit via the window — page by shifting since/until. See src/control-plane/routes/auth.js for the full handler.

Each line is one event:

{"id":"01HZ...","timestamp":"2026-06-02T15:34:17.881Z","actor":{"kind":"session","user_id":"u_1","ip":"10.0.0.7"},"event":"credit.granted","target":{"customer_id":"cust_42","grant_id":"cg_1"},"diff":{"amount_usd":[null,25]},"outcome":"success","chain":{"prev":"<sha256>","self":"<sha256>"}}

2.3 Cron push pattern (Splunk HEC)

The recommended pattern is an in-cluster CronJob that pages the pull endpoint every 5 minutes with the prior window and POSTs each line to the customer's SIEM. Sample for Splunk HEC:

# audit-to-splunk.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: audit-to-splunk
  namespace: meterbox
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: shipper
              image: alpine/curl:8
              env:
                - name: BP_ADMIN_KEY
                  valueFrom: { secretKeyRef: { name: bp-admin, key: key } }
                - name: SPLUNK_HEC_URL
                  value: https://splunk.customer.internal:8088/services/collector/raw
                - name: SPLUNK_HEC_TOKEN
                  valueFrom: { secretKeyRef: { name: splunk-hec, key: token } }
              command: ["/bin/sh", "-c"]
              args:
                - |
                  SINCE=$(date -u -d '6 minutes ago' +%Y-%m-%dT%H:%M:%SZ)
                  UNTIL=$(date -u -d '1 minute ago' +%Y-%m-%dT%H:%M:%SZ)
                  curl -sS -H "x-admin-key: $BP_ADMIN_KEY" \
                    "http://control-plane.meterbox:3000/cp/audit/export?since=$SINCE&until=$UNTIL" \
                    | curl -sS -X POST "$SPLUNK_HEC_URL" \
                        -H "Authorization: Splunk $SPLUNK_HEC_TOKEN" \
                        --data-binary @-

The 1-minute trailing buffer accounts for clock skew between MeterBox and the SIEM; the 6-to-1 window means each event ships exactly once under normal operation and at-least-once on retry (Splunk dedups via id).

2.4 Cron push — Datadog Logs intake

Same skeleton, different POST target:

# diff vs Splunk version — env + args only
env:
  - name: DD_API_KEY
    valueFrom: { secretKeyRef: { name: datadog, key: api-key } }
args:
  - |
    SINCE=...; UNTIL=...
    curl -sS -H "x-admin-key: $BP_ADMIN_KEY" "$BP_EXPORT_URL?since=$SINCE&until=$UNTIL" \
      | jq -c '{ddsource:"meterbox",service:"control-plane",message:.}' \
      | curl -sS -X POST "https://http-intake.logs.datadoghq.com/api/v2/logs" \
          -H "DD-API-KEY: $DD_API_KEY" \
          -H "Content-Type: application/json" \
          --data-binary @-

2.5 Cron push — S3 (for Sumo Logic, Cribl, anything S3-source)

args:
  - |
    SINCE=...; UNTIL=...
    KEY="audit/$(date -u +%Y/%m/%d/%H%M).ndjson"
    curl -sS -H "x-admin-key: $BP_ADMIN_KEY" "$BP_EXPORT_URL?since=$SINCE&until=$UNTIL" \
      | aws s3 cp - "s3://customer-bp-audit/$KEY"

2.6 Retention math

Audit events are pruned in-cluster per the retention policy (configurable in SettingsPage → Security → Data retention). The default is 365 days, which is the SOC 2 minimum. The SIEM is the long-term system of record — operators should set the in-cluster retention to the shortest value that still covers the SIEM's ingest backfill window (typically 7–14 days), not the longest.

2.7 Quarterly verification

Once per quarter, run the SIEM-side count against the MeterBox-side count for the same 24h window. Any drift > 0.1% is investigated as a Sev-2.

3. VPC peering + break-glass network access

The Managed tier runs in the customer's VPC. MeterBox has no inbound control-plane connectivity to the cluster except through the customer's network — there is no MeterBox-owned VPN, no MeterBox jumphost, no phone-home. Three deployment shapes:

3.1 Shape A — Peered VPC (default)

The MeterBox SRE laptops sit in MeterBox's own ops VPC, which is peered to the customer's VPC via cross-account VPC peering (AWS), shared VPC (GCP), or VNet peering (Azure). Peering rules:

• Customer-side ACL: only the bp-ops-bastion security group can reach the customer's bp-jumphost SG on port 22.
• MeterBox-side: MeterBox's ops VPC is dedicated to SRE bastions only — no shared services, no databases.
• Peering is one-way for traffic initiation (SRE → customer); no reverse path from the customer cluster back into MeterBox.

3.2 Shape B — Transit Gateway

For customers running a multi-account / multi-region landing zone, peering goes through the customer's Transit Gateway (AWS) or Cloud Router (GCP). The TGW route table allowlists the MeterBox ops VPC CIDR; the customer's security team owns the routes and can revoke any time.

3.3 Shape C — Full air-gap

No peering, no internet egress. SRE access is via the customer's own operator portal (Apple Remote Desktop / Citrix / customer-managed SSH gate). MeterBox provides the runbook + scripted upgrade flow; the customer's ops team runs the commands. This is the only path that satisfies certain defense / classified-network requirements.

In this shape, image upgrades flow through a sneakernet pipeline: MeterBox publishes the new image tag + SHA, customer pulls into their air-gap image registry, then runs the helm upgrade.

3.4 Required egress (any shape)

The control plane has a small allowlist of egress destinations, all configurable and all optional:

There is no telemetry to MeterBox — no error reporting, no usage analytics, no version-check ping. Egress to MeterBox is impossible because no MeterBox endpoint is configured. This is the same posture as Self-Managed.

3.5 Network policies enforced in-chart

# from values.yaml — Managed-tier overlay enables both:
security:
  networkPolicies:
    enabled: true

The chart's network-policies.yaml defines:

• Default-deny ingress for all pods.
• Control plane allows ingress from the customer ingress controller on the configured port.
• Data plane allows ingress from the control plane only (same namespace, by pod selector).
• Postgres + Redis allow ingress from control plane + data plane only.
• All pods allow DNS egress to kube-system/coredns only.

A misconfigured ingress controller is the failure mode to watch — if a customer's ingress doesn't match the labels in the chart's allowlist, the control plane will fail liveness. Fix is to either relabel the ingress controller or update the chart's security.ingressSelector.matchLabels.

3.6 Operator audit trail

Every kubectl session from the jumphost is recorded by the customer's auditd (Linux) or equivalent and streamed to the customer's SIEM by the customer's own pipeline. MeterBox does not see SRE shell-history; the customer does. This is deliberate: it puts the customer in control of operator attribution evidence, which auditors prefer.

The MeterBox-side audit (§ 2) covers everything that happened through the APIs. The customer-side jumphost audit covers everything that happened outside the APIs (kubectl, helm, image rollouts). Together they form the full picture; either alone is incomplete by design.

4. Compliance cadences (the calendar)

Single calendar for every recurring commitment the policy set makes. Owners keep the logs in the linked locations; the Security lead reviews the calendar at each quarterly review.

Missed cadence → open a Sev-2; sustained miss (two consecutive quarters) → Sev-1 and a CEO-signed exception per the InfoSec policy § 12. The cadence log IS the SOC 2 evidence for CC1.5 / CC4.1 / CC7.4 / CC7.5 / CC9.2 / A1.3.

Cross-references

• docs/on-prem.md — customer-facing tier comparison + chart install
• docs/compliance/audit-trail-architecture.md — hash-evidence chain detail
• docs/compliance/soc2-evidence-map.md — how the runbook procedures map to SOC 2 controls
• docs/compliance/penetration-testing.md — quarterly external pentest scope
• deploy/helm/meterbox/values.yaml — every helm knob referenced above
• src/control-plane/services/backup-mirror.js — KMS envelope encryption + restore path