Incident Response Plan

Owner: Security. Review cadence: annual or on material change. Tabletop cadence: quarterly.

This plan describes how MeterBox responds to security and availability incidents. It is the runbook on-call follows and the document an auditor reads to satisfy SOC 2 CC7.3 (incident evaluation) and CC7.4 (incident response). It pairs with vulnerability-disclosure-policy.md (how reports come in from outside) and audit-trail-architecture.md (forensic evidence).

Severity tiers

Pick the tier on observed impact, not suspected cause. Escalate, never de-escalate, until containment is confirmed.

Sev	Definition	Examples	Customer comms	Time to first ack	Time to resolution target
Sev0	Active customer-data compromise OR control plane down OR data-plane down for all tenants	Confirmed unauthorized read of another tenant's data; `/cp/*` 5xx for all; mass billing failure	Status page + email per tenant within 60m	15m	4h
Sev1	Single-tenant data plane down OR availability SLO breach OR active attack (no confirmed data exposure)	One tenant's `/v1/*` 5xx; rate of failed logins > 100/m; auth bypass under investigation	Affected tenants only, within 2h	30m	8h
Sev2	Material bug with no data/availability impact OR security finding requiring patch within 7d	Webhook signature mismatch on a low-volume tenant; medium-CVSS dep flagged by Trivy	Bundled into the next release note	4h business hours	7d
Sev3	Tracking-only — no customer impact	Internal lint failure, expired dev cert	None	next business day	next sprint

Roles (Incident Command structure)

One person fills each role. The same person may fill multiple roles on a small incident, but the assignment is explicit.

Incident Commander (IC) — makes decisions, owns timeline, runs the channel. Has authority to page anyone, pause deploys, roll back.
Communications Lead (CL) — drafts status page + customer email, posts to the incident channel, files the post-mortem stub.
Subject Matter Expert (SME) — does the technical work. Multiple SMEs allowed.
Scribe — keeps the running timeline in the incident channel.

On Sev0/Sev1, IC and CL are different people.

On-call rotation

Primary + secondary on-call, weekly rotation. Schedule lives in PagerDuty (or equivalent — the policy is rotation + escalation, not the tool).
Escalation: primary unacked at 5m → secondary; secondary unacked at 5m → engineering lead; engineering lead unacked at 5m → security lead.
Compensation: on-call is paid. No on-call without explicit consent and a documented handoff.

Decision tree

        ┌─ alert fires / report received
        │
        ▼
   1. Triage (≤15m for Sev0/1, ≤4h Sev2)
      • Reproduce / confirm impact
      • Pick severity (see table above)
      • Open #inc-<yyyymmdd-short-name> channel
      • Page IC, CL, SME per severity
        │
        ▼
   2. Contain
      • Stop the bleeding (revoke key, scale down, block IP,
        roll back deploy). Containment > root cause.
      • Confirm no further damage via the audit log + metrics.
        │
        ▼
   3. Eradicate + recover
      • Apply the real fix (patch, config change, rotation).
      • Verify by reproducing the failure path.
      • Re-enable any disabled surface.
        │
        ▼
   4. Communicate
      • Status page update for Sev0/1 (every 60m until resolved).
      • Customer email for Sev0; affected tenants for Sev1.
      • Internal channel + leadership update.
        │
        ▼
   5. Close
      • Mark incident resolved with the resolution criterion that was met.
      • Schedule the post-mortem (within 5 business days for Sev0/1).

Communications templates

Customer notification (Sev0/Sev1)

Subject: [MeterBox] Incident — <one-line summary>

What happened: <one paragraph, plain English>
When: <UTC window>
Customer impact: <what the customer saw / couldn't do>
What we've done: <containment + recovery so far>
What we're doing next: <next concrete step, with ETA>
Where to follow updates: <status page URL>
Questions: contact@meterbox.ai

We will follow up with a post-mortem within 5 business days.

Internal incident-channel topic

#inc-<yyyymmdd-name>
SEV: <0|1|2|3> · IC: @<handle> · CL: @<handle> · SME: @<handle>
Status: <triage | contain | eradicate | recover | resolved>
Last update: <UTC timestamp>
Status page: <URL or "N/A">

Post-mortem (blameless)

# Post-mortem — <incident name>

- **Severity:** <Sev0|1|2|3>
- **Detected at:** <UTC>     **Resolved at:** <UTC>     **Duration:** <h:mm>
- **Detection source:** <alert | customer report | internal observation>
- **Customer impact:** <tenants affected, customer-visible behavior, dollar/SLA impact>

## Timeline (UTC)
<bulleted; one line per state change>

## What went well
<bullets — keep this section, it's where the next IC learns>

## What went poorly
<bullets — process, tooling, knowledge gaps. NOT individuals.>

## Root cause
<one paragraph: the proximate cause AND the systemic cause that allowed it>

## Action items
| # | Action | Owner | Due | Linear/issue |
| - | ------ | ----- | --- | ------------ |

Action items must be tracked to closure. The IC reviews open action items at the start of each weekly engineering sync.

Tabletop exercise log

We run a tabletop exercise quarterly to keep the muscle warm. The log is the SOC 2 evidence that the cadence is met.

# Tabletop — <YYYY-Qn>

- **Date:** <YYYY-MM-DD>     **Facilitator:** @<handle>
- **Scenario:** <one paragraph — e.g. "Tenant reports their `/v1/customers` list contains a different tenant's customers">
- **Participants:** @<handle>, @<handle>, …

## Observations
<bullets — where did the plan hold up, where did it bend>

## Action items
| # | Action | Owner | Due |
| - | ------ | ----- | --- |

Logs live in docs/compliance/tabletops/. The most recent four (one year) are the SOC 2 evidence.

Related docs

vulnerability-disclosure-policy.md — how external reports come in.
penetration-testing.md — annual external assessment + findings track.
audit-trail-architecture.md — append-only, hash-chained /cp/* audit log + GET /cp/audit/verify.
soc2-evidence-map.md — Common Criteria → evidence mapping.