Incident Response Process: Page to Resolution

During a production incident, the instinct is to dive into logs and try things. That works for trivial issues. For real incidents, especially those over 30 minutes or affecting customers, structure wins over heroics.

The Incident Command System — roles first

• Incident Commander coordinates the response, makes decisions, prevents chaos — does NOT debug
• Communications Lead updates the status page, internal channel, briefs customer support
• Subject Matter Experts investigate and fix — database expert, K8s expert, service owner
• Scribe documents the timeline in real-time for long incidents
• #1 failure mode: senior engineer becomes IC and also debugs; now no one is coordinating

The phases end-to-end

Detection and acknowledgment

• Target MTTA is 5 minutes; slower suggests alerting or on-call process is broken

Severity assessment

• SEV-1: major outage, users can't use the product, revenue impact — full response, wake up on-call, page leadership
• SEV-2: degraded, subset of users or features — full response during business hours
• SEV-3: minor nuisance — business-hours fix, single engineer
• Start conservative; severity can escalate

Channel and roles

• Open dedicated incident channel for SEV-1 and SEV-2
• Pin key info: IC, Comms, SMEs, status page link, severity, customer impact, start time
• IC declares themselves explicitly: 'I'm IC. Bob is Comms. Let's go.'

Mitigate first, root-cause later

• Most important discipline in incident response
• If deploy-correlated, roll back before you know what's broken
• Priority is stopping the bleeding; root cause is 10x easier with no fire
• Common mitigations: rollback, scale up, failover, circuit-break, feature flag off, DNS shift, restart

Communication in parallel

• Internal channel updates every 15 minutes minimum, even if no progress
• External status page within 15 minutes of SEV-1 confirmation
• Format: 'Investigating' first, 'Monitoring' when mitigated, 'Resolved' when stable
• Comms Lead briefs customer support with talking points; they handle user questions
• Silence kills coordination and trust

Root-cause investigation

• After mitigation, SMEs dig in
• Capture artifacts: logs, thread dumps, heap dumps, query plans, deploy diff
• These become postmortem evidence

Resolution

• Metrics back to baseline
• Held for 15 minutes (prevent premature closure)
• No new customer reports
• Comms posts 'Resolved'

Hand-off for long incidents over 4 hours

• IC explicitly hands to next shift
• New IC reads channel history, old IC verbally briefs
• Prevents burnout; standard SRE practice

Postmortem within 5 business days

• Blameless, systemic learning

The metrics to track

• MTTD: time to detect
• MTTA: time to acknowledge
• MTTR: time to resolve
• Each targets different improvements; measure separately

Common pitfalls

• IC also debugging; responses fragment
• Silence in the channel; everyone assumes the worst
• No status page update; customer trust erodes
• Rolling back blindly when DB migration isn't reversible
• Hero mode 10-hour shifts; on-call rotation is broken or system too fragile
• Pivoting to wrong hypothesis in parallel investigation; IC should assign hypotheses to SMEs to investigate in parallel, not sequential

Practice matters

• Game days: chaos engineering exercises (shut down an AZ, kill a service)
• Teams practice IC roles and tooling in low-stakes scenarios
• First-time IC during a real SEV-1 is a bad time to learn the tool

The strategic answer

• Structure over heroics
• Roles explicit; IC not debugging
• Mitigate before root-cause
• Comms every 15 minutes internal, within 15 minutes external
• Hand-off long incidents; don't hero
• Postmortem to turn pain into learning