Handling Production Incidents: Runbook and Process

Production incidents follow a structured process for a reason — uncoordinated debugging during an outage wastes time, duplicates effort, and extends the time users are affected. The process is: detect, communicate, mitigate (not fix), investigate, resolve, and review.

Detect and communicate first

• The first action in an incident is not to open logs — it's to declare the incident, assign a commander, and open a communication channel (Slack incident channel, PagerDuty war room)
• The incident commander coordinates who investigates what; without clear ownership, five engineers all pull thread dumps while the load balancer misconfiguration causing the outage goes uninvestigated

Mitigate before fix

• Mitigation is the fastest way to restore service — it doesn't require understanding the root cause; revert the last deployment, increase replica count, disable a feature flag, increase a timeout, switch to a cached response
• Which means users may be unaffected within minutes, even before you know exactly what failed; the root cause investigation follows mitigation, not the reverse

The 5-minute rule

• If you've been investigating for 5 minutes without a clear direction, escalate or widen the scope — don't go deeper into one hypothesis while the real cause is elsewhere
• Look at recent deployments, recent configuration changes, and recent infrastructure events (cloud provider status page) before diving into application logs

Post-incident review (PIR)

• A blameless PIR within 48 hours captures: timeline of events, contributing factors, what worked, what didn't, and action items with owners and due dates
• The goal is to identify systemic issues — missing alerting, insufficient runbooks, inadequate testing — not to assign blame; the same failure happening twice means the PIR's action items weren't implemented