Incident Severity, SLOs, Error Budgets

Incident management is more than responding to pages. It's the organizational framework that decides which incidents get which response, how much reliability to invest in, and when to halt feature work to fix stability.

Severity tiers

• SEV-1: major outage, critical path unavailable, significant user or revenue impact; full response, IC plus Comms plus SMEs, leadership paged, status page update within 15 minutes, postmortem mandatory
• SEV-2: degraded, subset of users or features; business-hours full response, best-effort overnight
• SEV-3: minor or internal; single engineer, business hours
• SEV-4: near-miss, auto-recovered; documented only
• Key: tie severity to customer impact, not engineering judgment about complexity

SLOs are customer commitments

• Service Level Objective: explicit reliability target; example: 99.9% of login requests succeed within 1 second, measured over 28 days
• Four parts: SLI (the metric), target (99.9%), time window (28 days), user journey (login)
• SLI must measure what users care about: 'checkout-request success rate' not 'CPU utilization'

Error budget derives from SLO

• 99.9% SLO = 0.1% error budget = 43 minutes per month of badness allowance
• Any combination of downtime, slow requests, errors consumes the budget
• When budget depletes faster than budget regenerates, that's a signal

Burn-rate alerts are better than raw thresholds

• Raw threshold 'error rate > 1%' fires on brief spikes; noisy
• Burn rate: 'burning at 14x normal rate for 5 minutes' fires only on sustained problems
• Multi-window pattern: both 1-hour and 5-minute windows must show high burn; reduces false positives
• Google SRE workbook has canonical examples

Error budget policy is the organizational teeth

• Over 50% remaining: ship features normally
• 25 to 50% remaining: extra review on risky changes, reliability work in sprint
• 10 to 25% remaining: feature freeze for risky changes, manual approval for deploys
• Under 10% remaining: full feature freeze, all engineering on reliability
• Budget exhausted: mandatory freeze until budget regenerates, executive review
• This policy requires product leadership buy-in; otherwise it's paper

The cultural shift SLOs bring

• Reliability becomes a shared decision between engineering and product
• 'Is this outage okay?' becomes a math question, not a feelings question
• Product accepts feature velocity trade for reliability when budget is tight
• Biggest win: moves reliability from 'engineering's job' to 'everyone's trade-off'

99.999% is rarely worth it

• 5-nines = 5.26 minutes per year of downtime
• Multi-region active-active, extreme redundancy, massive investment
• Most businesses need 99.9 to 99.95%; match SLO to business value and user expectations
• Pick achievable SLOs; tighten as engineering maturity grows

SLO vs SLA

• SLA is external contract ('99.9% uptime or refund')
• SLO is internal target, often stricter (99.95% internal for 99.9% external)
• Buffer absorbs unexpected badness before SLA breach

Practical starting points

• Identify top 3 to 5 critical user journeys
• Define SLI for each: success rate, latency p99, freshness
• Set SLOs at current realized performance plus small buffer
• Deploy burn-rate alerts
• Publish error budget dashboard
• Document error budget policy with product leadership signature
• Iterate: tighten SLOs as capability grows

The strategic answer

• Severity tiers standardize response to incidents
• SLOs make reliability a measurable commitment
• Error budgets convert reliability into a trade-off vs feature work
• Burn-rate alerts detect sustained problems without noise
• Error budget policy gives the framework teeth
• Executive buy-in is prerequisite; otherwise policy is paper