Debugging a P99 Latency Spike with Observability Tools

The alert wakes me at 2 AM — p99 latency on payment-service jumped from 200ms to 2 seconds for 10 minutes. What I do in the next 10 minutes determines whether this is a 15-minute incident or a 2-hour incident.

Step 1: Confirm the alert is real

• Open the main service dashboard; is p99 actually sustained, or was it one flaky request?
• Confirm: 'yes, sustained at 1.5 to 2 seconds; baseline 200ms'

Step 2: Check the deploy timeline

• Open the deploy dashboard; did we ship payment-service in the last 30 minutes?
• If yes, consider rollback first, investigate after; the SRE best practice when there's clear deploy correlation
• If no, it's infra or downstream; don't rollback blindly

Step 3: APM tells me which endpoint is slow

• Open Datadog or New Relic; look at service overview
• Is p99 elevated across all endpoints, or only one?
• All endpoints: likely infra — DB saturated, GC pressure, noisy neighbor
• One endpoint: localized regression, much faster to debug

Step 4: Drill into a slow trace

• Sort traces by duration; pick one at 1.8 seconds
• Look at the waterfall; find the fat span
• Common patterns: DB span eating the time; downstream API call eating the time; many small spans summing up (GC or CPU); gap between spans (thread pool starvation)

Step 5: Check component metrics for the slow span

• If DB: CloudWatch RDS metrics, pg_stat_statements, lock contention
• If downstream: check their dashboards, are they slow for everyone or just us, ping their oncall
• If GC: JVM heap metrics, gc pause duration, is heap near max
• If thread pool starved: thread dump reveals blocking calls

Step 6: Compare with baseline

• Overlay 'now' vs '24 hours ago' in the dashboard
• Did DB query count per request change? Did response payload size grow? Heap usage shape changed?
• The key question: 'what changed?'

Step 7: Decide rollback or forward-fix

• Rollback if: clear deploy correlation, severe impact, forward-fix > 30 min away, rollback path known-good
• Forward-fix if: simple config change, feature flag off, scale up, rollback would regress other changes
• Default to rollback when unsure; a 5-minute rollback beats a 2-hour forward-fix with users hurting

Step 8: Root-cause after stability

• Capture artifacts: thread dumps, heap dumps, logs, traces from the incident window
• Reproduce locally or staging
• Profile with async-profiler or JFR to find hotspots
• Review code diff between pre- and post-deploy
• Write the postmortem: timeline, contributing factors, action items

Common root causes for post-deploy p99 spikes

• N+1 query introduced in a JPA entity
• New code path hitting a slow external API without timeout
• Cache TTL expired, all misses hit DB (cache miss cascade)
• GC thrashing from heap pressure
• Connection pool too small for new traffic pattern
• Thread pool starvation from blocking I/O in reactive code
• Serialization of a new large field in JSON response

What I'd emphasize in an interview

• I use the funnel: alert, confirm, deploy timeline, APM, trace, span, component metrics
• I don't jump to reading logs until traces and metrics have narrowed the search
• Deploy correlation is the first question; rollback early saves user pain
• The four golden signals (latency, traffic, errors, saturation) are my checklist
• Feature flags and SLO-based alerting reduce this class of incident over time