Debugging a Complex Production Issue

When production degrades I lean on observability instead of intuition — my rule is never form a hypothesis until you have a trace in hand, because guessing under pressure is how a twenty-minute fix turns into a two-hour outage. We had a Spring Boot checkout path where a significant slice of requests started timing out during peak hours. I filtered Datadog traces by latency percentile and every slow path bottlenecked at the same Postgres join that had silently lost its covering index during a recent schema migration.

I confirmed the regression with EXPLAIN ANALYZE, rebuilt the index concurrently on the replica first to validate safety, and temporarily shed non-critical writes to a Redis queue while the primary index built. Within minutes of the index becoming active, p99 latency dropped from multiple seconds to under two hundred milliseconds and error rates fell to baseline noise. The entire cycle from PagerDuty alert to full resolution was about twenty-two minutes, well inside our thirty-minute SLA.

Afterward I wrote a blameless postmortem, added migration validation as a required CI step, and set up a Datadog alert that fires whenever any query p99 crosses two hundred milliseconds. Those guardrails turned a one-off heroic fix into a durable safety net. The tip I always close with: instrument your query plans in staging under production-shaped load, because the next missing index will not announce itself — it will hide behind a routine migration and wait for traffic to spike before it bites.