Production Debugging with Distributed Traces — Finding Latency Bottlenecks
Fast recall
Quick revision
Core answer
Practical debugging workflow using traces, and the ability to identify common latency patterns from a trace waterfall.
Looking only at the first service in the trace rather than following the critical path through the full span tree.
Describe N+1 query detection from traces (many narrow database spans), sequential vs parallel fan-out patterns, and adding custom span attributes to narrow down root cause faster.
Interview-ready guide
Interview answer
Debugging production latency follows a structured flow that moves from metrics (which service?) to traces (which span?) to logs (what happened in that request?). Skipping straight to logs is slower and misses the systemic view.
Step 1 — Identify the slow service from metrics
- Start with the P99 latency dashboard: which service's P99 exceeds the SLO? A Grafana dashboard with
histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m]))per service shows which service is the bottleneck - If the latency spike aligns with an increase in request rate: the service may be under-provisioned; if latency is high even at normal throughput, the bottleneck is internal (database, downstream service)
Step 2 — Find a representative slow trace
- Open Jaeger or Zipkin and filter by the slow service and time window; sort by duration; pick the slowest traces
- The trace waterfall view shows every span and its duration — a long database span indicates a slow query; a long span in a downstream service indicates that service is the bottleneck; a gap between spans indicates network latency or serialization overhead
Step 3 — Drill into the problem span
- For a long database span: the span tags show the SQL query; run
EXPLAIN ANALYZEon that query in a database client; the execution plan reveals missing indexes, sequential scans, or incorrect join strategies - For a long downstream service span: open the child trace for that service (trace ID is the same); apply the same flow recursively to find the bottleneck in the downstream service
Step 4 — Correlate with logs
- Copy the trace ID from the slow trace; search in Elasticsearch or CloudWatch Logs Insights:
traceId = "abc123"returns all log entries from all services for that specific request - Log entries often contain context the trace doesn't: the exact payload that triggered a slow code path, a WARN that appeared before the ERROR, a retry that consumed the extra time
Fix and validate
- Apply the fix (add an index, increase timeout, optimize a query); monitor the P99 latency metric; deploy and compare before-vs-after to confirm improvement
Production Debugging with Distributed Traces — Finding Latency Bottlenecks
Direct Answer
Start from a Prometheus P99 alert to identify the slow service. Open Jaeger/Zipkin, filter traces for that service above the latency threshold. Examine the waterfall to find the widest span. Drill down to identify whether the bottleneck is a synchronous downstream call, an N+1 database query, or a missing async operation.
Common Mistakes & Pitfalls
Looking only at the first service in the trace rather than following the critical path through the full span tree.
Check your understanding
Answer these without looking back at the article.
- How do you identify an N+1 query problem from a distributed trace?
- What is the critical path in a trace and how do you identify it?
- How do you propagate OpenTelemetry trace context through CompletableFuture?
Related interview prep categories
1Looking for a structured roadmap?
Every module on this page is also part of the full Java backend interview track.