Debugging Missing Traces and Instrumentation Gaps
Fast recall
Quick revision
Core answer
/metrics), check sampling rate (might be sampled out), look for async code that loses MDC/trace context, and verify every downstream call forwards traceparent headers. Fix with: explicit manual instrumentation for gaps, always-on sampling for error traces, and context propagation helpers for async.Deep understanding of how distributed tracing actually works and fails
Assuming auto-instrumentation covers everything; not considering sampling; missing async context loss
Discuss W3C Trace Context, head vs tail sampling, MDC propagation in CompletableFuture/reactive code, and exception-triggered forced sampling
Interview-ready guide
Interview answer
Users report a specific bug with timestamp and error message, but the trace isn't in our APM. Before assuming the APM is broken, there are four common reasons for missing traces.
The four reasons, in order of likelihood
- Request didn't reach instrumented code — rejected at WAF, rate limit, or ingress
- APM agent didn't capture it — agent crashed, not installed, or auto-instrumentation doesn't cover this framework
- Trace was sampled out — head-based sampling at 1% drops 99% of traces
- Context propagation broke — downstream call didn't forward traceparent header, or async code lost context across threads
The debug flow, systematic and in order
- Start outermost: access logs at load balancer, ingress, app. Did the request arrive?
- Agent health: is the APM agent running, connected, not rate-limited?
- Auto-instrumentation coverage: is this framework actually covered? Test with a known endpoint
- Sampling: is the sample rate so low that specific requests are usually dropped?
- Context propagation: does the downstream call forward traceparent? Is async code preserving context?
The sampling fix
- Head-based sampling (decide at request start) drops traces before they can be flagged as interesting
- Tail-based sampling decides after the trace completes — keeps errors and slow traces preferentially
- Force-sample on error: always keep traces with error status, regardless of sample rate
- Configure in the APM agent or OTel Collector tail sampling processor
The async context propagation problem
- ThreadLocal-based context (Spring RequestContextHolder, SLF4J MDC) doesn't cross thread boundaries
- Every thread pool, @Async method, CompletableFuture needs explicit context propagation
- Spring fix: implement a TaskDecorator that copies MDC and OTel context across threads
- Reactor fix: Hooks.enableAutomaticContextPropagation() in recent versions
W3C Trace Context is the standard
- traceparent header format: 00-traceId-spanId-flags
- Baggage header for user context
- All major APMs (Datadog, New Relic, Grafana Tempo, Jaeger, Zipkin) support W3C
- Use it for interoperability; avoid vendor-specific propagation
Common places where traces go missing
- Custom HTTP clients without instrumentation; wrap them or use instrumented clients
- Kafka/SQS consumers; need manual header inject/extract at producer/consumer
- Thread pools: context bound to ThreadLocal; switching threads loses it
- Batch jobs: no HTTP means no auto-span; create spans explicitly
- GraphQL resolvers: single HTTP request with dozens of resolver calls; needs special handling
- Native image (GraalVM): reflection issues; verify OTel configures correctly
The safety net: structured logs with traceId
- Even if trace is sampled out, logs with traceId in MDC let you recover context
- Search logs for the traceId; see partial request flow
- Always include traceId in structured log lines
The preventive patterns
- Force-sample on error: configure in agent so errors always reach APM
- Tail-based sampling for production debugging
- TaskDecorator for async Spring
- Wrap CompletableFuture.thenApplyAsync() executors with context-aware wrapper
- Synthetic monitors with 100% sampling on critical paths
- OpenTelemetry instrumentation as the backend-neutral bet
What I'd emphasize in the interview
- Missing traces is usually not an APM bug; it's an instrumentation gap
- Systematic debug: access logs, agent health, coverage, sampling, propagation
- Async context propagation catches everyone; TaskDecorator pattern is the fix
- Force-sample-on-error is a setting everyone should have
- Invest in OTel instrumentation for portability and completeness
Debugging Missing Traces and Instrumentation Gaps
Direct Answer
Missing traces mean one of: (1) the request didn't reach instrumented code, (2) instrumentation didn't capture it (sampling, async context loss, uncaught exception path), (3) trace context wasn't propagated across a hop. Systematic debug: check access logs (request arrived?), verify APM agent is running (/metrics), check sampling rate (might be sampled out), look for async code that loses MDC/trace context, and verify every downstream call forwards traceparent headers. Fix with: explicit manual instrumentation for gaps, always-on sampling for error traces, and context propagation helpers for async.
Common Mistakes & Pitfalls
Assuming auto-instrumentation covers everything; not considering sampling; missing async context loss
Check your understanding
Answer these without looking back at the article.
- How does W3C Trace Context propagate across services and what headers are involved?
- What is a TaskDecorator in Spring and how does it solve async context propagation?
- What's the difference between head-based and tail-based sampling and when to use each?
- How do you instrument Kafka consumers/producers for distributed tracing?
Related interview prep categories
1Looking for a structured roadmap?
Every module on this page is also part of the full Java backend interview track.