Distributed Tracing for LLM Calls with Micrometer and Spring AI
Fast recall
Quick revision
Core answer
ObservationRegistry. When Spring AI's observation support is enabled, each ChatModel.call() automatically creates a span tagged with model name, token counts, and prompt metadata. These spans propagate through your service's existing trace context (Brave/Zipkin or OpenTelemetry). Add custom attributes via Observation.createNotStarted(...) for prompt version, feature name, and token budget. Visualise in Zipkin, Jaeger, or Grafana Tempo.Whether the candidate can extend existing Spring Boot tracing infrastructure to cover LLM calls without reinventing the wheel
Starting a new trace for each LLM call instead of participating in the existing request trace — loses the causal chain from HTTP request to LLM call to database
Explain how to propagate trace context to async LLM calls on virtual threads, and how to attach prompt hash and token count as span attributes for cost attribution per trace
Interview-ready guide
Interview answer
Spring AI integrates with Micrometer Tracing via ObservationRegistry. When Spring AI's observation support is enabled, each ChatModel.call() automatically creates a span tagged with model name, token counts, and prompt metadata. These spans propagate through your service's existing trace context (Brave/Zipkin or OpenTelemetry). Add custom attributes via Observation.createNotStarted(...) for prompt version, feature name, and token budget. Visualise in Zipkin, Jaeger, or Grafana Tempo.
Distributed Tracing for LLM Calls with Micrometer and Spring AI
Direct Answer
Spring AI integrates with Micrometer Tracing via ObservationRegistry. When Spring AI's observation support is enabled, each ChatModel.call() automatically creates a span tagged with model name, token counts, and prompt metadata. These spans propagate through your service's existing trace context (Brave/Zipkin or OpenTelemetry). Add custom attributes via Observation.createNotStarted(...) for prompt version, feature name, and token budget. Visualise in Zipkin, Jaeger, or Grafana Tempo.
Deep Dive & Architecture
LLM costs are driven by token usage (input + output tokens per call).
Track them via ChatResponse.getMetadata() which exposes Usage with prompt and completion token counts.
Expose these as Micrometer Counter metrics pushed to Prometheus/Grafana. Control costs through: token budgets per request via ChatOptions.withMaxTokens(...), prompt caching where supported, model tiering (fast cheap model for simple tasks, larger for complex), and circuit breakers to stop runaway calls. Set cost alerts at the API provider level as a last line of defence.
No per-request token tracking — discovering cost overruns at the monthly invoice rather than in real-time metrics
Common Mistakes & Pitfalls
Starting a new trace for each LLM call instead of participating in the existing request trace — loses the causal chain from HTTP request to LLM call to database
Check your understanding
Answer these without looking back at the article.
- How do you monitor and control LLM API costs in a Java production service?
- What metrics should you track for LLM-powered Spring Boot services?
Looking for a structured roadmap?
Every module on this page is also part of the full Java backend interview track.