Key Metrics for LLM-Powered Spring Boot Services
Fast recall
Quick revision
Core answer
Timer on the full LLM call), token usage (Counter for prompt and completion tokens), error rate (failed calls by error type), cost (token counts × per-1K-token price as a derived metric in Grafana), and quality proxies (fallback rate, retry rate, empty-response rate). Use Micrometer's ObservationRegistry and Spring AI's built-in ChatModel observation support to capture latency and metadata automatically. Expose via Spring Boot Actuator's /actuator/prometheus endpoint.Whether the candidate knows that LLM services need a different metrics vocabulary than standard REST services — token cost and quality proxies on top of the usual RED metrics
Only tracking latency — missing token cost, error rates, and quality signals like empty or malformed responses
Describe how to wire ObservationRegistry into Spring AI for automatic span creation, and how to add custom tags (model name, feature, prompt version) for sliced dashboards
Interview-ready guide
Interview answer
Track five families: latency (Timer on the full LLM call), token usage (Counter for prompt and completion tokens), error rate (failed calls by error type), cost (token counts × per-1K-token price as a derived metric in Grafana), and quality proxies (fallback rate, retry rate, empty-response rate). Use Micrometer's ObservationRegistry and Spring AI's built-in ChatModel observation support to capture latency and metadata automatically. Expose via Spring Boot Actuator's /actuator/prometheus endpoint.
Key Metrics for LLM-Powered Spring Boot Services
Direct Answer
Track five families: latency (Timer on the full LLM call), token usage (Counter for prompt and completion tokens), error rate (failed calls by error type), cost (token counts × per-1K-token price as a derived metric in Grafana), and quality proxies (fallback rate, retry rate, empty-response rate). Use Micrometer's ObservationRegistry and Spring AI's built-in ChatModel observation support to capture latency and metadata automatically. Expose via Spring Boot Actuator's /actuator/prometheus endpoint.
Deep Dive & Architecture
LLM costs are driven by token usage (input + output tokens per call).
Track them via ChatResponse.getMetadata() which exposes Usage with prompt and completion token counts.
Expose these as Micrometer Counter metrics pushed to Prometheus/Grafana. Control costs through: token budgets per request via ChatOptions.withMaxTokens(...), prompt caching where supported, model tiering (fast cheap model for simple tasks, larger for complex), and circuit breakers to stop runaway calls. Set cost alerts at the API provider level as a last line of defence.
No per-request token tracking — discovering cost overruns at the monthly invoice rather than in real-time metrics
Common Mistakes & Pitfalls
Only tracking latency — missing token cost, error rates, and quality signals like empty or malformed responses
Check your understanding
Answer these without looking back at the article.
- How do you monitor and control LLM API costs in a Java production service?
- How do you add distributed tracing to LLM calls using Micrometer and Spring AI?
Looking for a structured roadmap?
Every module on this page is also part of the full Java backend interview track.