Handling LLM API Rate Limits and Retry Logic in Spring Boot
Fast recall
Quick revision
Core answer
@Retryable with exponential backoff for transient 429 errors. Layer Resilience4j circuit breaker on top to stop hammering the API when rate limits are sustained. For token-based limits, use a token bucket (e.g., Guava RateLimiter) before sending requests. Always log retry attempts and propagate a meaningful error to callers when the circuit opens.Whether the candidate distinguishes per-minute rate limits from token-per-minute limits, knows the tools (Spring Retry, Resilience4j), and can articulate when to stop retrying
Retrying immediately on a 429 — it makes the problem worse. Not distinguishing RPM (requests per minute) from TPM (tokens per minute). Opening the circuit breaker too aggressively on a cold start.
Mention request-level token counting before sending (estimate prompt tokens), tracking remaining quota from response headers, and degrading gracefully to a cached response or a simpler prompt
Interview-ready guide
Interview answer
Use Spring Retry's @Retryable with exponential backoff for transient 429 errors. Layer Resilience4j circuit breaker on top to stop hammering the API when rate limits are sustained. For token-based limits, use a token bucket (e.g., Guava RateLimiter) before sending requests. Always log retry attempts and propagate a meaningful error to callers when the circuit opens.
Handling LLM API Rate Limits and Retry Logic in Spring Boot
Direct Answer
Use Spring Retry's @Retryable with exponential backoff for transient 429 errors. Layer Resilience4j circuit breaker on top to stop hammering the API when rate limits are sustained. For token-based limits, use a token bucket (e.g., Guava RateLimiter) before sending requests. Always log retry attempts and propagate a meaningful error to callers when the circuit opens.
Deep Dive & Architecture
All three are supported by Spring AI starters: spring-ai-openai, spring-ai-anthropic, spring-ai-vertex-ai-gemini.
The ChatClient interface is identical across providers — swap the starter and properties to switch.
Key differences: OpenAI uses OpenAiChatModel, Anthropic uses AnthropicChatModel, Google uses VertexAiGeminiChatModel. Auth differs — OpenAI and Anthropic use API keys, Gemini on Vertex AI uses service account credentials. Token context windows and pricing vary significantly.
Manually wiring HTTP clients when Spring AI's auto-configuration handles it; not using the ChatClient builder pattern and going straight to the lower-level model
Common Mistakes & Pitfalls
Retrying immediately on a 429 — it makes the problem worse. Not distinguishing RPM (requests per minute) from TPM (tokens per minute). Opening the circuit breaker too aggressively on a cold start.
Check your understanding
Answer these without looking back at the article.
- How do you integrate the OpenAI API into a Spring Boot application using Spring AI?
- What is function calling in LLM APIs and how do you implement it with Spring AI?
- Compare OpenAI, Anthropic Claude, and Google Gemini APIs from a Java backend integration perspective
Looking for a structured roadmap?
Every module on this page is also part of the full Java backend interview track.