Mocking LLM Calls in Unit Tests with Spring AI

Direct Answer

Spring AI provides InMemoryChatModel — a test-only ChatModel implementation that returns pre-scripted ChatResponse objects without making any API calls. Use it in @SpringBootTest slices or plain unit tests. For Mockito-based tests, use @MockitoBean ChatModel and when(chatModel.call(any())).thenReturn(fixedResponse). Both approaches isolate your service code from the real LLM, making tests fast, deterministic, and free.

Deep Dive & Architecture

Test LLM features at three levels: unit tests mock the ChatModel with a fixed response fixture (no real API calls); integration tests use Spring AI's InMemoryChatModel or @MockitoBean on ChatModel with scripted responses for deterministic behaviour; evaluation tests send real (or pre-recorded) prompts and score outputs using RelevancyEvaluator or custom scoring logic.

The key is separating the parts you can assert exactly (JSON structure, key presence) from the parts you can only score (semantic quality).

Never let tests hit a live LLM API in CI — cost, flakiness, and latency make the suite unreliable.

No mocking strategy — tests call the real API, making the suite slow, expensive, and flaky when the API is rate-limited

Common Mistakes & Pitfalls

Not knowing InMemoryChatModel exists — writing complex Mockito stubs from scratch or worse, hitting the real API in tests