Testing LLM-Powered Features in Spring Boot — Strategies and Tools
Fast recall
Quick revision
Core answer
ChatModel with a fixed response fixture (no real API calls); integration tests use Spring AI's InMemoryChatModel or @MockitoBean on ChatModel with scripted responses for deterministic behaviour; evaluation tests send real (or pre-recorded) prompts and score outputs using RelevancyEvaluator or custom scoring logic. The key is separating the parts you can assert exactly (JSON structure, key presence) from the parts you can only score (semantic quality). Never let tests hit a live LLM API in CI — cost, flakiness, and latency make the suite unreliable.Whether the candidate has a principled strategy for testing non-deterministic LLM features rather than giving up or always hitting the real API
No mocking strategy — tests call the real API, making the suite slow, expensive, and flaky when the API is rate-limited
Explain the three testing layers explicitly, describe how to use Spring AI's evaluation framework for semantic scoring, and discuss contract testing for prompt templates
Interview-ready guide
Interview answer
Test LLM features at three levels: unit tests mock the ChatModel with a fixed response fixture (no real API calls); integration tests use Spring AI's InMemoryChatModel or @MockitoBean on ChatModel with scripted responses for deterministic behaviour; evaluation tests send real (or pre-recorded) prompts and score outputs using RelevancyEvaluator or custom scoring logic. The key is separating the parts you can assert exactly (JSON structure, key presence) from the parts you can only score (semantic quality). Never let tests hit a live LLM API in CI — cost, flakiness, and latency make the suite unreliable.
Testing LLM-Powered Features in Spring Boot — Strategies and Tools
Direct Answer
Test LLM features at three levels: unit tests mock the ChatModel with a fixed response fixture (no real API calls); integration tests use Spring AI's InMemoryChatModel or @MockitoBean on ChatModel with scripted responses for deterministic behaviour; evaluation tests send real (or pre-recorded) prompts and score outputs using RelevancyEvaluator or custom scoring logic. The key is separating the parts you can assert exactly (JSON structure, key presence) from the parts you can only score (semantic quality). Never let tests hit a live LLM API in CI — cost, flakiness, and latency make the suite unreliable.
Deep Dive & Architecture
Test LLM features at three levels: unit tests mock the ChatModel with a fixed response fixture (no real API calls); integration tests use Spring AI's InMemoryChatModel or @MockitoBean on ChatModel with scripted responses for deterministic behaviour; evaluation tests send real (or pre-recorded) prompts and score outputs using RelevancyEvaluator or custom scoring logic.
The key is separating the parts you can assert exactly (JSON structure, key presence) from the parts you can only score (semantic quality).
Never let tests hit a live LLM API in CI — cost, flakiness, and latency make the suite unreliable.
No mocking strategy — tests call the real API, making the suite slow, expensive, and flaky when the API is rate-limited
Common Mistakes & Pitfalls
No mocking strategy — tests call the real API, making the suite slow, expensive, and flaky when the API is rate-limited
Check your understanding
Answer these without looking back at the article.
- How do you mock LLM calls in unit tests using Spring AI's test support?
- What is AI evaluation testing and how do you measure LLM output quality in Java?
Looking for a structured roadmap?
Every module on this page is also part of the full Java backend interview track.