AI Evaluation Testing — Measuring LLM Output Quality in Java
Fast recall
Quick revision
Core answer
EvaluationRequest and RelevancyEvaluator which use a second LLM call as a judge: given the original question, the retrieved context, and the LLM's answer, the evaluator scores whether the answer is relevant. Beyond that, measure: format compliance (JSON schema validation), factual accuracy (string/regex checks for known facts), and regression comparison against a golden dataset.Whether the candidate understands that LLM quality can't be measured with assertEquals and knows the eval-as-a-judge pattern
Skipping eval testing entirely because 'you can't test AI' — a common myth that leads to undetected prompt regressions in production
Describe the LLM-as-judge pattern with its own bias risks, discuss building a golden dataset for regression testing, and mention frameworks like DeepEval or RAGAS for RAG pipeline evaluation
Interview-ready guide
Interview answer
AI evaluation testing (eval testing) measures the quality of LLM outputs beyond pass/fail assertions — because LLMs are non-deterministic and outputs are semantic, not exact. Spring AI provides EvaluationRequest and RelevancyEvaluator which use a second LLM call as a judge: given the original question, the retrieved context, and the LLM's answer, the evaluator scores whether the answer is relevant. Beyond that, measure: format compliance (JSON schema validation), factual accuracy (string/regex checks for known facts), and regression comparison against a golden dataset.
AI Evaluation Testing — Measuring LLM Output Quality in Java
Direct Answer
AI evaluation testing (eval testing) measures the quality of LLM outputs beyond pass/fail assertions — because LLMs are non-deterministic and outputs are semantic, not exact. Spring AI provides EvaluationRequest and RelevancyEvaluator which use a second LLM call as a judge: given the original question, the retrieved context, and the LLM's answer, the evaluator scores whether the answer is relevant. Beyond that, measure: format compliance (JSON schema validation), factual accuracy (string/regex checks for known facts), and regression comparison against a golden dataset.
Deep Dive & Architecture
Test LLM features at three levels: unit tests mock the ChatModel with a fixed response fixture (no real API calls); integration tests use Spring AI's InMemoryChatModel or @MockitoBean on ChatModel with scripted responses for deterministic behaviour; evaluation tests send real (or pre-recorded) prompts and score outputs using RelevancyEvaluator or custom scoring logic.
The key is separating the parts you can assert exactly (JSON structure, key presence) from the parts you can only score (semantic quality).
Never let tests hit a live LLM API in CI — cost, flakiness, and latency make the suite unreliable.
No mocking strategy — tests call the real API, making the suite slow, expensive, and flaky when the API is rate-limited
Common Mistakes & Pitfalls
Skipping eval testing entirely because 'you can't test AI' — a common myth that leads to undetected prompt regressions in production
Check your understanding
Answer these without looking back at the article.
- How do you test LLM-powered features in a Spring Boot application?
- How do you mock LLM calls in unit tests using Spring AI's test support?
Looking for a structured roadmap?
Every module on this page is also part of the full Java backend interview track.