RAG Chunking Strategies for Document Ingestion — How Chunk Size and Overlap Affect Retrieval Quality
Fast recall
Quick revision
Core answer
TokenTextSplitter with 512-token chunks, 100-token overlap) is the safe default — fast and predictable. Sentence/paragraph chunking preserves semantic units; better for prose-heavy docs. Hierarchical chunking stores parent+child: embed small child chunks for precision retrieval, but return the larger parent context to the LLM. Semantic chunking groups sentences by embedding similarity shift — highest quality but slowest to ingest. Chunk size trades off recall vs precision: small chunks improve embedding accuracy but may miss multi-sentence context; large chunks capture more context but dilute the embedding signal.Whether the candidate understands that chunking is a critical design decision, not a boilerplate step — and specifically how chunk size and overlap interact with embedding quality and context window budget
Treating chunking as 'just split at N tokens' without thinking about semantic boundaries. Also: not knowing the chunk-size vs embedding-quality tradeoff, or thinking bigger chunks are always better because they contain more information
Describe hierarchical (parent-child) chunking — the insight that retrieval precision and LLM context size are separate concerns. Also mention the practical effect of chunk overlap on sentence-boundary queries
Interview-ready guide
Interview answer
Chunking is how you split documents before embedding. It's the most underrated part of the RAG pipeline — bad chunking breaks retrieval before the LLM ever sees the question.
Fixed-size token chunking (Spring AI default)
TokenTextSplitterwith 512-token chunks and 100-token overlap- Fast, predictable, good enough for most structured documents
- Weakness: splits mid-sentence; overlap partially compensates
Sentence and paragraph chunking
- Split at natural language boundaries — semantically coherent
- Better for prose-heavy docs where paragraphs are self-contained ideas
- Variable chunk size makes token-budget planning harder
Hierarchical (parent-child) chunking
- Small child chunks (128 tokens) for precise embedding and retrieval
- Larger parent chunks (512 tokens) returned to the LLM for context richness
- Decouples retrieval precision from answer context — best balance for technical docs
Semantic chunking
- Detect topic shifts by measuring cosine similarity drops between consecutive sentence embeddings
- Highest quality chunks — each one represents a single coherent concept
- Expensive at ingest time — embedding every sentence before splitting
The core tradeoff
- Small chunks: precise embeddings, low context richness
- Large chunks: diluted embeddings, high context richness
- Overlap prevents sentence-boundary misses — use 10–20% of chunk size as overlap
Practical advice
- Start at 512 tokens + 100-token overlap. Run 20 manual test queries, check top-5 retrieval results, and adjust chunk size based on what you find. Reach for semantic or hierarchical strategies only when the simpler approach has a documented quality gap.
RAG Chunking Strategies for Document Ingestion — How Chunk Size and Overlap Affect Retrieval Quality
Direct Answer
Chunking splits documents into segments that get independently embedded and stored. The strategy directly controls what the vector search retrieves. Fixed-size token chunking (Spring AI's TokenTextSplitter with 512-token chunks, 100-token overlap) is the safe default — fast and predictable. Sentence/paragraph chunking preserves semantic units; better for prose-heavy docs. Hierarchical chunking stores parent+child: embed small child chunks for precision retrieval, but return the larger parent context to the LLM. Semantic chunking groups sentences by embedding similarity shift — highest quality but slowest to ingest. Chunk size trades off recall vs precision: small chunks improve embedding accuracy but may miss multi-sentence context; large chunks capture more context but dilute the embedding signal.
Common Mistakes & Pitfalls
Treating chunking as 'just split at N tokens' without thinking about semantic boundaries. Also: not knowing the chunk-size vs embedding-quality tradeoff, or thinking bigger chunks are always better because they contain more information
Check your understanding
Answer these without looking back at the article.
- How do you measure retrieval quality to know if your chunking strategy is working?
- What is the parent-child retrieval pattern and how do you implement it in Spring AI?
- How does chunk overlap affect the size of your vector index and query latency?
- When would semantic chunking be worth the extra ingestion cost?
- How do you handle tables and structured data in RAG chunking — they break sentence splitters?
Looking for a structured roadmap?
Every module on this page is also part of the full Java backend interview track.