How to benchmark chunking strategies and embedding models on real RAG corpora
Chunking often matters as much as the embedding model itself — the 2025 NAACL Vectara study tested 25 chunking configurations across 48 embedding models and found chunking choice can shift retrieval quality by up to about 9 percentage points on the same corpus — but you must benchmark end-to-end because retrieval recall and answer accuracy can move in opposite directions.