Everyone has built a RAG demo; few have shipped a RAG system. The demo loads a PDF into LangChain, vibes for five minutes in a notebook, and gets applause. The system faces 40,000 messy documents, users who ask questions in three languages, and a boss who notices when answers are wrong. In my post on production agent architecture I covered the agent loop — here I want to zoom into the part that actually determines quality: retrieval.
Because here's the uncomfortable truth: once you're above a certain model threshold, retrieval quality is the product. The LLM can only reason over what you hand it.
Ingestion decisions haunt you forever
Ingestion is boring, which is exactly why teams rush it and pay later. The decisions that matter:
- Chunk size: 300–500 tokens with ~15% overlap is my default starting point. Smaller chunks retrieve precisely but lose context; larger ones keep context but dilute the embedding until everything matches everything.
- Split on structure, not character count. A chunk that starts mid-sentence and ends mid-table is garbage no matter its size. Respect headings, paragraphs, and code blocks — recursive splitting on separators beats fixed-window chopping every time.
- Attach metadata religiously: source document, section title, date, version. Filters on metadata fix half of all "wrong answer" complaints ("ignore anything older than 2025").
- Deduplicate near-identical docs. Five versions of the same policy PDF means five competing chunks fighting for top-k.
Chunking is craft; embeddings are commodity
Teams agonize over embedding model benchmarks while shipping chunks that start mid-sentence. Wrong priority. Modern embedding models are all good and their differences shrink yearly; no model rescues a chunk whose boundaries destroy meaning. If you change one thing about ingestion, make it "split where ideas naturally end."
Hybrid search, always
Pure vector search has a blind spot exactly where business users live: exact tokens. SKUs, error codes, function names, invoice numbers — embeddings blur these into semantic mush. ERR_2041 should match ERR_2041, not a paragraph about "general connection issues."
The fix is hybrid search: run BM25 (keyword) and vector search in parallel, merge results with Reciprocal Rank Fusion. It's a one-line scoring formula over two result lists, and it consistently outperforms either approach alone.
Rerank: the cheapest quality jump you can buy
Here's my standard move: retrieve k=50 cheaply with hybrid search, then rerank down to the top 5 with a cross-encoder. The bi-encoder embeddings score documents independently; a cross-encoder reads query and document together, so it catches relevance signals vector search structurally cannot see.
Cost: an extra 100–200ms per query. Quality jump: routinely the largest single improvement in the whole stack. If your pipeline does only two clever things, make them hybrid search and reranking.
Evaluate or fly blind
The failure mode of RAG projects is tuning by vibes — change the chunker, ask three questions, feel good, deploy. Then quality quietly regresses and nobody knows why.
What works instead:
- Build a golden set of 30–50 real questions paired with the source passages that must be retrieved. Mine them from actual support tickets and user queries, not from your imagination.
- Measure recall@k — did the right chunk appear in the top-k at all? This is the metric upstream of answer quality.
- Re-run the set after every change to chunker, embedding model, or k. It takes minutes and catches regressions before users do.
You don't need an eval platform to start. A script, a JSON file of questions, and honesty are enough.
Ground everything
Last layer: force citations. Every claim links back to the chunk it came from; if retrieval confidence is low, say "I couldn't find this in the documentation" instead of improvising. Users forgive a missing answer far more easily than a confident wrong one — and cited answers let them verify in seconds, which is what turns skeptical users into believers.
Users don't care which embedding model you picked. They care whether the answer cites the right page. Spend your effort accordingly.
