RAG & Embeddings Rules

Retrieval quality caps generation quality — no amount of prompt engineering fixes an LLM answering from the wrong chunk. Treat the retrieval pipeline (chunking, indexing, filtering, ranking) as the primary system to get right, not a preprocessing step before the "real" work of prompting.

Chunking strategy

• Split on semantic boundaries (headings, paragraphs, list items) rather than fixed character counts — a chunk that cuts a sentence or a table row in half is worse than a slightly uneven chunk size.
• Use overlap between adjacent chunks (roughly 10-20% of chunk length) so a fact sitting near a chunk boundary isn't stranded without its surrounding context in either chunk.
• Size chunks for the retrieval unit, not the model's context window — a chunk should be small enough to be specifically relevant (so it doesn't dilute the top-k results with unrelated content) and large enough to be self-contained (so it makes sense read in isolation).
• Preserve structural metadata per chunk (source document, section heading, page number) at chunking time — this is far cheaper to capture once than to reconstruct later, and it's what makes citation possible.

Metadata filtering before vector search

• Apply hard filters (tenant/user ID, document type, date range, permission scope) before or alongside the vector search, never as a post-filter on top-k results — filtering after retrieval means a query can return zero usable results even though relevant documents exist, because the irrelevant-but-close vectors crowded out the top-k budget.
• Never rely on the embedding model to encode access control — a similar-looking chunk from a document the user isn't authorized to see must be excluded structurally, not hoped away by semantic distance.

Hybrid search

• Combine keyword/full-text search (BM25 or equivalent) with vector similarity rather than using vector search alone — exact terms (product codes, names, acronyms, error messages) are exactly what embeddings are weakest at, and keyword search catches them reliably.
• Merge the two result sets with a defined fusion method (reciprocal rank fusion or a weighted score combination) rather than picking one arbitrarily when they disagree — an undefined tie-break makes retrieval quality unpredictable across queries.

Retrieval evaluation

• Build a test set of real queries with known-correct source chunks before shipping, and measure retrieval metrics (recall@k, precision@k) directly — don't evaluate the system only by eyeballing final generated answers, which obscures whether a bad answer came from bad retrieval or bad generation.
• Re-run the evaluation set whenever the chunking strategy, embedding model, or index changes — these changes silently shift what gets retrieved, and regressions are invisible without a fixed benchmark to compare against.

Citation of retrieved sources

• Return the source (document, section, chunk ID) alongside every retrieved chunk fed to the model, and instruct the model to cite which source backs each claim in its answer — an uncited RAG answer is unverifiable and impossible to debug when it's wrong.
• Never let the model answer from chunks it wasn't actually given — if retrieval returns nothing relevant, the correct behavior is saying so, not letting the model fall back on parametric knowledge while still presenting the answer as sourced.

Staleness and re-indexing discipline

• Re-index or incrementally update embeddings whenever source content changes — a RAG system serving answers from a stale index confidently returns outdated facts with no signal to the user that anything is wrong.
• Track a last-indexed timestamp per document and surface it (in logs at minimum, in the UI where relevant) so staleness is diagnosable rather than silently accumulating.
• Prefer incremental re-indexing of changed documents over full corpus rebuilds where the pipeline supports it — full rebuilds on every change don't scale and create windows where the index is unavailable or inconsistent.

Context budget discipline

• Retrieve a bounded top-k of the most relevant chunks and pass only those to the model — never stuff the entire corpus or an entire document into context "to be safe." More irrelevant context measurably degrades an LLM's ability to use the relevant parts, on top of the direct cost and latency hit.
• Re-rank the initial retrieval set with a cheaper relevance step before the final top-k cut where recall from the first pass is generously wide — a rerank step recovers precision that pure vector similarity alone often misses.