Most retrieval-augmented generation demos collapse the moment real users show up. These are the boring things we check before a RAG system is allowed anywhere near production.
Evaluation first
No eval set, no merge. We build a golden set of 200–500 Q&A pairs reviewed by a subject-matter expert before writing a single embedding. Every change is graded against it.
Retrieval quality
- Hybrid search — dense + BM25 — with a fallback to keyword-only for exact product codes.
- Chunking informed by the document type (code, policy, transcript), not a fixed 512 tokens.
- Re-ranking with a small cross-encoder when latency budget allows.
Guardrails that matter
Input filtering, output filtering, and cost ceilings. Every call is observable: prompt, retrieved chunks, model response, cost, latency. We replay problem queries in CI.
The quiet part
80% of RAG quality comes from the data pipeline, not the model.
Get the ingestion, normalization and eval loop right first. The rest is tuning.
Popular Posts
