RAG without the kabuki: pragmatic AI for legacy systems

Every enterprise RAG demo follows the same script: a polished chat window, three rehearsed questions, applause. Then it meets the real corpus — forty years of scanned PDFs, SharePoint exports, and a wiki nobody has gardened since 2019 — and the answers turn confidently wrong. We ship retrieval systems to manufacturing clients, where a wrong answer can stop a production line. This is the boring pipeline that survives.

Retrieval should be boring on purpose

The demo stack reaches for the most exotic embedding model and a vector store with a waitlist. Ours is deliberately dull: hybrid search — BM25 plus embeddings, fused with reciprocal rank — over chunks that follow the document structure instead of a fixed token window. Keyword search catches part numbers and error codes that embeddings smear; embeddings catch the paraphrases keywords miss. Neither is clever. Together they are reliable.

• Chunk on headings and tables, never mid-procedure — a maintenance step split in half is worse than no answer.
• Store the source page and revision with every chunk; the citation is the feature, not the garnish.
• Rerank the top fifty with a small cross-encoder. It is the cheapest accuracy you will ever buy.

Eval gates on by default

The pipeline refuses to deploy if it gets worse. Before any change — new chunking, new model, new prompt — a golden set of three hundred real questions runs through the whole system, and CI compares answer faithfulness and retrieval recall against the current build.

eval:
  golden_set: data/golden-v3.jsonl   # 300 real operator questions
  thresholds:
    retrieval_recall@10: 0.87        # may not drop
    faithfulness: 0.92               # judged, sampled 20%
    refusal_correctness: 0.95        # knows when NOT to answer
  on_regression: block_deploy

If you cannot show me the eval diff, you are not deploying a model. You are deploying a mood.

Our ML review checklist, line one

Teach it to say “I don’t know”

The highest-value behavior in an industrial assistant is refusing gracefully. When retrieval confidence is low, the system says so and shows the nearest documents instead of synthesizing fiction. Operators trust it more for it — and the refusal rate became our favorite health metric: when it climbs, the corpus has drifted, not the model.