Building a RAG System That Actually Works (Not Just a Demo)

Most RAG Implementations Are Just Expensive Search Engines

Every AI startup pitch deck in 2026 mentions RAG — Retrieval-Augmented Generation. The idea is simple: instead of hoping the LLM memorized your data during training, you fetch relevant documents at query time and stuff them into the context window. The LLM reads your actual data and answers based on it.

Simple concept. Terrible execution, almost everywhere I look.

The typical RAG implementation: dump all your documents into a vector database, embed each chunk with text-embedding-ada-002, and do a cosine similarity search at query time. Top 5 chunks go into the prompt. Ship it.

This approach fails for the same reason keyword search fails — relevance is not just about textual similarity. When a customer asks "what’s your return policy for damaged items?", the vector search might return chunks about shipping damage, product damage reports, and return policies — three different documents that each partially answer the question but none of which gives the complete answer.

The Architecture That Actually Works

I have built RAG systems for two production use cases: a customer support agent that answers questions from a 400-page product catalog, and a sales agent that references live inventory and pricing data. Both share the same architecture, which I call structured retrieval.

The key insight: do not treat all your documents as equal-weight text chunks. Classify them by type and retrieval strategy:

• Lookup data (pricing, inventory, specs) — query a structured API or database, not a vector store. Prices change daily; embeddings do not update themselves.
• Policy documents (returns, warranty, shipping) — store as complete documents with metadata tags. Retrieve by tag match, not similarity. When someone asks about returns, pull the entire return policy, not a 512-token chunk of it.
• Knowledge base articles (how-tos, troubleshooting) — these are the only ones where vector similarity actually helps. Chunk at the section level (H2 headers), not arbitrary token counts.

Chunking Strategy Matters More Than Your Embedding Model

I have seen teams spend weeks evaluating embedding models (ada-002 vs. Cohere vs. Voyage vs. BGE) while chunking their documents into fixed 512-token blocks with 50-token overlap. The chunking strategy has 5x more impact on retrieval quality than the embedding model.

Rules I follow after building multiple production systems:

1. Chunk at semantic boundaries. Headers, section breaks, paragraph groups. Never mid-sentence. A document about "MacBook Air M2 Specifications" should be one chunk, not split across three.
2. Include the document title and section header in every chunk. The chunk "16GB unified memory, 256GB SSD" is useless without context. The chunk "MacBook Air M2 Specs > Memory and Storage: 16GB unified memory, 256GB SSD" is retrievable.
3. Separate facts from procedures. "The warranty covers hardware defects for 1 year" is a fact. "To file a warranty claim, email support@..." is a procedure. They answer different question types and should be tagged differently.
4. Never chunk below 200 tokens. Tiny chunks lose context and produce hallucinated answers. If a section is under 200 tokens, merge it with the adjacent section.

The Reranking Layer

Vector similarity gives you candidates. Reranking gives you answers. Every production RAG system I build has a two-stage retrieval pipeline:

1. Broad retrieval: Pull top 20 chunks from the vector store. This is fast and cheap.
2. LLM reranking: Send those 20 chunks to a fast model (Claude Haiku or GPT-4o-mini) with the user’s question and ask: "Which of these chunks directly answers this question? Rank them." Take the top 3-5.

The reranking step typically improves answer accuracy from ~65% (vector-only) to ~90% (vector + rerank). The cost is one additional LLM call with ~4000 tokens of input — roughly $0.001 per query. Negligible.

Hallucination Prevention

The number one failure mode of RAG is not bad retrieval — it is the LLM ignoring the retrieved context and generating an answer from its training data. I have seen a support agent confidently quote a return policy that the company does not have, because the LLM’s training data included a similar company’s policy.

Three techniques that actually prevent this:

• Explicit grounding instructions: "Answer ONLY based on the provided documents. If the answer is not in the documents, say ‘I don’t have that information’ and offer to connect the customer with a human."
• Citation requirements: "Include the document name and section in your answer." This forces the LLM to reference specific sources, making hallucinations obvious and auditable.
• Confidence scoring: After generating the answer, ask the LLM: "On a scale of 1-5, how confident are you that this answer is supported by the provided documents?" Anything below 4 gets escalated to a human.

Live Data Integration

The second production RAG system I built handles live inventory queries. A customer asks "do you have any MacBook Airs under $500?" and the system needs to check current stock and pricing — not yesterday’s embedding.

The solution: tool-augmented RAG. The LLM has access to a check_inventory function that queries the live Shopify API. When the question involves pricing, availability, or stock status, the LLM calls the function instead of searching the vector store.

The decision of "search docs vs. call API" is made by the LLM itself, based on the question type. This is where Cloudflare Workers shine — the function calls execute at the edge with sub-10ms latency to the Shopify API, so the customer gets a real-time answer without noticeable delay.

Cost Breakdown

A RAG system for a small business with 200 knowledge base articles, 50 products, and handling 100 queries per day:

• Vector database: $0/month (Cloudflare Vectorize free tier handles 5M vectors)
• Embedding generation: $2-5 one-time cost for initial indexing, pennies for updates
• Query costs: ~$0.003 per query (retrieval + rerank + generation) = ~$9/month
• Infrastructure: $0/month (Cloudflare Workers free tier)

Total: under $15/month for a production RAG system that correctly answers customer questions from your actual data, with citations, 24/7.

Compare that to hiring someone to answer customer questions for 8 hours a day, or paying $99/month for a chatbot SaaS that still needs constant training.

What I Learned

Start with 20 real customer questions. Before building anything, collect the 20 most common questions your business receives. Build the RAG system to answer those 20 correctly, with citations. Then expand. Starting with "index everything" produces a system that answers nothing well.

Monitor retrieval quality, not just answer quality. When the system gives a wrong answer, 80% of the time the problem is retrieval (wrong chunks pulled), not generation (LLM hallucinating). Log the retrieved chunks alongside every answer so you can debug retrieval failures.

Update your index when your data changes. This sounds obvious but most RAG tutorials skip it. I use Cloudflare Workers Cron to re-index changed documents nightly. A stale index is worse than no index — it gives confidently wrong answers.

Related Articles