RAG Explained: Retrieval-Augmented Generation From Scratch

Retrieval-augmented generation is usually introduced with an architecture diagram containing six boxes and a logo. It is a dictionary lookup and a string concatenation. Here it is in full, before any of the boxes.

The problem retrieval solves

A model only knows what is in its weights and what is in the context you sent. Weights are frozen at training time and contain nothing about your company’s refund policy. So the entire question is: how does the relevant paragraph get into the context window before the model needs it?

If your corpus is small, the answer is “paste all of it”. RAG only becomes necessary when the corpus is larger than the window, or larger than you are willing to pay for on every request. At that point you need to select. And selection is a search problem, which is why every hard part of RAG is a search problem wearing a hat.

The whole thing, in code

No vector database, no orchestration framework, no chain. Two HTTP calls to a provider and one matrix multiply.

import os, requests, numpy as np

API = "https://api.example-provider.com/v1"
H   = {"Authorization": "Bearer " + os.environ["API_KEY"]}

# 1. YOUR CORPUS, already split into passages of a few hundred words.
DOCS = [
    "Refunds are issued to the original payment method within 14 days...",
    "Enterprise plans are invoiced annually and are not eligible for...",
    # ... a few thousand more
]

def embed(texts, model="text-embedding-3-small"):
    r = requests.post(API + "/embeddings", headers=H,
                      json={"model": model, "input": texts})
    r.raise_for_status()
    return np.array([d["embedding"] for d in r.json()["data"]], dtype="float32")

# 2. THE INDEX. This array is the whole "vector store".
INDEX = embed(DOCS)
INDEX /= np.linalg.norm(INDEX, axis=1, keepdims=True)   # unit vectors

# 3. RETRIEVAL. Cosine similarity is a dot product once both sides are unit.
def retrieve(question, k=4):
    q = embed([question])[0]
    q /= np.linalg.norm(q)
    scores = INDEX @ q                       # one float per chunk
    top = np.argsort(-scores)[:k]
    return [(DOCS[i], float(scores[i])) for i in top]

# 4. GENERATION. The retrieved text is just string concatenation.
SYSTEM = ("Answer using only the numbered sources below. If they do not "
          "contain the answer, say that you do not know. Cite as [n].")

def answer(question):
    hits = retrieve(question)
    sources = "\n\n".join(
        "[%d] %s" % (i + 1, text) for i, (text, _) in enumerate(hits))
    r = requests.post(API + "/chat/completions", headers=H, json={
        "model": "a-small-instruct-model",
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user",
             "content": sources + "\n\nQuestion: " + question},
        ],
    })
    return r.json()["choices"][0]["message"]["content"], hits

That runs. It answers questions about your documents. Everything a vector database sells you is an optimisation of line 3 or an operational convenience around line 2 — neither of which you need on day one, and both of which are easier to evaluate once you have a baseline that works.

Three lines are quietly doing more work than they appear to. The normalisation after embed(DOCS) is not cosmetic: without it, INDEX @ q is a raw dot product, which rewards vectors with a large magnitude, which in practice means long chunks. Skip it and your retrieval develops a systematic bias toward whichever passages happened to be verbose — a bug that never raises an error and looks like a mysterious ranking problem.

The single embed(DOCS) call is also a simplification. Every provider caps the number of inputs and the total tokens in one embedding request, so a real ingest batches — a few hundred chunks at a time — with retry on rate limits and a checkpoint so a failure halfway through 40,000 chunks does not mean starting again. And DOCS arriving pre-split is the sleight of hand mentioned below: producing that list from actual PDFs, wiki exports and HTML is most of the real work.

The four parts, and which one you will get wrong

The one you will get wrong is chunking. Embedding models are commodities and retrieval is arithmetic, but the decision about where a document is cut determines what can ever be retrieved. A chunk that begins “This does not apply to annual plans” with the antecedent three hundred words earlier is unretrievable no matter how good the embedding model is, because nothing in it is about the thing it is about.

When brute force stops being enough

The INDEX @ q above is a dense matrix-vector product. Its cost is exactly n_chunks × dimensions multiply-adds. For 5,000 chunks at 1,536 dimensions that is 7.7 million operations — well under a millisecond in NumPy, and the memory is 5,000 × 1,536 × 4 bytes ≈ 30 MB.

Scale that up. At one million chunks the index is 6.1 GB and each query is 1.5 billion multiply-adds; you are now at tens of milliseconds per query on one core, and you cannot hold it in a web process. Somewhere between those two numbers — usually a few hundred thousand chunks — an approximate index earns its complexity. Below it, a NumPy array in memory and a rebuild on deploy is a legitimate production architecture, and it has the enormous advantage that recall is 100% by construction, so any retrieval failure is a chunking or embedding failure and you know where to look.

What to add first

In rough order of value per hour spent, and none of them before you have a set of about fifty real questions with the chunk that should answer each one:

• Keyword search alongside the vectors. Embeddings are bad at exact identifiers — part numbers, error codes, function names. BM25 is excellent at them and costs almost nothing to run in parallel.
• A reranker. Retrieve 25, rerank to 5. This is the largest single quality improvement available for the least structural change.
• Metadata filters. Most bad retrievals are correct answers to the wrong version, the wrong tenant or the wrong year, and a filter fixes those without touching the model.
• Better chunks. Last on this list only because it is hardest to do blind. Once you have the fifty questions, it moves to first.

Notice what is not on that list: a framework. Every item above is a change to twenty lines of the code shown here, and each one is independently testable against the same fifty questions. Frameworks become useful when you have several pipelines to keep consistent, not when you have one to get working — and starting with one makes it substantially harder to answer the question that matters most in the first month, which is “which of the four stages is producing this bad answer?”

The other thing worth internalising early is that this system has no memory of being wrong. If the right chunk is not in the top four, the model receives no signal that anything is missing; it answers from what it was given, fluently. Every quality control you will eventually add — the refusal instruction, the citations, the evaluation set — exists to convert that silent failure into a visible one.