What is RAG? Retrieval-augmented generation, with code

RAG, short for retrieval-augmented generation, means looking up the information a question needs before a language model answers it. You search your own documents for the passages most relevant to the question, put those passages into the prompt, and ask the model to answer from them. The model then answers from your data instead of from memory, and can cite where the answer came from.

It is how support bots answer from a help center, how internal assistants answer from a company wiki, and how “chat with your PDF” works.

What it looks like

A customer asks: “Can I get my money back on a yearly subscription?”

1. Retrieve. The system searches the help center and finds: “Annual plans can be refunded within 30 days of purchase.”
2. Augment. It builds a prompt: the question, plus that passage, plus the instruction “answer only from these sources”.
3. Generate. The model answers: “Yes, annual plans can be refunded within 30 days of purchase [1].”

Without step 1, the model would guess, and a confident guess about your refund policy is worse than no answer.

How does RAG work?

There are two phases.

Once, ahead of time (indexing):

1. Split your documents into passages, a paragraph or a few each. This is called chunking.
2. Turn each passage into an embedding, a list of numbers that captures its meaning.
3. Store the passages and their embeddings, usually in a vector database.

For every question (retrieval and generation):

1. Turn the question into an embedding with the same model.
2. Find the passages whose embeddings are closest to it.
3. Put the best few into the prompt, and let the model answer.

Search by meaning is the classic way to retrieve, but not the only one. Keyword search works too, and many systems combine both.

RAG in about 50 lines

This runs entirely on your machine with Ollama. Pull an embedding model and a chat model first:

ollama pull embeddinggemma
ollama pull qwen3

Save this as rag.py. It uses only Python’s standard library:

import json
import urllib.request

OLLAMA = "http://localhost:11434"
EMBED_MODEL = "embeddinggemma"
CHAT_MODEL = "qwen3"

chunks = [
    "Annual plans can be refunded within 30 days of purchase.",
    "Monthly plans are not refundable, but you can cancel at any time.",
    "To reset your password, use the 'Forgot password' link on the sign-in page.",
    "Two-factor authentication can be turned on under Settings > Security.",
]


def post(path, payload):
    body = json.dumps(payload).encode()
    req = urllib.request.Request(OLLAMA + path, data=body, headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req) as resp:
        return json.load(resp)


def embed(texts):
    return post("/api/embed", {"model": EMBED_MODEL, "input": texts})["embeddings"]


def similarity(a, b):
    return sum(x * y for x, y in zip(a, b))


def retrieve(question, vectors, k=2):
    q = embed([question])[0]
    ranked = sorted(zip(chunks, vectors), key=lambda c: similarity(q, c[1]), reverse=True)
    return [text for text, _ in ranked[:k]]


def answer(question, context):
    sources = "\n".join(f"[{i + 1}] {text}" for i, text in enumerate(context))
    prompt = (
        "Answer the question using only the sources below. "
        "Cite them like [1]. If they don't contain the answer, say you don't know.\n\n"
        f"Sources:\n{sources}\n\nQuestion: {question}"
    )
    reply = post("/api/chat", {
        "model": CHAT_MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "stream": False,
    })
    return reply["message"]["content"]


if __name__ == "__main__":
    vectors = embed(chunks)
    question = "Can I get my money back on a yearly subscription?"
    context = retrieve(question, vectors)
    print(answer(question, context))

Run python rag.py. Then change the question to something the chunks don’t cover, like “Do you ship to Canada?”, and check that the model says it doesn’t know. That instruction is the most important line in the prompt.

Everything a production system adds is a better version of one of these functions: smarter chunking, a vector database instead of a list, hybrid search, re-ranking the results, and checking the citations.

The traps

The retrieval trap: when the right passage isn’t retrieved, the model doesn’t know it’s missing. It answers from whatever it got, fluently. Most “the AI made it up” reports in RAG systems are retrieval failures. Test retrieval on its own, as shown below.

• Bad chunks. Split mid-sentence and a passage loses its meaning; make passages too long and the embedding blurs. A table split from its header is useless. Look at your chunks before you tune anything else.
• Stale index. The index is a copy. When a document changes, its passages must be embedded again, and deleted documents must be removed, or the system keeps quoting the old refund policy.
• Access rights. If the index holds documents not every user may read, filter by the user’s permissions before retrieval. A prompt that says “don’t reveal confidential documents” does not protect them once they are in the context.
• Prompt injection. Retrieved text goes into the prompt, so a document that says “ignore your instructions” is talking to your model. Treat retrieved text as data, never as instructions.
• Cost and caching. Providers bill a repeated start of a prompt at a reduced price through prompt caching. Retrieved passages change with every question, so the prompt changes early and the cache misses. In long conversations, such as a coding agent’s session, that adds up quickly. Put stable instructions first and retrieved text as late as you can.

How do I know if retrieval works?

Test it on its own, before you look at answers. Write down a handful of real questions and which passage should answer each, then check whether that passage comes back in the top results. Save this as recall.py next to rag.py:

from rag import chunks, embed, similarity

tests = [
    ("Can I get my money back on a yearly subscription?", 0),
    ("I forgot my password", 2),
    ("How do I turn on 2FA?", 3),
]

vectors = embed(chunks)
k = 2
hits = 0
for question, expected in tests:
    q = embed([question])[0]
    ranked = sorted(range(len(chunks)), key=lambda i: similarity(q, vectors[i]), reverse=True)
    found = expected in ranked[:k]
    hits += found
    print(f"{'ok  ' if found else 'MISS'} {question}")
print(f"recall@{k}: {hits}/{len(tests)}")

The number at the end, called recall at k, is the share of questions whose right passage made it into the top k. If it’s low, no prompt fixes the answers: change the chunking, the embedding model or add keyword search first, then run the test again.

• Use real questions, in the words your users type, not paraphrases of your own headings. Those are too easy.
• Include questions with no answer in your documents, and check the final answer says so.
• Keep the test file. Run it every time you change chunking, the model or the documents.

What we built, twice, and what it taught

We built RAG twice at Contenox, and deleted it once.

The first was Bob, a hosted document search for teams: a workspace connected its sources, picked an embedding model once, and the index was built over everything that synced in. The honest test it passed was paraphrase: questions worded nothing like the documents, across unrelated domains, still found the right one by meaning rather than word overlap.

The second was a local index over a code workspace, in Go, on SQLite, 3,191 lines including tests. Four decisions from it are worth copying:

• Two legs, fused by rank, not by score. SQLite’s FTS5 ranks the top 200 chunks by bm25; a cosine pass over those same 200 ranks them again by meaning. The two lists are merged with Reciprocal Rank Fusion, score = 1/(60 + rank) summed over both legs, with the constant RRF was published with. Equal weights on purpose: a bm25 score and a cosine are not comparable numbers, a rank is, and without a held-out corpus any weight would be a guess. An identifier like ErrNoIndex is a rare token bm25 finds exactly and an embedding blurs; a sentence that answers the question in other words is invisible to bm25 and found by the vector leg.
• Chunks of 400 tokens with 5 overlapping lines, so a function split across two chunks still has its signature in both. Files over 512 KB are counted and skipped; so are node_modules, dist, .git and the rest of the noise list the file-search tool already used.
• An empty question is refused, not embedded. Embedding whitespace produces a vector that ranks arbitrary chunks highly, which comes back as a confident wrong answer.
• Vectors are bound to the model that made them. Every index generation records a fingerprint of the embedding model and endpoint. Change the model and search refuses until the index is rebuilt, because cosine between two models’ vectors is noise that looks like a score.

func fuse(lexical, vector map[string]int) map[string]float64 {
	score := map[string]float64{}
	for id, rank := range lexical {
		score[id] += 1 / float64(60+rank)
	}
	for id, rank := range vector {
		score[id] += 1 / float64(60+rank)
	}
	return score
}

Then we deleted it for code, for the reasons in the next section. The design is still right for documents; for a repository that changes under your hands, the index was always one branch switch behind.

RAG, long context or search?

Models now read hundreds of thousands of tokens at once, so RAG isn’t the only option.

The third column is where AI coding agents went. Instead of embedding a repository up front, they give the model tools such as file search and let it look things up as it works. We went the same way with Contenox, after building the index above: embeddings for code were impossible to cache, burned tokens with hosted models, and with local models meant re-indexing the project again and again on the same machine you were coding on. Searching the live files has none of those problems.

RAG still wins for large document collections that people ask questions about: help centers, wikis, contracts, research papers. It loses on questions that need everything: “summarize all 400 tickets from last week” isn’t retrieval. Process the documents in batches instead.

Related guides

• What are embeddings?: the search by meaning RAG is built on
• What is a vector database?: where the embeddings live, and whether you need one
• What is MCP?: give an agent search tools instead of a fixed index
• AI coding costs: prompt caching and why prompt order matters
• How to run local LLMs for coding: the Ollama setup this example uses