RAG: grounding answers in retrieved context
Models confidently make things up. RAG — Retrieval-Augmented Generation — is the standard fix: instead of asking the model from memory, you retrieve relevant snippets, stuff them into the prompt as context, and instruct the model to answer only from that context. The answer is then *grounded* in real source text you control.
The flow is three steps:
- Retrieve — pull the snippets relevant to the question (this is where your vector store earns its keep).
- Augment — build a prompt that pastes those snippets in under a
Context:heading, with a clear instruction to use only them. - Generate — hand that prompt to the model and return its answer.
The single most important part is the "I don't know" path. When retrieval finds nothing relevant, you must NOT call the model — there's no context to ground it, so it would invent an answer. Instead, return an honest "I don't know" *before* the model is ever called. Refusing to answer when you have no grounding is the difference between a trustworthy AI feature and a plausible-sounding liar.
Syntax
snippets = retrieve(question)
if not snippets:
return "I don't know — no relevant context found." # model NOT called
prompt = build_context_prompt(question, snippets)
return llm(prompt)
# prompt: instruction line, "Context:", each snippet as "- ...", question, "Answer:"Worked examples
Building the grounded context prompt
def build_context_prompt(question, snippets):
parts = ["Answer using ONLY the context below.", "Context:"]
for s in snippets:
parts.append(f"- {s}")
parts.append(f"Question: {question}")
parts.append("Answer:")
return "\n".join(parts)
print(build_context_prompt("How long for a refund?", ["Refunds take 14 days."]))
# Output:
# Answer using ONLY the context below.
# Context:
# - Refunds take 14 days.
# Question: How long for a refund?
# Answer:End-to-end answer when context exists
_KB = {"refund": ["Refunds are processed within 14 days."]}
def retrieve(query):
for key, snippets in _KB.items():
if key in query.lower():
return list(snippets)
return []
def llm_stub(prompt): # in production, a real model call
body = prompt.split("Context:", 1)[1]
snippets = [l[2:] for l in body.splitlines() if l.startswith("- ")]
return "Based on the context: " + " ".join(snippets)
def answer_question(question):
snippets = retrieve(question)
if not snippets:
return "I don't know — no relevant context found."
return llm_stub(build_context_prompt(question, snippets))
print(answer_question("What is your refund policy?"))
# Output:
# Based on the context: Refunds are processed within 14 days.The 'I don't know' path — model is never called
# (retrieve / answer_question as above)
print(answer_question("Do you sell rockets?"))
# Output:
# I don't know — no relevant context found.
#
# retrieve() returned [], so answer_question returns the honest refusal
# BEFORE reaching llm_stub — no chance for the model to invent an answer.Common mistakes & gotchas
Skipping the empty-retrieval guard = hallucinations
If you build a prompt and call the model even when retrieval found nothing, the context block is empty and the model falls back to inventing an answer. The if not snippets: return ... check must return and stop *before* the model call — that early return is the whole safety mechanism, not an optimisation.
Garbage retrieval = garbage answer
RAG is only as good as what you retrieve — a grounded answer over the wrong snippets is still wrong, just confidently so. The quality of your embeddings and your top-k cut-off directly set the ceiling on answer quality. When RAG gives bad answers, inspect what was retrieved before you blame the model.
"Answer from the context" is an instruction, not a lock
Telling the model to use only the provided context strongly biases it but doesn't physically prevent it leaning on training data. Keep the instruction explicit and prominent, and for anything high-stakes, verify the answer's claims actually appear in the retrieved snippets rather than assuming grounding held.
Why it matters
RAG is the backbone of almost every real-world AI product — support bots, documentation search, internal Q&A over company data — because it lets a general model answer about *your* specific, private, up-to-date information without retraining. And the "I don't know" path is what makes it shippable: a feature that admits ignorance beats one that fabricates, every time.