RAG explained: how AI looks things up before it answers

Updated 22 Oct 2026 · Reviewed every quarter

Ever noticed an AI answer with little numbered links? That's RAG at work: the AI looks things up before it answers, instead of relying on memory. It's behind AI search engines, "chat with your documents" tools and many company chatbots. Here's how it works in four simple steps, where you're already using it, and what can still go wrong.

Without RAG: a closed-book exam. The AI answers from what it remembers from training. Usually fine, but it can be out of date or simply make things up.

With RAG: an open-book exam. The AI looks things up first, then answers from what it found, and can show you where.

RAG stands for retrieval-augmented generation. The name comes from a 2020 research paper by Facebook AI (now Meta), University College London and New York University. arxiv.org

How it works, in four steps

Follow one everyday question through a company's HR assistant.

1 You ask a question
You type a question into a chatbot, a search box or your company's help assistant.
You ask your company's HR assistant: "How many days of holiday do I get?"
2 It searches first
Before writing anything, it searches a library: the web, your files, or a company's documents. It looks for meaning, not just matching words, so "holiday" also finds "annual leave".
It searches the staff handbook, the leave policy and the HR FAQ.
3 It picks the best few passages
It takes the handful of paragraphs that look most relevant and passes them to the AI model, together with your question.
It finds the paragraph: "Full-time staff get 25 days of annual leave, plus public holidays."
4 It answers, and shows its sources
The AI writes an answer based on those passages, and good tools show where each part came from, so you can check.
"You get 25 days of annual leave, plus public holidays [1]." The [1] links to the leave policy.

The Long Room library at Trinity College Dublin: a long hall of tall wooden bookshelves under an arched ceiling — The Long Room at Trinity College Dublin. Photo: Diliff, CC BY-SA 4.0

Think of a very fast librarian

Ask a librarian a question and they don't recite the answer from memory: they go to the right shelf, pull out the best books, read the relevant pages and tell you what they say, pointing to the page.

RAG does the same in seconds. The "library" can be the whole web, a company's handbook, or just the PDFs you uploaded. The answer can only be as good as what's on the shelves.

You're probably using it already

PerplexitySearches the web for every question and answers with numbered sources.
ChatGPT searchLooks things up on the web when a question needs fresh information, and links its sources.
Google AI OverviewsThe AI summary at the top of Google results is written from the pages it found.
Gemini Notebook (formerly NotebookLM)Answers only from the documents you upload, with a citation for each point.
Microsoft 365 CopilotAnswers from your work emails, chats and files, and only those you're already allowed to open.
Customer help chatbotsMany company chatbots answer from the company's own help pages and policies.

Microsoft, for example, describes Copilot as answering from your work content, limited to what you're allowed to see (Microsoft Learn). For how AI search engines use it, see AI and search.

RAG vs the alternatives

Three ways to get an AI to use information it wasn't trained on.

RAG: look it up
The AI searches a library each time, then answers from what it found.
Best for
Facts that change, large collections of documents, answers you need to check.
Stays up to date?
Yes: update the documents and answers update too.
Effort
Medium: someone has to keep the library tidy.
Fine-tuning: teach it
Extra training that changes the model itself, like sending it on a course.
Best for
A particular style, tone or type of task.
Stays up to date?
No: new facts need new training.
Effort
High: needs examples, time and money.
Paste it in
Copy a document straight into the chat and ask about it.
Best for
One-off questions about a single document.
Stays up to date?
Only for that chat.
Effort
Low, but limited by how much text fits in one chat.

When it goes wrong

RAG cuts down on made-up answers, but it doesn't make AI perfect.

It finds the wrong pageIf the search picks a joke or a bad website, the answer repeats it. Google's AI once told people to put glue on pizza, from a years-old joke on Reddit.More AI fails →Source: blog.google
The library is out of dateIf last year's policy is still in the files, the AI may confidently quote last year's rules.
It mixes up what it readEven with good sources, AI can misquote them. In a BBC test, 13% of quotes from BBC articles were changed or made up.AI and the news →Source: pressgazette.co.uk
It can't find anything, and guessesWhen nothing relevant turns up, some tools still answer from memory. Good ones say "I couldn't find that".

Getting good answers

When you use AI

Look for the source links, and open at least one for anything important.
Ask "Where did you find that?" or "Quote the exact sentence".
For your own documents, use a tool that answers only from what you upload.
Be specific: "our 2026 leave policy" finds better passages than "holidays".

If your business wants one

Clear out old and duplicate documents first: the AI will use whatever it finds.
Check who can see what: the assistant should only show people documents they're allowed to open.
Keep sensitive data out unless the tool and contract protect it.
Test it with 20 real questions you already know the answers to, before rolling it out.

The short version is in the glossary: What is RAG? Related: hallucination, AI agents and using AI without worrying about your privacy.

How we chose the evidence

We explain RAG from the original 2020 research paper and the companies’ own descriptions of their tools (such as Microsoft’s Copilot documentation), with real examples of what goes wrong taken from our AI fails and AI and the news guides. The everyday example of an HR assistant is illustrative. We review this page every quarter.

← All guides