What Is RAG, and Why Can It Still Give the Wrong Answer?

The quick answer

The United States National Institute of Standards and Technology describes RAG as pairing a generative AI model with a separate information-retrieval system or knowledge base. The retriever identifies relevant information for a query and supplies it to the model without requiring the model itself to be retrained.

That separation is the main attraction. A business can update a policy document today and make the new version searchable tomorrow. It does not need to rebuild the language model whenever a price, rule or procedure changes.

A useful RAG system therefore has two jobs:

1. Find evidence that is relevant to the question.
2. Use that evidence to produce a faithful answer.

Many demonstrations focus on the second job because the final answer is what users see. In practice, weak retrieval is often the deeper problem. A model cannot faithfully use evidence that never reached it.

Think of the system as having two kinds of knowledge

A language model contains patterns learned during training. This is sometimes called parametric knowledge because the information is represented indirectly in the model’s parameters. It is useful, broad and difficult to inspect fact by fact.

A RAG system adds a second store: documents, database records or other material that can be searched at the time of the request. This is often called non-parametric memory or an external knowledge base.

The original 2020 RAG paper by Patrick Lewis and colleagues described this combination as joining a model’s parametric memory with an explicit non-parametric memory. Their work showed that retrieval could improve performance on knowledge-intensive tasks while making it possible to connect outputs to external material.

The word “memory” can be misleading. The system is not remembering documents in the human sense. It is storing searchable representations and retrieving likely matches when a question arrives.

What happens before the answer appears

The user sees one question and one response. Behind that simple exchange, a typical RAG system may perform a dozen operations.

1. Documents are collected

The source may be a folder of PDFs, a company wiki, support tickets, a product database, legislation, web pages or several systems combined.

2. Content is extracted

Text must be pulled from each source. This sounds routine until the collection contains scanned PDFs, multi-column layouts, tables, handwritten notes, screenshots, footnotes or forms. A document that looks readable to a person can be badly damaged during extraction.

3. The text is divided

Large documents are normally broken into smaller passages called chunks. The system stores each chunk together with metadata such as the file name, page number, date, department, language or access level.

4. Search representations are created

The system may index ordinary keywords, numerical embeddings or both. These representations allow it to compare the user’s question with the stored material.

5. Candidate passages are retrieved

Several passages that appear relevant are selected. Some systems then rerank them with a second model or apply filters for dates, document types, permissions or business units.

6. The model receives a temporary evidence packet

The user’s question, instructions and selected passages are placed into the model’s context window. The model is told to answer from the supplied material, often with citations.

7. The answer is generated

The model writes a response. Depending on the design, citations may be attached during generation or added afterward by matching answer text back to retrieved chunks.

Every stage transforms information. The final answer can only be understood by examining the entire route, not the language model alone.

Why documents are cut into chunks—and why chunking matters

A 200-page manual is too large and too unfocused to send in full for every question. RAG systems therefore divide documents into smaller pieces.

If chunks are too large, search results may contain a relevant sentence buried among several pages of unrelated text. The extra material can distract the model and consume context space.

If chunks are too small, meaning can be broken apart. A rule might be in one chunk and its exception in the next. A table heading may be separated from its values. A pronoun such as “it” may lose the paragraph that explains what “it” refers to.

If only Chunk A is retrieved, the answer may be technically supported by a source and still wrong for the person asking.

Good chunking follows the structure of the content. Policies may be divided by numbered clauses. Support documents may be divided by question and answer. Contracts may require sections, definitions and referenced schedules to remain connected. Tables often need special treatment so row and column meaning survives extraction.

There is no universally correct chunk size. It depends on the document type, typical questions, model context limits and how much neighboring text is required to interpret a passage correctly.

Embeddings in plain English

Keyword search looks for matching words. That works well when the question and document use the same language. It may struggle when they describe the same idea differently.

An embedding converts a piece of text into a long list of numbers representing aspects of its meaning. Texts with similar meaning tend to have numerical representations that are closer together.

Embeddings are useful, but they are not a human understanding of meaning. They can place passages near one another because they discuss similar topics while missing the exact distinction the user needs.

For example, “refunds are permitted within 30 days” and “refunds are not permitted after 30 days” are semantically close. A similarity search may retrieve both. The ranking system and language model must preserve the decisive word not.

Many production systems combine semantic search with keyword search. Keywords are strong for exact names, product codes, dates and legal phrases. Embeddings are strong for paraphrases and broader conceptual similarity. Hybrid retrieval can use both signals.

Finding a related passage is not the same as finding the answer

Retrieval systems usually return the passages with the highest scores. A high score means “similar according to this search method,” not “contains the complete and correct answer.”

Consider the question: “Can a contractor approve an emergency purchase?” A company knowledge base may contain:

• a purchasing policy describing emergency approvals;
• a contractor handbook describing general responsibilities;
• an old procedure that has been superseded;
• a project-specific delegation letter;
• a finance FAQ using similar words but applying only to employees.

The system must identify not merely the topic, but the controlling rule for the correct person, date, location and circumstance.

Useful retrieval therefore depends on metadata and filters. The system may need to know:

• which version is current;
• which department owns the rule;
• which country or legal entity it applies to;
• whether the user is allowed to see the document;
• whether the source is authoritative or only advisory;
• whether another document overrides it.

A folder full of documents is not automatically a knowledge base. Without governance, it is a collection of competing statements.

What a citation proves—and what it does not

Citations make a RAG answer easier to inspect. They are valuable. They are not a guarantee of correctness.

A system may cite a passage that is relevant to the subject but does not support the exact statement. It may combine two sources and attach only one. It may state a conclusion more strongly than the source. It may cite an outdated document because that document ranked highly.

The strongest citation design links individual claims to exact passages, preserves document dates and versions, and makes the source easy to open. The system should also be able to say that the available evidence is incomplete or conflicting.

“I found no clear answer in the approved sources” is often a better result than a polished guess.

Where RAG works well

RAG is most useful when the answer depends on a bounded collection of information that changes independently of the model.

RAG is especially valuable when users need to see where the answer came from. It can also reduce the need to paste long documents into every conversation.

It is less suitable when the task requires precise calculation, real-time transactional state, complex authorization or guaranteed completeness. Those needs may require structured databases, deterministic code or direct system queries rather than document retrieval alone.

How RAG systems fail

The phrase “the model hallucinated” is often used for every bad answer. In RAG systems, the failure may have happened much earlier.

The source never entered the collection

The correct policy may be missing, still in an email attachment or stored in a system that was not indexed.

The source was parsed incorrectly

A scanned PDF may have poor text recognition. A table may lose its headings. A multi-column page may be read in the wrong order.

The passage was split badly

The system retrieves a rule without its exception, definition or effective date.

The search query was weak

The system may search using the user’s wording without adding the terminology used in the documents.

The wrong version ranked first

Old documents may be more detailed, contain more matching words or have stronger embeddings than the current version.

The evidence was relevant but insufficient

A passage may mention the subject without resolving the question. The model fills the gap with a plausible conclusion.

Too much context was supplied

More documents do not always help. A crowded context can contain contradictions, duplicated text and distracting material.

The model ignored or distorted the evidence

The correct passage reaches the model, but the answer reverses a condition, drops a qualification or combines unrelated clauses.

The citation was attached loosely

The source supports the topic but not the exact claim beside it.

Each failure requires a different remedy. Improving the language model will not fix missing documents. Increasing the number of retrieved chunks will not fix bad permissions. Adding citations will not fix outdated policies.

RAG creates a security and access-control problem

A knowledge assistant may search salary records, legal advice, customer information, security procedures and ordinary public documents through the same interface. The system must not retrieve information merely because it is relevant. It must retrieve only information the user is permitted to access.

Access rules should be enforced before sensitive passages are placed into the model’s context. Asking the model to “avoid confidential information” is not a sufficient security boundary.

Important controls include:

• document-level and passage-level permissions;
• identity-aware filtering;
• separation between business units or clients;
• audit logs showing which sources were retrieved;
• retention and deletion rules;
• protection against malicious instructions embedded in documents;
• review of third-party storage and model-provider data practices.

Documents themselves may contain hostile content. A retrieved web page or uploaded file can include instructions intended to manipulate the model, reveal hidden data or redirect its behavior. Systems that combine RAG with agents and tools need especially strong boundaries because retrieved text may influence actions, not only answers.

How to test a RAG system properly

A pleasant demonstration is not an evaluation. Real testing must separate retrieval quality from answer quality.

The test set should include easy questions, ambiguous questions, outdated terminology, spelling errors, conflicting documents and questions with no approved answer. It should also include requests from users with different permission levels.

Evaluation should continue after launch. Documents change. Users ask unexpected questions. Search quality can drift as the collection grows. A system that performed well with 500 clean documents may behave differently with 100,000 mixed files.

Useful operational measures include retrieval success rate, supported-claim rate, citation accuracy, refusal quality, permission failures, response time and the proportion of answers corrected by reviewers.

A buyer’s checklist for “chat with your documents” products

Before accepting a claim that a product is grounded, private or source-backed, ask for specific answers.

1. Which source types can it parse reliably?
2. How are scanned pages, tables and images handled?
3. How are documents divided into chunks?
4. Does it use keyword, semantic or hybrid retrieval?
5. Can retrieval be filtered by date, department, region and document status?
6. How are superseded documents excluded?
7. Are permissions inherited from the source system?
8. Can administrators see the exact passages retrieved?
9. Do citations point to pages and passages or only whole files?
10. Can the system decline when support is insufficient?
11. How is retrieval accuracy evaluated?
12. What happens to uploaded documents and generated logs?
13. How quickly do changes and deletions reach the index?
14. Can results be separated by client, tenant or legal entity?
15. How are malicious or untrusted documents treated?

A vendor that cannot explain retrieval clearly may still have a good product. But buyers should not mistake a fluent interface for a controlled knowledge system.

The practical conclusion

RAG is not a feature that simply “adds company knowledge” to an AI. It is an information system with ingestion, parsing, indexing, search, ranking, permissions, generation, citations and monitoring.

When those parts are designed well, RAG can turn a static document collection into a useful question-and-answer service. It can make updates faster, reveal supporting material and keep sensitive knowledge within defined boundaries.

When those parts are weak, the system can produce an answer that is confident, cited and wrong.

The right standard is therefore not whether the AI can quote a document. It is whether the system consistently retrieves the controlling evidence, preserves its conditions, respects access rules and shows enough provenance for a person to verify the result.

Grounding is not the presence of a citation. Grounding is a traceable connection between the question, the correct evidence and the claims in the answer.