Retrieval-augmented generation demos beautifully and fails in specific, predictable ways. Here is where it goes wrong and what to do about it.
The demo is the easy part
A retrieval-augmented generation prototype takes an afternoon. Chunk some documents, embed them, retrieve the top few by similarity, hand them to a model with the question. It answers well, everyone is impressed, and the project gets funded.
Then it meets real users, real documents and real edge cases, and the failures are rarely the ones people expect. They are almost never the model being “not smart enough”. They are retrieval returning the wrong passage, the corpus being stale, or the system answering confidently when it should have declined.
“A RAG system that cannot say “I don’t know” is not a knowledge system. It is a confident stranger.”
Retrieval fails before generation does
When a RAG answer is wrong, the instinct is to blame the model or reach for a bigger one. In our experience the passage the model needed was simply never retrieved. Debugging generation when the fault is retrieval wastes a lot of time.
Instrument the retrieval step separately from the answer. For a set of known questions, record whether the correct source appeared in the retrieved context at all. If it did not, no amount of prompt engineering will save the answer, and you now know exactly which half of the system to fix.
The common causes are mundane: chunks split mid-table so the header is separated from the numbers; documents where the answer is spread across sections that never co-occur; and queries phrased in user language against a corpus written in internal jargon.
Chunking is a content problem, not a parameter
Most tutorials suggest a fixed chunk size with some overlap. That works on prose and falls apart on the documents businesses actually keep: tables, forms, policy documents with deep nesting, and slide exports where the meaning lives in the layout.
Chunk along the document’s own structure instead — sections, rows, clauses — and carry the surrounding context into each chunk. A table row is meaningless without its column headers and the heading above it. This is unglamorous preprocessing work, and it moves answer quality more than any model choice.
You may not need a vector database
Vector search exists because you could not fit a corpus into a model’s context. Context windows have grown considerably, and a great many internal knowledge bases — a product manual, a policy set, a service catalogue — now fit comfortably inside one.
If yours does, putting the whole corpus in a cached system prompt removes an entire category of bugs. There is no retrieval to tune, no embedding model to keep in sync, no index to rebuild, and no similarity threshold to guess at. It is frequently cheaper than the infrastructure it replaces.
Vector search earns its place at genuine scale, with strict latency budgets, or where per-user permission trimming has to happen at retrieval time. Reach for it when you have that problem — not by default because the reference architecture had one.
Refusal is a feature you have to build
The single most valuable behaviour in a production RAG system is declining to answer. Models are trained to be helpful, and left alone they will produce a fluent answer from weak or irrelevant context rather than admit the corpus does not cover the question.
The mechanism we rely on is simple: require a citation. If the system cannot point at a specific retrieved passage supporting its answer, it does not return the answer — it hands off. This doubles as an off-topic classifier, because questions outside the corpus naturally produce no citation.
It also changes the failure mode from dangerous to merely disappointing. A user who is told “I can’t answer that, here is a human” is far better served than one given a confident invention.
Without an eval you are guessing
Teams routinely ship prompt changes based on trying three questions by hand. That is not measurement, and it is how systems get quietly worse over months while everyone believes they are improving.
Build a held-out set of real questions with known-good answers before launch — a hundred is plenty to start. Score every change against it automatically. The point is not a perfect score; it is knowing the direction of travel when you swap a model, adjust chunking or rewrite the prompt.
This is also the artefact that makes the system maintainable by someone who did not build it. Without it, nobody downstream can safely change anything.
What we do differently
We instrument retrieval and generation separately, require citations before an answer is returned, and ship an evaluation suite with every AI feature we build. When a corpus fits in context, we say so rather than selling a vector database nobody needed.
The assistant on this site is built the same way, on the same content you are reading. If it cannot ground an answer, it will tell you and point you at a human.
Where ITLabz can help
If you’re tackling problems like these, our engineers have built them in production across lending, fintech, logistics, manufacturing and retail. We’d be glad to compare notes.