· 3 min read · AI Engineering
RAG Is a Data Engineering Problem Wearing a Trench Coat
When retrieval quality disappoints, the model is usually innocent. The documents, chunking, and metadata are where the bodies are buried.
Improve retrieval before you upgrade the model: clean sources, sensible chunks, rich metadata, and a measurable retrieval test set will beat a bigger model almost every time.
Garbage in, fluent garbage out
A language model will happily summarise whatever you retrieve for it, including the outdated policy PDF from 2019 that nobody deleted. If your corpus contains three versions of the same document, your system does not have a knowledge base. It has an argument waiting to happen, and the model will referee it differently every run.
The unglamorous fixes come first: one canonical source per document, deletion pipelines that actually delete, and ingestion jobs you can rerun safely. This is the same discipline as any other data pipeline, because that is exactly what it is.
Chunking is an editorial decision
Slicing documents every fixed number of characters is easy to code and quietly destructive. It cuts sentences in half, separates a heading from the paragraph it describes, and strands tables away from their captions. Chunk along the document's own structure instead: sections, headings, and complete procedures, with enough overlap that a retrieved piece still makes sense on its own.
Metadata does the quiet heavy lifting. Source, date, owner, and access level on every chunk let you filter before you search, cite where an answer came from, and keep a private document out of the wrong user's answer. Retrieval without access metadata is a security incident with good grammar.
“The model gets the credit and the blame, but retrieval decides what it is allowed to know.”
Measure retrieval on its own
Keep a small set of real questions with the passages that should be retrieved for each. When an answer goes wrong, check that set first: did the right passage come back at all? If retrieval missed, no prompt will save you. If retrieval hit and the answer still wandered, then you have a generation problem worth tuning.
That separation turns vague complaints about AI quality into two ordinary engineering questions with owners, tests, and fixes. Ordinary is good. Ordinary ships.