Most enterprise AI is a retrieval system with a language model at the end of it, and retrieval quality decides the answer far more than model choice does. Permissions, freshness and stage-level evaluation are where these projects actually fail.
Very high demand21 min readAI / LLM Application Engineer, Search Engineer, Machine Learning Engineer, Data Engineer on an AI product, AI Platform Engineer, Solution Architect for AI programmesUpdated 2026-07-28
Assumes you know: Comfortable building a service that reads from a datastore and calls an API, Some exposure to search, ranking or information retrieval concepts, Basic understanding of how a language model consumes a prompt
Overview
What this area actually covers
Retrieval-augmented generation is the practice of answering a question by first finding
relevant material and then asking a language model to answer using only that material.
Written out like that it sounds like a prompt technique. It is not: it is a search system,
an ingestion pipeline, an index with an update problem, a permission model, and an
evaluation problem with several stages, and the language model is the last and least
interesting component in the chain.
This section covers that whole chain. How documents get parsed and split. What an
embedding encodes and what it does not. How an approximate nearest-neighbour index trades
recall for latency. Why dense vectors alone fail on exact terms and what to combine them
with. How to tell whether a bad answer was a retrieval failure or a generation failure.
How to keep an index honest when the underlying documents change and when they are deleted.
How to enforce, inside retrieval, that a user only sees what they are entitled to see. And
finally agents — systems that call tools in a loop rather than answering once — which
belong here because tool use is the generalisation of retrieval and inherits all of its
problems plus several of its own.
The reason this is a separate section from Generative AI & Large Language
Models is a practical one. In a working enterprise AI system, the
prompting and model-selection work is perhaps a fifth of the effort and the retrieval
pipeline is most of the rest, and the two are examined differently: a GenAI interview asks
how you evaluated the output, and a retrieval interview asks what your recall at ten was
before reranking and after. The two sections overlap on evaluation and on injection
because a retrieval system is a delivery mechanism for injected content, and each treats
those from its own end.
There is an older, shorter treatment of this material on the site. The
Data & AI Engineering section has an LLMs & RAG subsection from an
earlier wave, which still stands and is a compressed introduction to what is now two
sections. It has not been removed. If you read both, treat this as the detailed version.
Two things get wrongly folded in. The first is vector databases as a topic in themselves:
choosing a vector store is a genuine decision with genuine trade-offs, and it is a much
smaller decision than how you chunk and how you rerank, and candidates who prepare a
comparison of vector stores have prepared for the least consequential question. The second
is agents as a synonym for AI: an agent is one architecture, appropriate when a task
genuinely requires several dependent steps whose sequence is not known in advance, and
inappropriate — slower, more expensive, harder to debug, and less reliable — for tasks a
single well-retrieved call handles.
These follow the flow of data rather than the flow of a request, which is why ingestion
comes first and evaluation sits in the middle. The middle placement is deliberate: you
cannot sensibly tune chunking, embeddings or reranking without the measurement in place,
so retrieval evaluation is not a final step but a prerequisite disguised as one. Freshness
and access control come next because they are the two things that make a demo into a
system somebody can use at work. Agents come last because they are a different shape built
on the same foundation.
Subsection
What it is for
Ingestion & Chunking
Turning real documents into retrievable units without losing their meaning
Embeddings & Vector Indexes
Representing meaning numerically, and searching that space fast enough
Hybrid Search & Reranking
Fixing what dense retrieval misses, and paying for precision where it counts
Retrieval Evaluation
Measuring each stage so you know which one to fix
Freshness & Incremental Indexing
Keeping the index truthful as the corpus changes
Access Control & Multi-tenancy
Making retrieval itself respect who is asking
Agents & Tool Use
Multi-step systems that act, and the failure modes that come with them
Ingestion & Chunking
Parsing documents that were built for humans — PDFs with two-column layouts, spreadsheets
used as forms, slide decks, scans with no text layer at all — and splitting the result into
units a retriever can return. It covers chunk size and overlap as retrieval decisions
rather than defaults, preserving structural context so a chunk carries its section heading
and its document title, attaching metadata you will later want to filter on, and the
specific misery of tables. It is first and it is its own subsection because it is where
most quality is silently lost: a chunk that has been severed from the heading that gave it
meaning is unrecoverable downstream, and no amount of reranking or prompting will restore
it. The unglamorous truth of this field is that a great many disappointing RAG systems have
a parsing problem.
Embeddings & Vector Indexes
What a text embedding actually encodes, how similarity in that space relates to relevance
and where the two diverge, the index structures that make nearest-neighbour search
tractable and what each gives up, the relationship between recall and latency as you tune
them, dimensionality and quantisation as a memory and cost decision, and the operational
event nobody plans for: changing embedding model, which invalidates every vector you have
and requires a full re-embed. It is separate because these are the mechanics with real
numbers attached, and because "recall at what cost" is the question an interviewer uses to
tell whether you have run an index or read about one.
Hybrid Search & Reranking
Why a purely dense retriever fails on product codes, error identifiers, surnames and any
term whose value is its exact form; how lexical scoring such as BM25 covers precisely that
gap; how to fuse two ranked lists that have incomparable scores; and where a cross-encoder
reranker earns the latency it costs. This exists as its own area because it is the highest
return per unit of effort in the whole pipeline and the most commonly skipped. A team that
adds lexical retrieval and a reranker to a naive dense pipeline typically improves answer
quality more than any model upgrade would, and the reasoning — cheap wide recall first,
expensive precise scoring on a short list — is a general architectural pattern worth
owning.
Retrieval Evaluation
Building a gold set of real questions with the passages that genuinely answer them,
measuring recall at k and precision separately from end-answer quality, checking
groundedness and whether citations actually support the claim, and — the point of the whole
subsection — attributing a bad answer to a stage. It sits in the middle of the section
because it is the instrument the other subsections need. Without it, tuning chunk size is
superstition and comparing two rerankers is a matter of taste. Expect material on
constructing a gold set cheaply, which is the real obstacle, and on why end-to-end
satisfaction scores cannot tell you what to change.
Freshness & Incremental Indexing
Applying inserts, updates and deletes to an index incrementally, the lag between a document
changing and the change being retrievable, reindexing an entire corpus without downtime,
and the specific correctness problem of deletion: a document removed from the source that
remains in the index is an active information leak, not merely stale data. It is its own
area because freshness is where RAG systems meet ordinary data engineering, and because it
is the first thing that breaks after launch. A demo indexes once. A system runs
continuously against a corpus that changes, including changing in ways — a document made
confidential, an employee offboarded — where being late is a compliance problem.
Access Control & Multi-tenancy
Enforcing per-user permissions inside the retrieval step, isolating tenants that share an
index, propagating a permission change into the index promptly, and understanding why
filtering results after retrieval is the wrong architecture. This is separate because it is
the single most common reason an enterprise RAG pilot never reaches production, and because
the correct approach is counter-intuitive to anyone who has only built with a vector store.
Post-filtering a top-k result set means the user's ten results become three, silently
degrading quality for exactly the people with the least access, and it makes the security
property depend on a stage that ran after the search. Filtering must happen inside the
search.
Agents & Tool Use
Designing tools and their schemas so a model can call them correctly, planning loops and
why they need step and cost budgets, idempotency and retries when a step may run twice,
returning control to a human before a consequential action, and recognising when an agent
is the wrong shape for the problem. It closes the section because it generalises
everything before it — retrieval becomes one tool among many — and because it multiplies
the failure modes. This is also where the safety material in
Generative AI & Large Language Models stops being theoretical: an
agent that can both read untrusted content and call a tool that acts is the configuration
in which injection becomes consequential.
The pipeline walked end to end
The value of drawing the pipeline is that it makes stage attribution possible. When someone
says "the AI gave a wrong answer", the useful response is a question about which of these
stages was responsible, and you cannot ask that question of a system you think of as a
single box.
Loading diagram…
Look at where the permission filter sits. Placing it before reranking rather than after
generation is the whole architectural argument of this section, and a diagram with that box
at the end describes a system that will not pass a security review. The other thing to
notice is that two arrows feed the retrieve step: the index built offline and the question
arriving online, which is why freshness is a property of the left-hand path and latency is
a property of the right-hand one.
Each stage fails in a way that looks identical from the outside — a wrong answer — and
distinguishing them is the core diagnostic skill.
Stage
How it fails
What the failure looks like
How you detect it
Parsing
Text is missing or scrambled
Confident answer with no basis
Search the index for a phrase you know exists
Chunking
Chunk severed from its context
Right topic, wrong entity or period
Read the retrieved chunks as a human would
Embedding and index
Relevant passage not in top k
Plausible answer from adjacent material
Recall at k against a gold set
Lexical gap
Exact term never matches
Fails only on codes, names, identifiers
Test a query set of exact identifiers
Reranking
Right passage retrieved, ranked low
Answer built from the second-best source
Compare ranks before and after reranking
Permissions
Retrieval returns what it should not
Correct answer, wrong audience
Query as a low-privilege user
Freshness
Index disagrees with source
Answer that was true last month
Diff a sample of source against index
Generation
Material was present, answer ignores it
Answer contradicts its own citation
Groundedness check against retrieved text
The row that surprises people is the fourth. A dense retriever will confidently return
semantically similar passages for a query containing an exact identifier and never surface
the one document that contains it, and because the failure is confined to a query class
that internal testing under-represents, it usually reaches production. This is the concrete
reason hybrid search exists.
Where it sits in a real system
A RAG system sits between a document estate nobody has curated and a user who expects a
search engine that answers. That position gives it two sets of dependencies that a normal
service does not have.
Upstream, it depends on systems that were never designed to be indexed: a document
management system, a wiki, a ticketing system, shared drives, and email. Each has its own
permission model, its own idea of what a document is, and its own change-notification
capability, which is often none. A meaningful fraction of the engineering in these projects
is connector work — reading a corpus incrementally, obtaining a change feed or simulating
one, and translating the source's permission model into something your index can enforce.
That last translation is the hard part, and it is why a project that looked like a
fortnight of prompt work becomes a quarter of integration.
Downstream, it feeds a component that will happily answer without adequate material. This
asymmetry defines the operational risk: the generation stage does not fail when retrieval
fails, it degrades invisibly. A conventional search product that finds nothing shows an
empty page and the user adjusts. A RAG product that retrieves nothing relevant produces a
fluent answer from whatever it did retrieve, and the user believes it. Designing for this
means requiring citations that a reader can check, defining an explicit "I could not find
this" path, and monitoring the retrieval scores themselves rather than only the final
answers.
Where agents change the calculation
An agent is a loop: the model decides which tool to call, the tool runs, the result comes
back, and the model decides again, until it concludes or a budget stops it. Retrieval is
then one tool among several. This is a genuine capability increase — a task requiring three
dependent lookups is not expressible as a single retrieval — and it comes with an
uncomfortable arithmetic.
The reliability of a multi-step loop is roughly the product of the reliability of its
steps. A step that is right nine times in ten is a good step, and eight of them in sequence
is a system that completes correctly less than half the time. That is not an argument
against agents; it is an argument for short loops, for steps that fail loudly rather than
returning something wrong, for validating each result rather than trusting it, and for
never letting a loop run unbounded. It is also the reason the strongest agent designs in
production are narrow: a fixed sequence of steps with a model choosing parameters, rather
than an open-ended planner deciding what to do next.
Three engineering consequences follow, and they are the ones an interviewer probes.
Idempotency stops being good hygiene and becomes a requirement, because a retried step may
have already had its effect and the model has no memory of that. Cost and latency become
unbounded unless you bound them, since a loop with no step budget can consume an
alarming amount of both on a single request. And the security posture changes entirely: a
model that reads untrusted content and can also call a tool with real-world effect is the
canonical injection target, which is why the mature pattern is to separate the reading
capability from the acting capability and to put a human in front of anything consequential.
Who does this work
The dominant title is AI or LLM application engineer, and in practice the work splits along
a line that job adverts rarely draw. On one side is pipeline and index work: connectors,
parsing, incremental updates, index tuning, permission propagation. That is data
engineering with an unusual sink, and data engineers who move into it are immediately
productive. On the other side is relevance work: chunking strategy, hybrid weighting,
reranking, gold sets, error analysis. That is information retrieval, and search engineers
who move into it discover that most of what they know still applies, because it does.
Machine learning engineers own embedding model selection, re-embedding operations, and
evaluation methodology. AI platform engineers build the shared retrieval service, the
evaluation harness and the guardrails so that five product teams do not each solve
permissions differently — badly, in four cases. Solution architects on AI programmes spend
their time on the questions that decide whether a project is viable at all: which corpora
are in scope, whose permission model governs them, what the compliance position is on
indexing a given store, and what happens at the boundary between two systems that disagree
about who may read what.
A week in this work is less exotic than the field's reputation. You read a sample of
retrieved chunks for questions the system got wrong, and find that a parser dropped table
cells. You add lexical retrieval because support engineers search by error code. You
discover that the permission filter is applied after retrieval and that this has been
quietly degrading results for contractors. You build a gold set of forty questions because
you cannot otherwise justify a change. You explain to a stakeholder that the reason the
answer was wrong is that the document saying otherwise was updated in March and your
reindex is monthly.
Demand, adoption and how that is changing
Demand is very high and it is the most concentrated demand in the AI hiring market: the
single most commonly funded enterprise AI project is a question-answering assistant over an
organisation's own documents, and that project is a retrieval project. The reason it keeps
being funded is that the corpus is real, the pain is real, and no previous technology made
the material answerable — enterprise search returned links and users wanted answers.
Adoption has moved through a recognisable arc that shapes what interviews now ask. The first
phase was demonstration, where a naive dense pipeline over a sample corpus was impressive
enough to secure a budget. The second phase, which is where a large number of organisations
currently sit, is the discovery that the demo does not survive the real corpus, the real
permission model or the real update cadence. Hiring is being driven by that second phase,
which is why the interesting questions are now about stage attribution, permissions and
freshness rather than about architecture diagrams. If you can speak credibly about why a
pilot stalled and what you changed, you are answering the question actually being asked.
Two shifts are worth naming without over-predicting. The component layer is consolidating:
established search engines have added vector capability, general-purpose databases have
added vector indexes, and providers ship retrieval and tool-calling features directly, which
means the "which vector database" question has become less consequential and the relevance
and permissions questions have become more so. And the agent story is in an earlier and
noisier phase than the retrieval story — genuinely useful narrow agents are in production,
open-ended autonomous ones much less so, and a candidate who describes agents as a solved
architecture is signalling that they have not operated one.
What makes it hard
Retrieval quality dominates model choice, and this is counter-intuitive to everyone. The
instinct when answers are poor is to change the model. In a retrieval system the model can
only work with what it was given, so if the passage containing the answer was not in the top
ten, no model will produce the answer. Time spent on chunking, hybrid retrieval and
reranking almost always outperforms time spent on model comparison, and the discipline of
checking what was retrieved before blaming what was generated is the single most valuable
habit in this area.
Permissions inside retrieval are architecturally awkward. Every access-control system you
have built filters a result set. Here the filter has to run inside an approximate
nearest-neighbour search, because filtering afterwards silently truncates results and makes
quality a function of privilege. That means the index must carry authorisation metadata, that
metadata must be updated when permissions change — promptly, since the failure mode is a
leak — and the source system's permission model must be faithfully translated into whatever
your index can express. When the source uses inherited folder permissions, group nesting and
per-item overrides, that translation is genuinely difficult and is where these projects
stall.
Freshness is a correctness property, not a performance one. Stale content produces an
answer that is confidently wrong with a citation attached, which is worse than no answer.
Deletion is the sharp edge: a document withdrawn from the source and still present in your
index will be retrieved and quoted, and depending on why it was withdrawn that is an
incident. Incremental indexing sounds routine until you meet a source with no change feed,
at which point you are diffing a corpus on a schedule and reasoning about how many hours of
lag is acceptable.
Evaluating a multi-stage system requires stage-level instrumentation. An end-to-end
score tells you the system got worse and nothing about which of six stages caused it.
Building per-stage measurement means constructing a gold set that maps questions to the
passages that answer them, which is manual work that nobody wants to fund and which is the
difference between improving the system and changing it. Expect an interviewer to ask how
you built your gold set, because the answer reveals whether you have done this.
Chunking has no correct answer, only a trade-off you must be able to state. Small chunks
retrieve precisely and arrive without context. Large chunks carry context and dilute the
embedding, so the relevant sentence is averaged away with the irrelevant paragraph around
it. The resolution is usually structural rather than numeric — split on the document's own
boundaries, keep headings with their content, retrieve small and expand to the surrounding
region before generating — and a candidate who answers a chunking question with a token
count has missed the point.
Agents multiply failure modes rather than adding them. Every additional step is another
place to go wrong, another retry that may duplicate an effect, another tool result the model
may misread, and more cost and latency. Debugging is harder because the failure is a
trajectory rather than a call, so you need traces that record every step's input and output
or you are guessing. And the security surface expands sharply the moment an agent can both
read untrusted content and take action.
Why study it
Study it if you want the AI skill with the most enterprise demand per unit of hype. The
retrieval assistant is the project organisations are actually funding, the failure modes are
concrete and diagnosable, and the expertise is transferable across employers in a way that
familiarity with one provider's features is not. Study it if you like systems where the
answer comes from careful measurement rather than from taste, because stage attribution
against a gold set is exactly that.
It is also a good route in for two groups who often assume AI work is closed to them. Data
engineers already own most of the ingestion, incremental update and permission-propagation
problem and need only the relevance material. Search engineers already own the relevance
half and find that BM25, fusion and reranking have not stopped mattering. In both cases the
gap is smaller than it appears from outside.
Be honest about who should look elsewhere. If you want to work on models — training,
adaptation, architectures — this is systems engineering and
Machine Learning is the direction. If you are interviewing for a
role centred on prompting, evaluation and inference economics rather than on a corpus,
Generative AI & Large Language Models is the closer fit. And if
your interest is specifically autonomous agents, be aware you are studying the least
settled material in the section: it is worth understanding, and building a career on it
today is a bet rather than a plan.
Your first hour
Build the smallest honest retrieval evaluation you can, over documents you already have. Do
not build a chatbot. The exercise is to find out which stage fails.
Take thirty documents you know well — internal wiki pages, a project's docs, a set of PDFs.
Chunk them twice, once naively at a fixed size with no overlap and once on their own
headings so each chunk carries its section title. Index both. Then write fifteen questions
of three deliberate kinds: five ordinary conceptual questions, five that hinge on an exact
term such as an identifier, a version string or a person's surname, and five whose answer is
genuinely not in the corpus. For each question, note by hand which chunk should be returned.
Now run the questions against both indexes and record only whether the correct chunk appears
in the top five.
The finding you are looking for is the third line. Both indexes fail every exact-term
question, and no amount of chunking work fixes it, because the fix is a lexical index
alongside the vector one. That single result — reached in an hour, on your own documents —
is the argument for hybrid search made from evidence rather than from reading, and it is a
far better interview answer than a description of an architecture.
If you have longer, add the keyword index and re-run. Then take the five questions whose
answers are absent and check what the system does with them, because a retrieval system that
cannot say "not found" will hallucinate exactly where it matters most.
What this is not
It is not a vector database tutorial. Choosing a vector store is a real decision about
operations, cost and whether you want another system to run, and it is a smaller lever than
chunking, hybrid retrieval or reranking. A candidate who prepares a store comparison has
prepared for a question that is rarely the deciding one.
It is not semantic search as a complete answer. Dense retrieval is one signal, strong on
paraphrase and meaning, structurally blind to exact form. Any production system over a
corpus containing identifiers, codes or names needs lexical retrieval as well, and treating
embeddings as a replacement for keyword search rather than a complement to it is the most
expensive common mistake in the field.
It is not fine-tuning, and the two are routinely offered as alternatives when they answer
different questions. Retrieval supplies facts the model does not have and can be updated in
minutes; adaptation changes behaviour, format and task shape and requires a dataset and a
retraining cycle. "Fine-tune it on our documents" is usually the wrong instinct, and saying
why is a good interview answer. The adaptation material is in
Generative AI & Large Language Models.
It is not agents, despite the two being sold together. Most valuable retrieval systems
answer once, and adding a planning loop to a problem a single well-retrieved call handles
buys latency, cost and new failure modes for no gain. Recognising when the loop is
unnecessary is part of the skill.
And it is not solved by a longer context window. Putting an entire corpus in the prompt is
not retrieval — it is expensive, it is slow, accuracy degrades as relevant material sits
further from the edges of a long context, and it has no permission model at all, which by
itself rules it out for the enterprise use case that funds most of this work.
When an answer is wrong, look at what was retrieved before you look at what was
generated. Most of the time the model was faithful to material that should never have
reached it.