RAG, Retrieval & Agents

What this area actually covers

Retrieval-augmented generation is the practice of answering a question by first finding relevant material and then asking a language model to answer using only that material. Written out like that it sounds like a prompt technique. It is not: it is a search system, an ingestion pipeline, an index with an update problem, a permission model, and an evaluation problem with several stages, and the language model is the last and least interesting component in the chain.

This section covers that whole chain. How documents get parsed and split. What an embedding encodes and what it does not. How an approximate nearest-neighbour index trades recall for latency. Why dense vectors alone fail on exact terms and what to combine them with. How to tell whether a bad answer was a retrieval failure or a generation failure. How to keep an index honest when the underlying documents change and when they are deleted. How to enforce, inside retrieval, that a user only sees what they are entitled to see. And finally agents — systems that call tools in a loop rather than answering once — which belong here because tool use is the generalisation of retrieval and inherits all of its problems plus several of its own.

The reason this is a separate section from Generative AI & Large Language Models is a practical one. In a working enterprise AI system, the prompting and model-selection work is perhaps a fifth of the effort and the retrieval pipeline is most of the rest, and the two are examined differently: a GenAI interview asks how you evaluated the output, and a retrieval interview asks what your recall at ten was before reranking and after. The two sections overlap on evaluation and on injection because a retrieval system is a delivery mechanism for injected content, and each treats those from its own end.

There is an older, shorter treatment of this material on the site. The Data & AI Engineering section has an LLMs & RAG subsection from an earlier wave, which still stands and is a compressed introduction to what is now two sections. It has not been removed. If you read both, treat this as the detailed version.

Two things get wrongly folded in. The first is vector databases as a topic in themselves: choosing a vector store is a genuine decision with genuine trade-offs, and it is a much smaller decision than how you chunk and how you rerank, and candidates who prepare a comparison of vector stores have prepared for the least consequential question. The second is agents as a synonym for AI: an agent is one architecture, appropriate when a task genuinely requires several dependent steps whose sequence is not known in advance, and inappropriate — slower, more expensive, harder to debug, and less reliable — for tasks a single well-retrieved call handles.

The seven areas underneath

These follow the flow of data rather than the flow of a request, which is why ingestion comes first and evaluation sits in the middle. The middle placement is deliberate: you cannot sensibly tune chunking, embeddings or reranking without the measurement in place, so retrieval evaluation is not a final step but a prerequisite disguised as one. Freshness and access control come next because they are the two things that make a demo into a system somebody can use at work. Agents come last because they are a different shape built on the same foundation.

Ingestion & Chunking

Parsing documents that were built for humans — PDFs with two-column layouts, spreadsheets used as forms, slide decks, scans with no text layer at all — and splitting the result into units a retriever can return. It covers chunk size and overlap as retrieval decisions rather than defaults, preserving structural context so a chunk carries its section heading and its document title, attaching metadata you will later want to filter on, and the specific misery of tables. It is first and it is its own subsection because it is where most quality is silently lost: a chunk that has been severed from the heading that gave it meaning is unrecoverable downstream, and no amount of reranking or prompting will restore it. The unglamorous truth of this field is that a great many disappointing RAG systems have a parsing problem.

Embeddings & Vector Indexes

What a text embedding actually encodes, how similarity in that space relates to relevance and where the two diverge, the index structures that make nearest-neighbour search tractable and what each gives up, the relationship between recall and latency as you tune them, dimensionality and quantisation as a memory and cost decision, and the operational event nobody plans for: changing embedding model, which invalidates every vector you have and requires a full re-embed. It is separate because these are the mechanics with real numbers attached, and because "recall at what cost" is the question an interviewer uses to tell whether you have run an index or read about one.

Hybrid Search & Reranking

Why a purely dense retriever fails on product codes, error identifiers, surnames and any term whose value is its exact form; how lexical scoring such as BM25 covers precisely that gap; how to fuse two ranked lists that have incomparable scores; and where a cross-encoder reranker earns the latency it costs. This exists as its own area because it is the highest return per unit of effort in the whole pipeline and the most commonly skipped. A team that adds lexical retrieval and a reranker to a naive dense pipeline typically improves answer quality more than any model upgrade would, and the reasoning — cheap wide recall first, expensive precise scoring on a short list — is a general architectural pattern worth owning.

Retrieval Evaluation

Building a gold set of real questions with the passages that genuinely answer them, measuring recall at k and precision separately from end-answer quality, checking groundedness and whether citations actually support the claim, and — the point of the whole subsection — attributing a bad answer to a stage. It sits in the middle of the section because it is the instrument the other subsections need. Without it, tuning chunk size is superstition and comparing two rerankers is a matter of taste. Expect material on constructing a gold set cheaply, which is the real obstacle, and on why end-to-end satisfaction scores cannot tell you what to change.

Freshness & Incremental Indexing

Applying inserts, updates and deletes to an index incrementally, the lag between a document changing and the change being retrievable, reindexing an entire corpus without downtime, and the specific correctness problem of deletion: a document removed from the source that remains in the index is an active information leak, not merely stale data. It is its own area because freshness is where RAG systems meet ordinary data engineering, and because it is the first thing that breaks after launch. A demo indexes once. A system runs continuously against a corpus that changes, including changing in ways — a document made confidential, an employee offboarded — where being late is a compliance problem.

Access Control & Multi-tenancy

Enforcing per-user permissions inside the retrieval step, isolating tenants that share an index, propagating a permission change into the index promptly, and understanding why filtering results after retrieval is the wrong architecture. This is separate because it is the single most common reason an enterprise RAG pilot never reaches production, and because the correct approach is counter-intuitive to anyone who has only built with a vector store. Post-filtering a top-k result set means the user's ten results become three, silently degrading quality for exactly the people with the least access, and it makes the security property depend on a stage that ran after the search. Filtering must happen inside the search.

Agents & Tool Use

Designing tools and their schemas so a model can call them correctly, planning loops and why they need step and cost budgets, idempotency and retries when a step may run twice, returning control to a human before a consequential action, and recognising when an agent is the wrong shape for the problem. It closes the section because it generalises everything before it — retrieval becomes one tool among many — and because it multiplies the failure modes. This is also where the safety material in Generative AI & Large Language Models stops being theoretical: an agent that can both read untrusted content and call a tool that acts is the configuration in which injection becomes consequential.

The pipeline walked end to end

The value of drawing the pipeline is that it makes stage attribution possible. When someone says "the AI gave a wrong answer", the useful response is a question about which of these stages was responsible, and you cannot ask that question of a system you think of as a single box.

flowchart TD
    A[Source documents] --> B[Parse and chunk<br/>with structure preserved]
    B --> C[Embed and index<br/>plus lexical index]
    D[User question] --> E[Retrieve wide<br/>dense plus keyword]
    C --> E
    E --> F[Filter by permission<br/>inside the search]
    F --> G[Rerank to a short list]
    G --> H[Generate grounded answer<br/>with citations]

Look at where the permission filter sits. Placing it before reranking rather than after generation is the whole architectural argument of this section, and a diagram with that box at the end describes a system that will not pass a security review. The other thing to notice is that two arrows feed the retrieve step: the index built offline and the question arriving online, which is why freshness is a property of the left-hand path and latency is a property of the right-hand one.

Each stage fails in a way that looks identical from the outside — a wrong answer — and distinguishing them is the core diagnostic skill.

The row that surprises people is the fourth. A dense retriever will confidently return semantically similar passages for a query containing an exact identifier and never surface the one document that contains it, and because the failure is confined to a query class that internal testing under-represents, it usually reaches production. This is the concrete reason hybrid search exists.

Where it sits in a real system

A RAG system sits between a document estate nobody has curated and a user who expects a search engine that answers. That position gives it two sets of dependencies that a normal service does not have.

Upstream, it depends on systems that were never designed to be indexed: a document management system, a wiki, a ticketing system, shared drives, and email. Each has its own permission model, its own idea of what a document is, and its own change-notification capability, which is often none. A meaningful fraction of the engineering in these projects is connector work — reading a corpus incrementally, obtaining a change feed or simulating one, and translating the source's permission model into something your index can enforce. That last translation is the hard part, and it is why a project that looked like a fortnight of prompt work becomes a quarter of integration.

Downstream, it feeds a component that will happily answer without adequate material. This asymmetry defines the operational risk: the generation stage does not fail when retrieval fails, it degrades invisibly. A conventional search product that finds nothing shows an empty page and the user adjusts. A RAG product that retrieves nothing relevant produces a fluent answer from whatever it did retrieve, and the user believes it. Designing for this means requiring citations that a reader can check, defining an explicit "I could not find this" path, and monitoring the retrieval scores themselves rather than only the final answers.

Where agents change the calculation

An agent is a loop: the model decides which tool to call, the tool runs, the result comes back, and the model decides again, until it concludes or a budget stops it. Retrieval is then one tool among several. This is a genuine capability increase — a task requiring three dependent lookups is not expressible as a single retrieval — and it comes with an uncomfortable arithmetic.

The reliability of a multi-step loop is roughly the product of the reliability of its steps. A step that is right nine times in ten is a good step, and eight of them in sequence is a system that completes correctly less than half the time. That is not an argument against agents; it is an argument for short loops, for steps that fail loudly rather than returning something wrong, for validating each result rather than trusting it, and for never letting a loop run unbounded. It is also the reason the strongest agent designs in production are narrow: a fixed sequence of steps with a model choosing parameters, rather than an open-ended planner deciding what to do next.

Three engineering consequences follow, and they are the ones an interviewer probes. Idempotency stops being good hygiene and becomes a requirement, because a retried step may have already had its effect and the model has no memory of that. Cost and latency become unbounded unless you bound them, since a loop with no step budget can consume an alarming amount of both on a single request. And the security posture changes entirely: a model that reads untrusted content and can also call a tool with real-world effect is the canonical injection target, which is why the mature pattern is to separate the reading capability from the acting capability and to put a human in front of anything consequential.

Who does this work

The dominant title is AI or LLM application engineer, and in practice the work splits along a line that job adverts rarely draw. On one side is pipeline and index work: connectors, parsing, incremental updates, index tuning, permission propagation. That is data engineering with an unusual sink, and data engineers who move into it are immediately productive. On the other side is relevance work: chunking strategy, hybrid weighting, reranking, gold sets, error analysis. That is information retrieval, and search engineers who move into it discover that most of what they know still applies, because it does.

Machine learning engineers own embedding model selection, re-embedding operations, and evaluation methodology. AI platform engineers build the shared retrieval service, the evaluation harness and the guardrails so that five product teams do not each solve permissions differently — badly, in four cases. Solution architects on AI programmes spend their time on the questions that decide whether a project is viable at all: which corpora are in scope, whose permission model governs them, what the compliance position is on indexing a given store, and what happens at the boundary between two systems that disagree about who may read what.

A week in this work is less exotic than the field's reputation. You read a sample of retrieved chunks for questions the system got wrong, and find that a parser dropped table cells. You add lexical retrieval because support engineers search by error code. You discover that the permission filter is applied after retrieval and that this has been quietly degrading results for contractors. You build a gold set of forty questions because you cannot otherwise justify a change. You explain to a stakeholder that the reason the answer was wrong is that the document saying otherwise was updated in March and your reindex is monthly.

Demand, adoption and how that is changing

Demand is very high and it is the most concentrated demand in the AI hiring market: the single most commonly funded enterprise AI project is a question-answering assistant over an organisation's own documents, and that project is a retrieval project. The reason it keeps being funded is that the corpus is real, the pain is real, and no previous technology made the material answerable — enterprise search returned links and users wanted answers.

Adoption has moved through a recognisable arc that shapes what interviews now ask. The first phase was demonstration, where a naive dense pipeline over a sample corpus was impressive enough to secure a budget. The second phase, which is where a large number of organisations currently sit, is the discovery that the demo does not survive the real corpus, the real permission model or the real update cadence. Hiring is being driven by that second phase, which is why the interesting questions are now about stage attribution, permissions and freshness rather than about architecture diagrams. If you can speak credibly about why a pilot stalled and what you changed, you are answering the question actually being asked.

Two shifts are worth naming without over-predicting. The component layer is consolidating: established search engines have added vector capability, general-purpose databases have added vector indexes, and providers ship retrieval and tool-calling features directly, which means the "which vector database" question has become less consequential and the relevance and permissions questions have become more so. And the agent story is in an earlier and noisier phase than the retrieval story — genuinely useful narrow agents are in production, open-ended autonomous ones much less so, and a candidate who describes agents as a solved architecture is signalling that they have not operated one.

What makes it hard

Retrieval quality dominates model choice, and this is counter-intuitive to everyone. The instinct when answers are poor is to change the model. In a retrieval system the model can only work with what it was given, so if the passage containing the answer was not in the top ten, no model will produce the answer. Time spent on chunking, hybrid retrieval and reranking almost always outperforms time spent on model comparison, and the discipline of checking what was retrieved before blaming what was generated is the single most valuable habit in this area.

Permissions inside retrieval are architecturally awkward. Every access-control system you have built filters a result set. Here the filter has to run inside an approximate nearest-neighbour search, because filtering afterwards silently truncates results and makes quality a function of privilege. That means the index must carry authorisation metadata, that metadata must be updated when permissions change — promptly, since the failure mode is a leak — and the source system's permission model must be faithfully translated into whatever your index can express. When the source uses inherited folder permissions, group nesting and per-item overrides, that translation is genuinely difficult and is where these projects stall.

Freshness is a correctness property, not a performance one. Stale content produces an answer that is confidently wrong with a citation attached, which is worse than no answer. Deletion is the sharp edge: a document withdrawn from the source and still present in your index will be retrieved and quoted, and depending on why it was withdrawn that is an incident. Incremental indexing sounds routine until you meet a source with no change feed, at which point you are diffing a corpus on a schedule and reasoning about how many hours of lag is acceptable.

Evaluating a multi-stage system requires stage-level instrumentation. An end-to-end score tells you the system got worse and nothing about which of six stages caused it. Building per-stage measurement means constructing a gold set that maps questions to the passages that answer them, which is manual work that nobody wants to fund and which is the difference between improving the system and changing it. Expect an interviewer to ask how you built your gold set, because the answer reveals whether you have done this.

Chunking has no correct answer, only a trade-off you must be able to state. Small chunks retrieve precisely and arrive without context. Large chunks carry context and dilute the embedding, so the relevant sentence is averaged away with the irrelevant paragraph around it. The resolution is usually structural rather than numeric — split on the document's own boundaries, keep headings with their content, retrieve small and expand to the surrounding region before generating — and a candidate who answers a chunking question with a token count has missed the point.

Agents multiply failure modes rather than adding them. Every additional step is another place to go wrong, another retry that may duplicate an effect, another tool result the model may misread, and more cost and latency. Debugging is harder because the failure is a trajectory rather than a call, so you need traces that record every step's input and output or you are guessing. And the security surface expands sharply the moment an agent can both read untrusted content and take action.

Why study it

Study it if you want the AI skill with the most enterprise demand per unit of hype. The retrieval assistant is the project organisations are actually funding, the failure modes are concrete and diagnosable, and the expertise is transferable across employers in a way that familiarity with one provider's features is not. Study it if you like systems where the answer comes from careful measurement rather than from taste, because stage attribution against a gold set is exactly that.

It is also a good route in for two groups who often assume AI work is closed to them. Data engineers already own most of the ingestion, incremental update and permission-propagation problem and need only the relevance material. Search engineers already own the relevance half and find that BM25, fusion and reranking have not stopped mattering. In both cases the gap is smaller than it appears from outside.

Be honest about who should look elsewhere. If you want to work on models — training, adaptation, architectures — this is systems engineering and Machine Learning is the direction. If you are interviewing for a role centred on prompting, evaluation and inference economics rather than on a corpus, Generative AI & Large Language Models is the closer fit. And if your interest is specifically autonomous agents, be aware you are studying the least settled material in the section: it is worth understanding, and building a career on it today is a bet rather than a plan.

Your first hour

Build the smallest honest retrieval evaluation you can, over documents you already have. Do not build a chatbot. The exercise is to find out which stage fails.

Take thirty documents you know well — internal wiki pages, a project's docs, a set of PDFs. Chunk them twice, once naively at a fixed size with no overlap and once on their own headings so each chunk carries its section title. Index both. Then write fifteen questions of three deliberate kinds: five ordinary conceptual questions, five that hinge on an exact term such as an identifier, a version string or a person's surname, and five whose answer is genuinely not in the corpus. For each question, note by hand which chunk should be returned.

Now run the questions against both indexes and record only whether the correct chunk appears in the top five.

Question                          Naive  Headings   Note
What does the retry policy do      yes    yes
Who owns the billing service       no     yes        Naive chunk lost the heading
Error code TX-4180 meaning         no     no         Exact term - dense retrieval cannot see it
Which region is the DR site        yes    yes
Cost of the enterprise tier        no     no         Correctly absent from corpus

Recall at 5 - naive 7/15, headings 11/15
All 5 exact-term questions fail in both indexes
The 5 absent-answer questions must be judged on refusal, not recall

The finding you are looking for is the third line. Both indexes fail every exact-term question, and no amount of chunking work fixes it, because the fix is a lexical index alongside the vector one. That single result — reached in an hour, on your own documents — is the argument for hybrid search made from evidence rather than from reading, and it is a far better interview answer than a description of an architecture.

If you have longer, add the keyword index and re-run. Then take the five questions whose answers are absent and check what the system does with them, because a retrieval system that cannot say "not found" will hallucinate exactly where it matters most.

What this is not

It is not a vector database tutorial. Choosing a vector store is a real decision about operations, cost and whether you want another system to run, and it is a smaller lever than chunking, hybrid retrieval or reranking. A candidate who prepares a store comparison has prepared for a question that is rarely the deciding one.

It is not semantic search as a complete answer. Dense retrieval is one signal, strong on paraphrase and meaning, structurally blind to exact form. Any production system over a corpus containing identifiers, codes or names needs lexical retrieval as well, and treating embeddings as a replacement for keyword search rather than a complement to it is the most expensive common mistake in the field.

It is not fine-tuning, and the two are routinely offered as alternatives when they answer different questions. Retrieval supplies facts the model does not have and can be updated in minutes; adaptation changes behaviour, format and task shape and requires a dataset and a retraining cycle. "Fine-tune it on our documents" is usually the wrong instinct, and saying why is a good interview answer. The adaptation material is in Generative AI & Large Language Models.

It is not agents, despite the two being sold together. Most valuable retrieval systems answer once, and adding a planning loop to a problem a single well-retrieved call handles buys latency, cost and new failure modes for no gain. Recognising when the loop is unnecessary is part of the skill.

And it is not solved by a longer context window. Putting an entire corpus in the prompt is not retrieval — it is expensive, it is slow, accuracy degrades as relevant material sits further from the edges of a long context, and it has no permission model at all, which by itself rules it out for the enterprise use case that funds most of this work.

When an answer is wrong, look at what was retrieved before you look at what was generated. Most of the time the model was faithful to material that should never have reached it.