Retrieval-augmented generation end to end: ingestion, embeddings and vector indexes, hybrid search and reranking, retrieval evaluation, permissions, and agentic tool use.
21 published across 7 topics.
RAG, retrieval and agents fundamentals58 short answers on one page, for revising rather than studying.Parsing real documents, chunk size and overlap as a retrieval decision, preserving structure and metadata, and handling tables and scans.
Chunk boundaries decide what can ever be retrieved, so you derive them from the questions the corpus must answer and from the document's own structure, then confirm the choice with retrieval recall on a gold set rather than by eye.
Route each format to a parser that respects its structure, treat OCR output as text with a confidence score rather than as truth, convert tables into rows that carry their headers, and record a section path and stable anchor per chunk so every retrieved passage can be cited back to a place in the original.
Each chunk needs its source document and version, its section path, an anchor precise enough to open, effective and ingestion dates, and the permission set that governed the original — because filtering, citation, freshness and access control are all served from metadata rather than from the vector.
What an embedding encodes, HNSW and IVF trade-offs, recall versus latency, dimensionality and quantisation, and re-embedding after a model change.
Flat search is exact and scales linearly, so it suits small collections and supplies ground truth for the rest. IVF trades recall for speed by probing a subset of clusters, HNSW does it with a graph at higher memory and build cost, and in both recall is a parameter you set rather than inherit.
Vectors from two models live in unrelated spaces, so there is no incremental path — you re-embed and reindex the whole corpus. The real cost is running both indexes side by side long enough to compare them on a gold set, keeping query and document models in step, and retuning thresholds calibrated on the old scores.
An embedding places text in a space where distance approximates the similarity its training data taught it. It does not encode relevance, and it represents exact identifiers, negation and numeric constraints so weakly that dense retrieval alone is unreliable wherever those decide the answer.
Why dense retrieval alone misses exact terms, combining BM25 with vectors, fusion strategies, and where a cross-encoder reranker earns its latency.
Read the failing queries first and sort them into causes, then apply the matching lever — rewriting for vague queries, filters for scope errors, smaller retrieval units for buried answers, lexical retrieval for literal terms, a reranker where recall sits below the cut — each with the metric that proves it worked.
It earns it when your gold set shows recall at 50 is much better than recall at 5, because that gap is what reranking converts into answer quality. The cost is a model pass per candidate with nothing precomputable, so the latency budget divided by measured per-batch cost sets the candidate count.
Dense retrieval handles paraphrase and misses literal tokens, while BM25 does the opposite, so you run both and merge the candidate lists. Merging by rank with reciprocal rank fusion is the robust default, because the two engines produce scores on scales that cannot be normalised reliably per query.
Building a gold set from real questions, recall at k versus end-answer quality, groundedness and citation checks, and diagnosing which stage failed.
Source the questions from real users and from randomly sampled documents rather than from the team, label each with a quotable answer string anchored to a document rather than a chunk id so the set survives re-chunking, and hold part of it back so you are not tuning against your own measurement.
Walk three gates in order for that one request — was the supporting passage retrieved, did it survive into the rendered prompt, does the answer contradict it. Each gate blames a different stage, which is why the diagnosis depends on having logged both the retrieved chunk ids and the prompt as sent.
Recall at k is the primary one, since a passage that never reaches the prompt cannot be used; precision at k measures wasted window; MRR or NDCG measure ordering. Read them beside the groundedness of the answer, because high recall with bad answers points at generation or context ordering.
Incremental updates and deletes, propagation lag, reindexing without downtime, and keeping an index honest about what no longer exists.
Ask the index directly whether the old text is still there, because that single check splits the problem into change detection, queue or write failure, a retire step that left duplicate chunks, or a downstream cache. Then fix the reason you learned about it from a user rather than an alert.
Detect change at the source rather than by re-crawling everything, upsert at document granularity so moved chunk boundaries do not orphan rows, propagate deletions as explicit tombstones the query filter respects, and publish the source-to-searchable lag as a measured SLA.
Build a shadow index beside the live one, dual-write from a recorded change-feed watermark so it never falls behind, verify on reconciliation counts plus a gold set and shadowed production queries, then cut over by moving an alias and keep the old index writable long enough to roll back.
Enforcing per-user permissions inside retrieval, tenant isolation in a shared index, and why post-filtering a result set is the wrong place to start.
Revoke once in the authorisation store, then propagate to the index ACLs, every cache holding text or results, any derived summaries and embeddings, and the conversation transcripts still carrying passages. Answers already delivered cannot be recalled, so the guarantee you offer is a bounded window rather than erasure.
Carry the authorised principal set on every chunk and pass the user's principals into the search as a filter, so the candidate set is legal before ranking. Filtering after retrieval leaks through counts and gaps, and destroys recall for the users with the narrowest access.
Choose between an index per tenant, per-tenant namespaces and one filtered index by weighing recall, blast radius and per-tenant fixed cost. Partitioning turns tenant selectivity from a search-quality problem into a routing decision, which is the argument most candidates miss.
Tool calling and schema design, planning loops and step budgets, idempotency and retries, handing control back to a human, and when an agent is the wrong shape.
Enforce step, token, wall-clock and monetary budgets in the orchestrator rather than in the prompt, detect repetition and lack of progress as separate conditions, and require an approval bound to specific arguments before any irreversible action. Exhausting a budget should hand back partial work, not vanish.
Whenever you can draw the flowchart. A fixed pipeline with a model at the two steps needing judgement is cheaper, testable and debuggable, and per-step error compounds across a planning loop. Reach for an agent when the tool sequence is open-ended and the environment verifies the work.
Give each tool one narrow purpose with a self-describing schema, return failures as structured results the model can act on rather than raising, take an idempotency key on anything that mutates, and put validation and authorisation in the executor because arguments produced by a model are untrusted input.