How do you choose between a flat index, IVF and HNSW for vector search?

Start by asking whether you need an approximate index at all

A flat index does no indexing. It stores the vectors and, for each query, computes the distance to every one of them, which makes it exactly correct by construction. Its cost is a full scan per query, and the scan is a dense matrix operation that modern hardware executes extremely well, so the collection size at which it becomes too slow is larger than people expect — particularly if the query rate is modest and the vectors are a few hundred dimensions.

That matters because the flat index is the free option: nothing to build, nothing to tune, no recall to lose, and updates are appends. For a corpus of a few tens of thousands of chunks serving occasional internal queries, choosing HNSW is complexity bought for nothing. And even when you do need an approximate index, you keep the flat path available on a sample, because it is the only way to know the ground truth against which the approximate index is measured.

IVF partitions the space, HNSW walks a graph

An inverted-file index clusters the vectors — conventionally with k-means into nlist cells — and stores each vector under its nearest centroid. A query compares itself to the centroids, picks the nprobe closest cells, and scans only those. The saving is the ratio of probed cells to total cells, and the loss is every true neighbour that happens to live in a cell you did not probe. That loss is real and structural: near a cell boundary, the nearest vector to your query is often on the other side of the boundary.

HNSW builds a multi-layer proximity graph. Sparse upper layers give long-range links used to arrive in the right neighbourhood quickly, and the dense bottom layer is walked greedily with a candidate list to refine. Search cost grows with the logarithm of the collection rather than linearly, which is why it dominates latency-sensitive deployments. The price is memory — you are storing graph edges per node in addition to the vectors — and build time, since inserting a node means searching the graph to find its neighbours.

	Recall	Query latency	Memory	Build cost	Updates
Flat	exact	linear in corpus	vectors only	none	append
IVF	tunable via nprobe	scan of probed cells	vectors plus centroids	clustering pass over a training sample	append, but centroids drift
HNSW	tunable via efSearch	logarithmic in corpus	vectors plus graph edges	graph search per insert	insert is fine, delete is awkward

The update column is the one candidates skip and operators care about. IVF needs a representative training sample to place its centroids, so an index trained on early data and then filled with a corpus that has drifted will have unbalanced cells and degraded recall long before anyone suspects the index. HNSW handles insertion natively, but true deletion means unlinking a node from a graph whose connectivity was the point, so implementations typically mark the node and filter it out at query time. Tombstones accumulate, and a corpus with heavy churn eventually needs a rebuild whether or not the API exposes one.

Recall is a setting, not a property

The most useful thing to say on this question is that "HNSW gives you high recall" is not a meaningful claim. Both approximate structures expose a knob that trades work for accuracy — nprobe for IVF, efSearch for HNSW — and both let you drive recall towards exact at a cost that approaches a full scan. There are also build-time parameters that set the ceiling, M and efConstruction for HNSW and nlist for IVF, and those you cannot change afterwards without rebuilding.

So the engineering task is not to select an index type and inherit its recall. It is to fix a latency budget, then find the highest recall each candidate structure reaches within that budget on your data, and choose between them on that comparison plus memory and operational cost. Doing this requires measuring recall, which requires ground truth, which is why you compute exact neighbours for a sample of queries with a flat index and keep that set as a fixture. Recall at k against that fixture is then a number you can watch in a test suite, and it will move when someone changes a parameter or when the corpus grows.

Quantisation sits orthogonally to all of this. Compressing each vector — scalar quantisation, or product quantisation splitting the vector into sub-spaces with their own codebooks — cuts memory substantially and costs some accuracy, and it composes with either structure. It is the lever to reach for when memory rather than latency is the binding constraint, and its accuracy loss can partly be bought back by rescoring the shortlist against the full-precision vectors.

Where borrowed benchmark numbers mislead

Published comparisons are run on public datasets with query sets drawn from those datasets, and the recall an index achieves depends on the geometry of the data and the distribution of the queries far more than the leaderboard framing suggests. Embeddings of your corpus may be clustered much more tightly than the benchmark's, which makes cell boundaries in IVF more damaging. Your queries may be short and vague where the benchmark's were well-formed questions. If your corpus has many near-duplicate chunks — very common when overlap is used during chunking — then the top k neighbours are nearly tied, and a small distance error reorders them, so measured recall drops without the answer quality changing at all.

The correction is cheap and almost nobody does it: build the fixture from real queries against your own vectors, and treat any external number as evidence about which structures are worth testing rather than as a prediction of what you will get. There is a second, sharper reason to do it. Recall at k measured on the index is not the metric you care about; end-answer quality is. It is entirely possible to lose measurable recall and lose nothing at the answer, because the passages you dropped were duplicates of ones you kept, and it is equally possible for a small recall loss to be concentrated on exactly the hard queries that matter. Only your own gold set distinguishes those two cases.

Choose the structure on memory, build cost and update behaviour, because the recall each one reaches within your latency budget is something you set — and the only number worth trusting is the one measured against exact neighbours on your own queries.