Loading...
Inverted indexes, relevance scoring, and the ingestion-to-query pipeline of a search product.
3 questions in System Design.
Polite web crawler system design starts with per-host queues, crawl-delay enforcement and URL normalisation. The crawler partitions by host so one worker owns the next-allowed fetch time, uses a durable seen set with a Bloom filter in front, and keeps freshness work separate from first-time discovery.
An inverted index maps terms to posting lists, built by an analysis chain that must run identically over documents and queries, ranked by BM25 plus business signals, and fed by a change stream of denormalised documents with reindexing behind an alias treated as routine.
Offline you need graded relevance judgements and a rank-aware metric to compare orderings; online you need an experiment sensitive enough to detect the effect, with click data corrected for the fact that position drives clicks on its own, and a guardrail on zero-result and abandonment rates.