You are crawling a billion pages politely. What stops you hammering one host, and how do you avoid fetching the same page twice?

Short answer

Design the crawler around host-level politeness: normalise URLs, enqueue by host, let one owner enforce the next fetch time, and use a durable seen set so retries do not explode duplicate work.

Keep web crawler explicit in the answer because that is the concept the interviewer is actually trying to test. A good web crawler explanation names the trade-off, the failure mode, and the evidence you would use before choosing. Use web crawler once more at the decision point so the answer reads as judgement rather than a detached example.

One derivation decides the architecture

State the rate before drawing anything. A billion pages in thirty days is a billion divided by 2.59 million seconds, which is about 386 pages a second sustained. Not peak, sustained, for a month.

Now apply the politeness rule. Say you allow one request per second to any single host. Then 386 pages a second requires at least 386 distinct hosts with work available at every instant, and realistically several times that, because hosts run out of queued URLs at different rates.

That number settles the first half. It says the frontier cannot be one queue, because a single queue ordered by priority will happily put a thousand URLs from the same popular domain at the front and then either violate politeness or stall on it. The queue has to be split by host.

Politeness as a partition key, not a lock

The instinct is a rate limiter: check a shared counter for the host before each fetch. It works, and it puts a coordinated read and write on the path of every one of 386 requests a second when it is not needed.

Partition the frontier by host. Every URL for example.com lands in the same shard, and that shard hands the host's queue to exactly one worker at a time. The worker holds two pieces of local state per host: the earliest time it may fetch again, and the parsed robots rules. Politeness becomes an invariant a single process maintains locally, with no lock, no shared counter, and no possibility of two workers racing to fetch from the same host.

flowchart LR
  S[Seen set with bloom front end] --> F[Frontier sharded by host]
  F --> Q[Per host queue with next allowed time]
  Q --> W[Fetcher worker]
  W --> R[Robots cache]
  W --> P[Parser and URL normaliser]
  P --> S

The loop is the interesting part: extracted links go back through the normaliser and the seen-set before re-entering the frontier, never straight into a queue. Everything that stops the crawler doing redundant work happens on that return path.

The rules have a specification. RFC 9309 standardised the exclusion protocol, covering where the file lives and how paths are matched. Crawl-delay is not in it - it is a widely-honoured convention, and a candidate who cites it as required is citing something that does not exist.

Two details in the robots handling separate a careful answer. The file is fetched per host, cached with its own lifetime, and that fetch counts against the host's politeness budget. And you have to decide what an unreadable robots file means: the protocol's guidance is that a 4xx may be read as no restrictions, while a server error should be treated as a complete disallow, because a site whose server is failing is the last one you should be pressing.

A fixed delay is also worse than one derived from the host's own behaviour. One request a second is timid against a host answering in 40 milliseconds and an attack against one taking four seconds, so scale the delay to a multiple of the observed response time.

The duplicate you avoid by never asking

The second half is usually answered with a data structure. The data structure is the smaller part of it.

Most duplicate fetching does not come from forgetting a URL. One page has many URLs. Differing scheme case, an explicit default port, a trailing slash, a fragment identifier that the server never sees, query parameters in a different order, tracking parameters that change nothing about the response. Each variation is a distinct string and the same bytes.

So normalisation comes first. It is a series of decisions rather than a function you can name: lowercase the scheme and host, drop the default port, resolve dot segments in the path, remove the fragment, sort the remaining query parameters, and strip parameters known to be irrelevant. The last is a judgement per site and it is where the gains are. A parameter carrying a session identifier generates unbounded distinct URLs for one page, and a crawler with no policy about it spends its whole budget there.

Say out loud that normalisation is lossy in both directions. Strip a parameter that mattered and you never fetch a real page; keep one that did not and you fetch the same page a hundred times.

Sizing the seen-set from the premise

A billion URLs at eighty bytes each is eighty gigabytes of strings. That does not sit in one machine's memory. Store a 64-bit hash of each normalised URL instead and it is eight gigabytes before index overhead, which is affordable but not free.

A Bloom filter at ten bits per key holds a billion entries in 1.25 gigabytes with a false-positive rate near one per cent. Now read that one per cent properly. A false positive says "already seen" about a URL that was never fetched, so ten million pages are silently dropped, with no error and no record of which ones. For a crawl whose entire purpose is coverage, that is not a tolerable approximation.

The fix is architectural. Use the filter as a front end that answers only the question it can answer without error. A negative is certain, so a definite "not seen" enqueues immediately with no further lookup, while a "maybe seen" falls through to an authoritative partitioned key-value store consulted for the small fraction of URLs that reach it. You keep the memory saving on the common path and you never lose a page to a probabilistic answer.

That store is partitioned by the hash of the URL, so the check happens on the shard that owns it. Note this is a different partitioning from the frontier's, and conflating the two is a common slip: the seen-set wants uniform spread, the frontier wants host affinity.

Content-level duplication catches a different case. Two URLs you had no reason to suspect return identical bodies, so hash the normalised body and keep a second set of digests. Near-duplicates need a similarity fingerprint with a threshold instead, and that threshold is a policy decision: set it loosely and you discard distinct pages that share a template.

The ceiling nobody wants to say out loud

Priority and politeness pull against each other. Pricing the conflict is what a senior answer adds.

Suppose one site holds a million pages you want. At one request per second, a million pages is 11.6 days of continuous crawling of that one host, during which its content is changing. No amount of worker capacity moves that number, because the constraint is a promise you made to the host rather than a resource you own.

So the conclusions are blunt. Per-host rate is a hard ceiling on coverage speed, covering a large site means negotiating a higher rate or accepting staleness, and the priority scheme has to decide which pages within a host to fetch first rather than how fast. Sitemaps and change-frequency estimates are how you spend a fixed budget well.

Bandwidth is worth checking against the same premise, because it usually is not the problem. At eighty kilobytes of HTML per page, 386 pages a second is about 31 megabytes a second, or a quarter of a gigabit. The billion pages themselves are eighty terabytes of stored HTML. The scarce resource is politeness, not the network.

Where crawls die

Three failures account for most of it, and none is about throughput.

Traps first. Calendars generating a next month for ever, faceted search producing a URL per filter combination, and session identifiers in the path all present an infinite site. Depth limits, a per-host page budget and a cap on URL length are crude, and they are what keep a crawl finite.

Then DNS, which is a request per hostname and often the slowest part of a fetch, so it needs its own cache. It also exposes the flaw in per-host politeness: hundreds of hostnames may resolve to one shared server, and being polite to each independently is being rude to the machine. Rate-limit by resolved address as well as by host.

Last, the lease. A URL handed to a worker that then dies must not be lost and must not be fetched twice, so URLs are leased with a visibility timeout rather than deleted on dequeue. A duplicate fetch costs one request; a lost URL costs a page.

Politeness is per host, so partitioning the frontier by host converts the hardest constraint in the system into a local invariant, and the seen-set is the second line of defence behind normalisation rather than the first.

© 2026 Preptima. Originally published at preptima.com.