You are crawling a billion pages politely. What stops you hammering one host, and how do you avoid fetching the same page twice?
Polite web crawler system design starts with per-host queues, crawl-delay enforcement and URL normalisation. The crawler partitions by host so one worker owns the next-allowed fetch time, uses a durable seen set with a Bloom filter in front, and keeps freshness work separate from first-time discovery.
Answer
Short answer
Design the crawler around host-level politeness: normalise URLs, enqueue by host, let one owner enforce the next fetch time, and use a durable seen set so retries do not explode duplicate work.
Keep web crawler explicit in the answer because that is the concept the interviewer is actually trying to test. A good web crawler explanation names the trade-off, the failure mode, and the evidence you would use before choosing. Use web crawler once more at the decision point so the answer reads as judgement rather than a detached example.
Was this answer helpful?
Likely follow-ups
- A single site holds a million pages you want. How long does your politeness policy take to cover it, and what would you negotiate?
- Two hundred hostnames resolve to the same IP address behind one shared server. Which of your limits is now wrong?
- The same article appears on forty syndicating sites with different boilerplate. What decides whether you treat those as one page?
- A worker dies holding a thousand leased URLs. What happens to them, and how do you avoid crawling them twice?
Related questions
- Design a notification system that sends 50 million messages a day across email, SMS and push.hardSame kind of round: design5 min
- A file changes on one laptop and must appear on three others, one of which is offline. What travels over the network, and what happens when two of them edited it?hardSame kind of round: design6 min
- A two-hour film must start playing within two seconds on a phone on 4G. What has to exist before the play button works?hardSame kind of round: case-study6 min
- How do you turn what the business tells you into a domain model that holds up once the exceptions arrive?hardSame kind of round: design7 min