hardDesignScenarioMidSeniorStaff
You must ingest all of English Wikipedia every day, and the source does not want to be hammered. What runs, how often, and what do you do about the run that fails at hour six?
Do not re-download Wikipedia daily. Use incremental ingestion: load the published dump once, follow the recent-changes feed, and fetch only the latest revision of each changed page in small, polite batches. Checkpointing each batch behind a revision-guarded write lets a run that fails at hour six resume.
6 min readUpdated 2026-10-09
Answer
Short answer
You do not download Wikipedia every day. Incremental ingestion means loading the published database dump once and then following the recent-changes feed. You fetch only the latest revision of each changed page, in small serial batches that respect the API's rules. Every batch is checkpointed and every write is guarded by revision ID, so the run that dies at hour six resumes from its last batch instead of starting again.
Was this answer helpful?
Likely follow-ups
- The change-stream consumer was down for longer than the stream's retention window. How do you recover without a full reload?
- A newly published dump disagrees with your table on twelve thousand pages. Which side do you trust, and what do you check first?
- The source starts answering with lag errors at hour four. What does your fetcher do, and what does the day's output look like?
- Downstream users want the table as of midnight UTC, not as of whenever the run finished. What changes?
Related questions
- Design a platform that runs agent tasks for hours and pauses for a human to approve risky steps.hardAlso on checkpointing and idempotency6 min
- Design a notification system that sends 50 million messages a day across email, SMS and push.hardAlso on idempotency and rate-limiting5 min
- Rate-limit an LLM gateway where one request can cost a hundred times another.hardAlso on rate-limiting5 min
- Your rate limiter has to work across twenty gateway instances. What does Redis do, and what breaks first?hardAlso on rate-limiting5 min