[Crawler] Make crawl passes resumable and non-overlapping

[longdang95]

Parent

#77 — Make the Reddit crawler conservative and rate-limited

What to build

Make the slow crawler resumable and non-overlapping. Use one global queue across all subreddit collections, persist and consume the last-successful-subreddit cursor after restarts, and prevent a second crawl pass from starting while the current pass or its backoff is active. After a complete pass, wait 96 hours before starting another. During pauses or outages, keep serving the last known ranking data; after recovery, resume and publish fresh data normally.

The slice must be verifiable through restart and scheduler simulations plus production smoke checks for the crawler state and public ranking endpoints.

Acceptance criteria

• A restart resumes from the next subreddit after the last successfully processed job instead of restarting the collection from the beginning.
• The queue traverses all collections under one global job budget, with no collection-specific bypass of the one-per-minute limit.
• A scheduled pass cannot overlap an active pass, an active backoff, or checkpoint finalization.
• The next scheduled pass begins only after the previous pass completes and the 96-hour cooldown has elapsed.
• A paused or idle crawler leaves existing public ranking data available.
• The coordinator harness verifies interruption, restart/resume, overlapping scheduler triggers, cooldown timing, and persistence of the last successful cursor with a fake clock.
• Production verification confirms crawler state transitions and that both public site endpoints and the ranking data endpoint remain healthy during a controlled pause.

Blocked by

• #79 — Add global request pacing and rate-limit backoff