[Crawler] Add global request pacing and rate-limit backoff

[longdang95]

Parent

#77 — Make the Reddit crawler conservative and rate-limited

What to build

Add a single conservative request governor to the active crawl path. After the credential gate is in place, process one global subreddit job per 60 seconds across every collection and enforce at least 10 seconds between every outbound Reddit HTTP request, including authentication, pagination, and retries. Treat HTTP 429 as a global condition: pause the crawl for five minutes, then use 10-, 20-, and 30-minute backoff intervals when rate limiting continues. Keep the existing top-100 and hot-10 data depth unchanged.

The slice must be verifiable with a controllable-clock coordinator harness and a production log/endpoint smoke test that demonstrates pacing without exposing credentials.

Acceptance criteria

• A single global coordinator limits subreddit-job starts to one per 60 seconds across all collections.
• Every outbound Reddit HTTP request is separated from the previous one by at least 10 seconds, including token, pagination, and retry requests.
• HTTP 429 pauses all crawling for five minutes, then applies 10-, 20-, and 30-minute backoff intervals for continued rate limiting.
• The current top-100 and hot-10 collection depth remains unchanged.
• The coordinator harness verifies request spacing, global job spacing, no concurrent workers, and the complete 429 backoff sequence using a fake clock.
• Production verification shows the active crawler making paced requests and the public ranking endpoint continuing to serve existing data.

Blocked by

• #78 — Disable invalid credential and fail closed on authentication