[PRD] Make the Reddit crawler conservative and rate-limited

[longdang95]

Reddit Crawler Safety and Rate Control PRD

Problem Statement

The live RedditList/SubRanking crawler is configured with two Reddit API credential sets. A read-only crawl of r/AskReddit proved that the first credential can authenticate and retrieve current posts, while the second returns HTTP 401 from Reddit's token endpoint.

The crawler currently starts one worker per configured credential over the same subreddit lists, waits only one second after each subreddit, performs multiple Reddit requests per subreddit, ignores the stored last-processed cursor when starting a full rescan, and allows scheduled passes to overlap. When credentials or Reddit access fail, the process can remain marked as running while producing stale data and repeated error logs.

Solution

Make the crawler conservative and fail closed:

• Keep only the working credential active. Retain the invalid second credential only in a protected rollback backup until it is securely rotated or removed.
• Use one active crawler worker and one global subreddit-job budget across all collections: start at most one subreddit job per minute.
• Enforce a minimum 10-second delay between every outbound Reddit HTTP request, including authentication checks and retries.
• Validate the active credential before each crawl pass. If validation fails, remain idle and log the reason.
• On HTTP 401 from the active credential, stop the current pass and remain idle until manually repaired and restarted.
• On HTTP 429, pause the entire crawl and apply escalating backoff: five minutes initially, then 10, 20, and 30 minutes for continued rate limiting.
• Resume from the last successfully processed subreddit after restarts.
• Prevent overlapping full passes; begin the next scheduled pass only after the previous one completes and the configured cooldown has elapsed.
• Preserve the current data depth of top 100 and hot 10 posts per subreddit.
• Log status and failures for operators; do not add email alerts in this change.

User Stories

1. As a site maintainer, I want only a verified Reddit credential to be used for live crawling, so that a known-invalid credential cannot generate unnecessary failures.
2. As a site maintainer, I want the invalid credential preserved outside the active configuration, so that rollback remains possible without exposing or deleting the secret prematurely.
3. As a site maintainer, I want one crawler worker to process the subreddit collection, so that multiple credentials cannot crawl the same subreddit list concurrently.
4. As a site maintainer, I want a global limit of one subreddit job per minute, so that the aggregate crawl rate is predictable across every collection.
5. As a site maintainer, I want at least 10 seconds between all Reddit HTTP requests, so that pagination, metadata calls, authentication, and retries cannot create request bursts.
6. As a site maintainer, I want a credential health check before each crawl pass, so that the crawler does not begin a long run when Reddit authentication is already unavailable.
7. As a site maintainer, I want an active-credential HTTP 401 to stop the current pass, so that the system does not repeatedly send requests with an invalid credential.
8. As a site maintainer, I want the crawler to remain idle after an authentication stop, so that it waits for a deliberate repair instead of repeatedly rediscovering the same failure.
9. As a site maintainer, I want HTTP 429 responses to pause the entire crawl, so that one rate-limited request causes the system to reduce pressure globally.
10. As a site maintainer, I want repeated HTTP 429 responses to increase the pause duration, so that the crawler gives Reddit enough time to recover from rate limiting.
11. As a site maintainer, I want the crawler to resume from its last successful subreddit, so that a restart does not repeat hours of already completed work.
12. As a site maintainer, I want the crawler to prevent overlapping full passes, so that a slow pass cannot compete with the next scheduled pass.
13. As a site maintainer, I want the current top-post and hot-post sample sizes preserved, so that rate control does not silently change ranking quality.
14. As a site maintainer, I want per-subreddit access errors to be isolated from global authentication failures, so that one unavailable community does not unnecessarily stop healthy crawling.
15. As a site maintainer, I want 404 handling to continue marking unavailable communities according to the existing behavior, so that banned or removed communities do not block the pass.
16. As a site maintainer, I want crawler logs to record active state, last successful subreddit, next retry time, and failure category, so that I can diagnose stale data without inspecting secrets.
17. As a site maintainer, I want no email alert integration in this change, so that the first safety release remains operationally small and log-driven.
18. As a visitor, I want rankings to continue serving the last known data while a crawl is paused, so that a Reddit outage does not take down the public website.
19. As a visitor, I want new ranking data to appear after a healthy crawl resumes, so that the safety controls protect freshness without permanently disabling updates.

Implementation Decisions

• The live crawler’s credential registry will contain one active client. The failed second client will be excluded from the runtime client list and retained only in a protected rollback artifact until replacement or secure rotation is approved.
• Credential values will not be copied into the PRD, logs, alerts, or test fixtures. Runtime configuration should be moved toward secret-backed injection as part of the implementation boundary, without changing the working credential during this task.
• The crawler coordinator will own one global queue and one active worker. A subreddit job means processing one subreddit once; a crawl pass means traversing a complete collection set.
• The coordinator will enforce a global 60-second minimum between the starts of completed subreddit jobs, regardless of collection.
• A shared request limiter will enforce a 10-second minimum between all outbound Reddit HTTP requests made by the crawler, including PRAW authentication, pagination, and retries.
• The current data collection depth will remain unchanged: top 100 and hot 10 posts, plus the existing metadata and statistics.
• A preflight credential check will run before each crawl pass. A failed preflight leaves the crawler idle and records a categorized failure.
• HTTP 401 from the active credential is a global authentication failure: stop the current pass, leave the process idle, and require manual repair/restart. No email notification is part of this PRD.
• HTTP 429 is a global rate-limit failure: pause the entire coordinator for five minutes, then use 10-, 20-, and 30-minute backoff intervals when the condition persists. The request limiter remains active after the pause.
• Per-subreddit 404 behavior remains isolated to that subreddit. Other community-level failures should be logged and skipped without deleting the entire source collection.
• The last-successful-subreddit checkpoint becomes an actual resume cursor. A restarted pass must continue from the next eligible subreddit rather than restarting the collection from its first line.
• The scheduler will use a non-overlap guard. A new pass cannot begin while the previous pass, its backoff, or its checkpoint finalization is still active.
• Operators will be able to determine the crawler state from structured logs, including ready, running, backing_off, blocked_auth, and idle transitions, without exposing credential material.

Testing Decisions

• The highest-value seam is one end-to-end crawl-coordinator harness around the existing crawl entry point, using a fake Reddit requestor, controllable clock, fake persistence layer, and deterministic subreddit fixtures. This seam should verify observable scheduling, request timing, state transitions, persistence, and resume behavior together.
• Tests must measure external behavior rather than implementation details: number and order of Reddit requests, minimum intervals, active-worker count, saved records, checkpoint position, and resulting crawler state.
• The harness must prove that a successful active credential can complete a subreddit job while the invalid credential is never constructed as an active worker.
• The harness must prove that the global job interval applies across collection boundaries and cannot be bypassed by a second worker.
• The harness must prove that every fake outbound Reddit request is separated by at least 10 seconds, including token, pagination, and retry requests.
• The harness must prove that preflight authentication failure prevents the first subreddit request and leaves the crawler idle.
• The harness must prove that a 401 stops the current pass, preserves the last successful checkpoint, and does not continue issuing requests.
• The harness must prove the 429 pause and escalating backoff sequence without waiting in real time by using a fake clock.
• The harness must prove that a restart resumes after the last successful subreddit and does not duplicate completed work.
• The harness must prove that a second scheduled pass cannot overlap the first, even when the first pass takes longer than the scheduler interval.
• The harness must prove that one 404 or other community-level failure does not stop healthy subreddits from being processed.
• The current repository has no established automated test suite for the live Flask crawler, so this change should introduce the smallest focused integration harness at the coordinator seam rather than broad unit coverage of PRAW internals.
• Production verification should include a read-only r/AskReddit crawl with the active credential, log inspection for the configured intervals and state transitions, and a public endpoint smoke test to confirm the website continues serving existing data.

Out of Scope

• Adding email, SMS, Slack, or other external alert delivery.
• Replacing or rotating the working Reddit credential.
• Permanently deleting the failed credential before a protected rollback copy and replacement process are agreed.
• Changing the ranking formulas, subreddit eligibility rules, ban-removal semantics, or MongoDB data schema.
• Reducing the top 100 or hot 10 data depth.
• Rebuilding the public frontend or changing public API response shapes.
• Moving the crawler into the local Next.js repository; the live crawler remains a separate Flask/Docker application.
• Adding automated credential rotation or Reddit account management.

Further Notes

The live crawler is the Flask/Docker application on the origin server, not the Next.js source tree in this repository. Before implementation, create a timestamped server backup of the active crawler configuration and verify it. After implementation, verify crawler status, request pacing, resume behavior, https://redditli.st/, https://subranking.com/, and the existing /load_data?type=largest&category=sfw&limit=1 endpoint.

The current live evidence supports the scope: the first credential authenticated and retrieved three newest r/AskReddit posts; the second returned HTTP 401 at Reddit's token endpoint. The process was still saving some subreddit records while producing many 401 failures, which is why the safe behavior must distinguish a global credential failure from a per-subreddit access failure.