Problem Statement
The live Reddit crawler has correctness failures despite the new single-credential and conservative pacing controls. Successful crawls emit summary-generation errors; generic subreddit failures are marked complete and skipped; 404 cleanup uses substring matching; duplicate inactive crawler definitions remain in the deployed monolith; and Reddit/MongoDB secrets are embedded in source.
Solution
Make the crawler use one explicit orchestration path with correct timestamp handling, durable failed-job resume, exact and atomic unavailable-list cleanup, protected runtime configuration, and deterministic regression seams. Preserve one active credential, 10-second request pacing, one-minute job pacing, one global queue, durable state, and non-overlap.
Acceptance criteria
- Successful crawls produce non-empty summaries and CSV exports.
- A generic subreddit failure is persisted as failed and is retried/resumed from that subreddit; it is never recorded as complete.
- 404 cleanup removes only the exact normalized entry and uses an atomic replacement.
- There is one active crawler implementation and one scheduler entry point.
- Reddit and MongoDB secrets are loaded from protected runtime configuration and startup fails closed when required values are missing.
- Deterministic tests cover success, 404, 403, 429, generic failure, resume, timestamp formatting, exact cleanup, atomic replacement, and configuration validation.
- Live verification confirms supervisor health, crawler state, one-minute job pacing, request pacing, public endpoints, and no new auth failures.
Testing seam
Use a single crawler-pass seam with fake Reddit responses, a temporary collection file, and an in-memory state adapter; use a live smoke check after deployment.
Out of scope
Email alerts, a second credential, throughput increases, public API/schema changes, and replacing the current PRAW/MongoDB/Supervisor/Flask stack.
Local PRD: docs/plans/2026-07-28-reddit-crawler-correctness-prd.md
This follows the existing safety work tracked in #77.
Problem Statement
The live Reddit crawler has correctness failures despite the new single-credential and conservative pacing controls. Successful crawls emit summary-generation errors; generic subreddit failures are marked complete and skipped; 404 cleanup uses substring matching; duplicate inactive crawler definitions remain in the deployed monolith; and Reddit/MongoDB secrets are embedded in source.
Solution
Make the crawler use one explicit orchestration path with correct timestamp handling, durable failed-job resume, exact and atomic unavailable-list cleanup, protected runtime configuration, and deterministic regression seams. Preserve one active credential, 10-second request pacing, one-minute job pacing, one global queue, durable state, and non-overlap.
Acceptance criteria
Testing seam
Use a single crawler-pass seam with fake Reddit responses, a temporary collection file, and an in-memory state adapter; use a live smoke check after deployment.
Out of scope
Email alerts, a second credential, throughput increases, public API/schema changes, and replacing the current PRAW/MongoDB/Supervisor/Flask stack.
Local PRD:
docs/plans/2026-07-28-reddit-crawler-correctness-prd.mdThis follows the existing safety work tracked in #77.