feat: add pypi ecosystem to blast (CM-1358)

[ulemons]

Summary

Adds PyPI as a supported blast-radius ecosystem, mirroring the existing npm/go/maven/cargo/nuget/rubygems pipeline (intel → dependents → reachability → report).

Changes

• Add comparePep440 to osv/versionCompare.ts — hand-written PEP 440 comparator (epoch, release tuple, pre/post/dev segments, local version ordering); PyPI is not semver so this is new logic, not a clone of an existing comparator.
• Add toBarePypiName / toPypiNormalizedName to packageIdentifier.ts (PEP 503 name normalization).
• Add findPackageIdByPurl to services/libs/data-access-layer/src/packages/osv.ts (protected file, needs code-owner approval). Needed because packages.name for pypi rows can drift from the canonical PEP 503 spelling (writer disagreement between the deps.dev and pypi ingestion paths), while packages.purl is always normalized — so the intel stage looks packages up by purl instead of by name for this ecosystem only.
• Add stages/pypi/pypiConstraint.ts — PEP 440 specifier-set matching (~=, ===, ==/!= with .* wildcard, <=/>=/</>), over-inclusive by design on unparseable input, matching the contract of every other ecosystem's constraint matcher.
• Add clients/pypiSource.ts — downloads and extracts a package's source (sdist preferred, wheel fallback) by reading the URL from pypi.org's per-version JSON API rather than constructing it (the real filename/hash can't be derived).
• Add stages/pypi/{intelPyPi,dependentsPyPi,dependentsScanPyPi,reachabilityConfig}.ts and agent/pypiPrompts.ts, and wire pypi into stages/ecosystems.ts's ECOSYSTEMS registry, blast-radius/ecosystemSupport.ts, and the public API's SUPPORTED_BLAST_RADIUS_ECOSYSTEMS zod enum.
• Range resolution for PyPI follows the Maven template rather than Cargo's: PyPI's OSV entries are ECOSYSTEM-typed ranges (not SEMVER-typed), even though PyPI — like Cargo — is a deps.dev EDGE ecosystem with a resolved dependent version available, which dependents-scan prefers as ground truth over the declared specifier.
• Scope: blast-radius only. PyPI advisories are not yet added to osv/schedule.ts's ingestion allowlist, so pypi analyses currently resolve OSV data live and carry a null advisory_id until a follow-up PR.
• Extend dispatch.test.ts and ecosystemSupport.test.ts with pypi routing/registration cases.

Type of change

• Bug fix
• New feature
• Refactor / cleanup
• Performance improvement
• Chore / dependency update
• Documentation

JIRA ticket

1358

[Copilot]

Pull request overview

Adds PyPI support to the blast-radius analysis pipeline.

Changes:

• Implements PEP 440 comparison and dependency constraints.
• Adds PyPI source extraction, analysis stages, and prompts.
• Registers PyPI in worker and public API routing.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
services/libs/data-access-layer/src/packages/osv.ts	Adds package lookup by purl.
services/apps/packages_worker/src/pypi/types.ts	Extends PyPI response types.
services/apps/packages_worker/src/osv/versionCompare.ts	Adds PEP 440 comparison.
services/apps/packages_worker/src/osv/__tests__/versionCompare.test.ts	Tests PyPI version ordering.
services/apps/packages_worker/src/blast-radius/stages/pypi/reachabilityConfig.ts	Configures PyPI reachability.
services/apps/packages_worker/src/blast-radius/stages/pypi/pypiConstraint.ts	Matches PyPI dependency constraints.
services/apps/packages_worker/src/blast-radius/stages/pypi/intelPyPi.ts	Implements PyPI intelligence stage.
services/apps/packages_worker/src/blast-radius/stages/pypi/dependentsScanPyPi.ts	Scans reverse dependencies.
services/apps/packages_worker/src/blast-radius/stages/pypi/dependentsPyPi.ts	Persists dependent candidates.
services/apps/packages_worker/src/blast-radius/stages/pypi/__tests__/pypiConstraint.test.ts	Tests constraint matching.
services/apps/packages_worker/src/blast-radius/stages/ecosystems.ts	Registers PyPI stages.
services/apps/packages_worker/src/blast-radius/stages/__tests__/dispatch.test.ts	Tests PyPI dispatch.
services/apps/packages_worker/src/blast-radius/packageIdentifier.ts	Adds PyPI name normalization.
services/apps/packages_worker/src/blast-radius/ecosystemSupport.ts	Marks PyPI as supported.
services/apps/packages_worker/src/blast-radius/clients/pypiSource.ts	Downloads and extracts distributions.
services/apps/packages_worker/src/blast-radius/clients/__tests__/pypiSource.test.ts	Tests source extraction.
services/apps/packages_worker/src/blast-radius/agent/pypiPrompts.ts	Adds Python analysis prompts.
services/apps/packages_worker/src/blast-radius/__tests__/ecosystemSupport.test.ts	Tests ecosystem registration.
backend/src/api/public/v1/packages/blastRadius.ts	Enables PyPI in API validation.

Suppressed comments (1)

services/apps/packages_worker/src/blast-radius/agent/pypiPrompts.ts:50

• Remove this section-header comment; the project convention explicitly forbids section-header comments (CLAUDE.md:82).

// ---------- STAGE 3: REACHABILITY ----------

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated 3 comments.

Suppressed comments (5)

services/apps/packages_worker/src/blast-radius/stages/pypi/pypiConstraint.ts:141

• An unparseable resolved version makes every compareVersion call return null, but null === 0 is false, so the dependency is incorrectly marked excluded. This contradicts the matcher’s over-inclusive contract and can drop a real vulnerable dependent; preserve unparseable-included when comparison is impossible.

  if (resolvedVersion) {
    const matched = vulnerableVersions.some((v) => compareVersion('pypi', resolvedVersion, v) === 0)
    return matched ? 'matched' : 'excluded'

services/apps/packages_worker/src/blast-radius/stages/pypi/pypiConstraint.ts:101

• PEP 440 specifier matching is not the same as total version ordering when a candidate has a local suffix. For example, ==1.0 and <=1.0 must match 1.0+local, while >1.0 must not; this comparator returns 1, producing false exclusions/inclusions. Apply operator-specific public/local-version rules rather than using the total-order result directly.

  const c = compareVersion('pypi', version, clause.version)
  if (c === null) return true // unparseable bound — over-inclusive

  switch (clause.op) {
    case '==':

services/apps/packages_worker/src/blast-radius/stages/pypi/pypiConstraint.ts:24

• A valid compatible specifier with an epoch, such as ~=1!2.0, is treated as unparseable because this regex captures only the 1 before !. That over-includes unrelated epochs instead of enforcing >=1!2.0, ==1!2.*; parse and retain the optional epoch in the wildcard prefix.

This issue also appears in the following locations of the same file:

• line 97
• line 139

  const releaseMatch = version.match(/^[0-9]+(?:\.[0-9]+)*/)
  if (!releaseMatch) return null
  const segments = releaseMatch[0].split('.')
  if (segments.length < 2) return null

  const prefix = segments.slice(0, -1).join('.')

services/libs/common/src/agentAuth.ts:59

• This shared auth utility is outside the stated PyPI blast-radius scope and has no production caller; agent/runner.ts still uses its existing inline API-key logic, while only the new test references this function. Move this refactor to a focused PR or explicitly wire and describe it so the shared-library change is reviewable in context.

export function resolveAgentAuth(opts: ResolveAgentAuthOptions = {}): AgentAuth {

services/libs/data-access-layer/src/osspckgs/packages.ts:10

• The PR description says this feature adds a PyPI-specific purl lookup in packages/osv.ts, but the implementation instead changes this existing shared lookup’s return type, affecting the unrelated Maven importer. Either keep the new lookup scoped as described or update the PR scope and all callers without unsafe coercion.

): Promise<Map<string, string>> {

[cursor]

PR Summary

Medium Risk
Large new pipeline surface (external PyPI fetches, archive extraction, PEP 440/version logic, LLM stages) plus a DAL change (findPackageIdByPurl); blast-radius-only scope limits broader product impact.

Overview
Adds PyPI as a blast-radius ecosystem end-to-end: public API validation and job submission accept ecosystem: 'pypi', and the worker routes intel, dependents, and reachability through new stages/pypi/* wired into ECOSYSTEMS.

The PyPI path mirrors other ecosystems but uses PEP 440 throughout—new comparePep440 in versionCompare, pypiConstraint for specifier matching (over-inclusive on bad input), and ECOSYSTEM-typed OSV ranges rather than semver. Intel resolves packages via findPackageIdByPurl (PEP 503–normalized packages.purl) because packages.name can drift; it fetches versions from pypi.org and downloads source with pypiSource (sdist preferred, wheel fallback, size limits). Dependents scan package_dependencies with resolved version as ground truth. Python-specific agent prompts and reachability config drive import/reexport analysis on extracted source.

Also introduces shared resolveAgentAuth in @crowd/common (Bedrock vs API key vs CLI, scoped Bedrock env vars) with tests, extends resolveAdvisoryAndPackageIds to accept string package IDs, and adds PyPiUrlInfo on PyPI JSON types for per-version distribution URLs.

^{Reviewed by Cursor Bugbot for commit bb3e7c4. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.

Suppressed comments (4)

services/apps/packages_worker/src/blast-radius/clients/pypiSource.ts:55

• This chooses any legacy sdist, but the download path only distinguishes .zip from tar and cannot decode legacy .tar.bz2/.tar.xz files. For affected older releases, an unsupported sdist is selected even when a usable wheel exists, so source preparation fails instead of falling back. Restrict selection to formats this client can extract (or add the missing decompressors).

  const sdist = urls.find((u) => u.packagetype === 'sdist')

services/apps/packages_worker/src/blast-radius/stages/pypi/pypiConstraint.ts:24

• Valid compatible-release specifiers can include an epoch, for example ~=1!1.4 (equivalent to >=1!1.4, ==1!1.*). Because this regex requires the release tuple at character zero, every such dependency is reported as unparseable-included and bypasses range filtering. Preserve the optional epoch when constructing the wildcard prefix.

  const releaseMatch = version.match(/^[0-9]+(?:\.[0-9]+)*/)
  if (!releaseMatch) return null
  const segments = releaseMatch[0].split('.')
  if (segments.length < 2) return null

  const prefix = segments.slice(0, -1).join('.')

services/apps/packages_worker/src/blast-radius/stages/pypi/pypiConstraint.ts:97

• PEP 440 ignores a candidate's local label when the specifier has no local label. Using the full comparator here makes ==1.0 reject vulnerable 1.0+linux, makes <=1.0 reject it, and makes >1.0 accept it, so valid dependents can be misclassified. Strip the candidate local label for public specifiers and add regression cases for equality and ordered operators.

  const c = compareVersion('pypi', version, clause.version)

services/libs/common/src/index.ts:24

• This exports a new shared authentication API, but repository-wide usage is limited to the new test; the production blast-radius runner still implements API-key/CLI auth inline and never calls resolveAgentAuth. The PR is scoped to PyPI support, while services/libs/common is protected and affects every service. Remove this export, utility, and test from this PR, or integrate the migration as a separately scoped change.

export * from './agentAuth'