Skip to content

test(vendored): real-production e2e for scan --mode vendored across every package manager - #169

Merged
Mikola Lysenko (mikolalysenko) merged 1 commit into
mainfrom
test/e2e-vendored-production
Aug 13, 2026
Merged

test(vendored): real-production e2e for scan --mode vendored across every package manager#169
Mikola Lysenko (mikolalysenko) merged 1 commit into
mainfrom
test/e2e-vendored-production

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

What

Adds crates/socket-patch-cli/tests/e2e_vendored_production.rs — the vendored-mode
counterpart to e2e_hosted_production.rs. The synthetic e2e_vendor_*_build.rs
capstones prove the vendoring mechanism with a hand-staged blob and
vendor --offline; they never contact production. This suite drives
scan --mode vendored against the real Socket production service and real
upstream registries, with no mocking anywhere, on the anonymous free public
proxy
(no token; SOCKET_NO_CONFIG=true, every SOCKET_* scrubbed with hostile
seeds so a dropped scrub reddens the suite).

Per package manager it proves the full loop: install pristine (asserted pristine —
anti-vacuity) → scan --mode vendored materializes the real free-tier patch
into .socket/vendor/<eco>/<uuid>/ and rewires the lock/manifest → cold reinstall
from ONLY the committable files
(empty caches, offline) → the installed bytes are
the vendored (patched) bytes, not the pristine registry bytes → idempotent
re-run (already_vendored, byte-stable lock) → vendor --revert byte-restores.
It proves byte delivery, not CVE efficacy — matching how the capstones assert.

#[ignore]-gated (stays out of the test/e2e jobs), soft-skips absent toolchains
unless SOCKET_PATCH_VENDORED_E2E_STRICT=1, cache-isolated via
common/cache_env.rs. Reuses the hosted suite's pinned patch catalog. Companion doc:
docs/testing/vendored-production-e2e.md.

Coverage

Package manager Fixture Cold offline delivery install Proof
npm minimist@1.2.2 npm ci full byte delivery
pnpm minimist@1.2.2 pnpm install --frozen-lockfile --offline delivery via pnpm-11 workaround (gap below)
yarn classic minimist@1.2.2 yarn install --frozen-lockfile --offline full byte delivery
yarn berry (node-modules) minimist@1.2.2 yarn install --immutable --check-cache full byte delivery
bun (text lockfile) minimist@1.2.2 bun install --frozen-lockfile full byte delivery
pip (requirements.txt) urllib3@1.26.18 pip install --no-index -r full byte delivery
uv (uv.lock) urllib3@1.26.18 uv sync --frozen --offline full byte delivery
cargo ([patch.crates-io]) traitobject@0.1.1 cargo fetch --offline --locked resolves from vendored path, zero downloads + byte check
bundler activestorage@7.0.2.2 ❌ CLI gap (below)
go zero-patch assertion (no free golang patches)
deno negative assertion (unsupported)
maven / nuget / composer canary (no free production patches)

Plus preflight_required_patches_are_published (catalog canary against
patches-api.socket.dev, mirrors the hosted preflight).

cargo uses cargo fetch --offline rather than cargo build: the production
traitobject patch injects a compile_error! unless the allow-unmaintained feature
is set (the patch's intent), so build fails by design — that's patch content, not a
vendoring defect. The leg proves the whole graph resolves from committable files with
zero registry downloads and byte-checks the vendored directory.

How to run

# everything, soft-skipping absent toolchains
cargo test -p socket-patch-cli --test e2e_vendored_production -- --ignored --test-threads=1

# CI: every "toolchain missing" soft-skip becomes a hard failure
SOCKET_PATCH_VENDORED_E2E_STRICT=1 \
  cargo test -p socket-patch-cli --test e2e_vendored_production -- --ignored --test-threads=1

Knobs: SOCKET_PATCH_VENDORED_E2E_GEM_STRICT=1 (promote the gem gap to a hard
failure), SOCKET_PATCH_VENDORED_E2E_CANARY_STRICT=1 (fail when maven/nuget/composer
gain a free patch).

Notes

cargo test -p socket-patch-cli --test e2e_vendored_production --no-run compiles
cleanly (the primary gate — the suite is #[ignore]). Ran the full --ignored set
locally against production, all 13 tests green (npm 11.6.2, pnpm 11.18.0, yarn
4.6.0 / corepack yarn@1.22.22, bun 1.3.14, uv, python3/pip, cargo, ruby+bundle all
present; composer + deno absent but their legs need no toolchain):

  • Ran green — npm, pnpm, yarn classic, yarn berry, bun, pip, uv, cargo,
    preflight, golang (zero-patch), deno (unsupported), canary.
  • Soft-skip — none locally; legs soft-skip only when their toolchain is absent.
  • CLI bugs surfaced (marked xfail, not fixed — tests-only PR):
    • pnpm ≥ 11 ignores package.json pnpm.overrides (moved to
      pnpm-workspace.yaml), so a frozen install refuses
      ERR_PNPM_LOCKFILE_CONFIG_MISMATCH even though the vendored tarball + lock are
      correct (the lockfile passes pnpm's supply-chain policy). The leg reproduces the
      documented workaround (mirror the override into pnpm-workspace.yaml) to prove
      the artifact IS installable, and fails loudly on any other error. Fix belongs in
      the CLI: emit pnpm-workspace.yaml overrides on pnpm ≥ 11.
    • gem vendoring refuses the platform-qualified purl
      (pkg:gem/activestorage@7.0.2.2?platform=ruby) with platform_gem_unsupported
      (status: partial_failure); download succeeds. The leg asserts redirect+download
      and tolerates the vendor failure, failing loudly on any other error and
      auto-upgrading to a full bundle install delivery proof once the CLI vendors
      platform gems.

🤖 Generated with Claude Code


Note

Low Risk
Tests-only change with no CLI or runtime behavior modifications; live production tests are ignored by default and isolated from normal CI.

Overview
Adds e2e_vendored_production.rs, a production counterpart to the hosted suite and the synthetic e2e_vendor_*_build capstones: scan --mode vendored against real patches-api.socket.dev and upstream registries on the free public proxy (scrubbed SOCKET_*, SOCKET_NO_CONFIG=true).

Each leg runs install → pristine-byte check → vendoring → cold offline reinstall from committable files only (patched bytes, not registry) → idempotent re-scan → vendor --revert. Coverage includes npm (npm/pnpm/yarn classic/berry/bun), PyPI (pip/uv), Cargo, plus preflight patch catalog checks, golang/deno guards, and a maven/nuget/composer canary.

docs/testing/vendored-production-e2e.md documents patches, PM matrix, known CLI gaps (pnpm ≥11 pnpm-workspace.yaml overrides, gem platform_gem_unsupported), and strict env knobs. Tests are #[ignore] with optional SOCKET_PATCH_VENDORED_E2E_STRICT hard-fail on soft-skips.

Reviewed by Cursor Bugbot for commit 4a2f836. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

Bugbot Autofix is ON. A cloud agent has been kicked off to fix the reported issues.

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 4a2f836. Configure here.

Comment thread crates/socket-patch-cli/tests/e2e_vendored_production.rs
Comment thread crates/socket-patch-cli/tests/e2e_vendored_production.rs
… every package manager

Adds `crates/socket-patch-cli/tests/e2e_vendored_production.rs`, the vendored-mode
counterpart to `e2e_hosted_production.rs`. It drives `scan --mode vendored`
against the REAL Socket production service and REAL registries on the anonymous
free public proxy (no token, `SOCKET_NO_CONFIG=true`), and proves the full
vendored loop per package manager: install pristine -> vendor the real free-tier
patch into `.socket/vendor/<eco>/<uuid>/` -> cold reinstall from ONLY the
committable files (empty caches, offline) -> the installed bytes are the vendored
(patched) bytes, not the pristine registry bytes -> idempotent re-run -> revert
byte-restores. Byte delivery, not CVE efficacy (matches the capstones).

`#[ignore]`-gated; soft-skips absent toolchains unless
`SOCKET_PATCH_VENDORED_E2E_STRICT=1`. Companion doc in docs/testing/.

Legs: npm, pnpm, yarn classic, yarn berry (node-modules), bun, pip
(requirements.txt), uv (uv.lock), cargo; preflight patch canary; golang
zero-patch, deno unsupported, and maven/nuget/composer canary negatives.

Two real CLI gaps surfaced and marked xfail (tests-only PR, not fixed here):
- pnpm >= 11 ignores package.json `pnpm.overrides` (moved to
  pnpm-workspace.yaml), so a frozen install refuses ERR_PNPM_LOCKFILE_CONFIG_MISMATCH
  even though the vendored artifact is correct; the leg proves delivery via the
  documented workaround and fails loudly on any other error.
- gem vendoring refuses the platform-qualified purl
  (`?platform=ruby`) with `platform_gem_unsupported`; the leg asserts
  redirect+download and tolerates the vendor failure, auto-retiring once fixed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mikolalysenko
Mikola Lysenko (mikolalysenko) merged commit 0770374 into main Aug 13, 2026
62 checks passed
@mikolalysenko
Mikola Lysenko (mikolalysenko) deleted the test/e2e-vendored-production branch August 13, 2026 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants