fix(vendor): gem wiring reconstruction, dir-artifact file inventory, per-section lock remotes

[Mikola Lysenko (mikolalysenko)]

STACKED on PR #175 (fix/gem-bundler-audit) — base branch is fix/gem-bundler-audit, not main.

Closes three verified vendored-integrity holes (all P2), each fixed at its seam and pinned by a test that fails without it. Every bundler-behavior assumption below was validated against a real bundler 4.0.15 in a scratch project before being baked in.

1. repair reconstruction synthesized empty wiring → revert stranded the pair edit

repair's no-ledger reconstruction (repair_vendor.rs) created ledger entries with wiring: []. A later vendor --revert of such a gem entry deleted the artifact dir, reported success with zero warnings, and left the Gemfile path: + lock PATH section pointing at the deleted dir — the next bundle install hard-fails.

• vendor::gem::reconstruct_gem_wiring (new, core): rebuilds full revert-capable wiring records from the backend's own recognizable pair edit — the exact-pin path: line / managed fence in the Gemfile, the grammar-strict PATH section, and the name (= version)! DEPENDENCIES pin (the same recognizers fix(gem): bundler vendored+hosted audit — 13 confirmed bugs fixed, tests hardened #175's re-vendor-new-uuid path trusts). Grammar-strict and fail-closed: any shape vendor does not write yields Err, never a guessed record. For an exact-pin project the reconstructed ledger entry is byte-identical to the original (pinned e2e assertion on state.json).
• Two documented degradations, both inherent to a lost ledger:
  • a pre-vendor range constraint (~> 3.1) is unrecoverable; the reconstructed original is the canonical exact pin of the locked version — a consistent, installable pair resolving to the same version (pinned).
  • a pre-vendor CHECKSUMS sha256= token is unrecoverable offline; no checksum record is emitted, revert leaves bundler's bare path-form entry, and repair surfaces vendor_checksum_unrecoverable. Empirically verified on bundler 4.0.15: a plain bundle install refills the line byte-identically; frozen installs fail with a self-explanatory empty/missing CHECKSUMS entry message until then (both bare and deleted behave identically, so nothing is deleted).
• repair_vendor.rs reconstruction seam (reconstruct_entry_wiring): routes per ecosystem. gem gets real wiring; every other ecosystem keeps empty wiring — npm reconstruction populates nothing either (checked, same hole), because its pre-vendor originals are registry integrity strings (sha512-… SRI etc.) no offline source can reproduce — and the gap now surfaces loudly as a vendor_wiring_unknown warning instead of silently.
• revert_gem empty-wiring guard: reverting a gem entry with no wiring records now fails loudly (vendor_wiring_unknown + the manual cleanup steps, files and artifact untouched) instead of silently deleting the artifact.

Out of owned scope (recorded, not changed): the npm-family (and pypi/cargo/…) revert of an empty-wiring reconstructed entry still "succeeds" silently, leaving the lock's file:.socket/vendor/… resolution pointing at the deleted artifact (pinned as current behavior by the pre-existing e2e test 7). The shared guard belongs in dispatch_revert_one (crates/socket-patch-cli/src/commands/vendor.rs) — outside this PR's owned files. Follow-up recommended; the reconstruction-time vendor_wiring_unknown warning added here at least fronts the hazard for every ecosystem.

2. Dir-shaped artifacts verified only their patched members

check_vendored_artifact's whole-file drift cross-check only covered file-shaped artifacts (.tgz/.whl/…). A vendored gem directory was verified purely by record.files afterHashes, so tampered/drifted unpatched files (or the stub gemspec) were blessed Healthy by repair and attested by VEX.

• Schema (state.rs): VendorArtifact.fileInventory — relative forward-slashed path → plain sha256 hex, BTreeMap (sorted on the wire), camelCase, skip_serializing_if none. Documented where the artifact schema is documented. Backward-tolerant: entries without it keep member-only verification.
• Recorded at vendor time by vendor_gem (stub gemspec included); repair's ledger restore / rebuild paths fill it via fill_artifact_fingerprint.
• Verified whole-tree inside verify_vendored_patch_record's dir arm (so both repair and VEX get it): missing, extra, and modified files all fail with the new vendor_inventory_mismatch tag → Corrupt → repair rebuilds; VEX refuses to attest while tampered. Planted symlinks/FIFOs fail the walk (vendor_artifact_unreadable) — never hashed through.
• Legacy entries (no inventory): behavior unchanged, plus a repair-time warning naming the gap (vendor_inventory_missing).
• Note: cargo/golang/composer dir-shaped backends don't record inventories yet (outside owned files) — their healthy entries now surface the same vendor_inventory_missing warning during repair. Follow-up: record inventories in those backends too.

3. Multi-source Gemfile.locks resolved every spec against the FIRST remote

inventory_gemfile_lock captured the first remote: across all GEM sections and built every spec's download URL from it, so specs under a second GEM source (private gem server) got a wrong URL — a 404 at best, a private gem name leaked to the public registry at worst.

• Empirics (bundler 4.0.15): a Gemfile source … do block produces one GEM section per source, one remote: each; multiple global sources hard-error (This Gemfile contains multiple global sources…), so a single section with several remote: lines can only be a legacy bundler 1.x multisource lock.
• Each spec now resolves against its own section's remote. A section with several remotes is genuinely ambiguous per spec → its specs stay discovery-only (resolved: None; the fetch layer then refuses with "no download URL") — fail-closed, never guessed.
• gem_remote_base (ledger recovery) had the same first-remote assumption, and worse: a vendored gem's spec block has moved into the PATH section, so its origin section is unrecoverable. Recovery (gem_remotes) now proceeds only when the lock's GEM sources agree on a single distinct remote; multiple distinct remotes refuse with "multiple GEM sources … ambiguous". No lock / no remote keeps the pre-existing rubygems.org default.

Which test pins which change (all RED on the base commit, verified by run)

Pin (fails without the fix)	Change
e2e repair_reconstructs_gem_wiring_and_revert_byte_restores — reconstructed state.json byte-equal to the original; revert byte-restores Gemfile+lock, artifact gone (base: "wiring": [], revert exit 0 with stranded pair)	gem wiring reconstruction
e2e revert_of_empty_wiring_gem_entry_fails_loudly — exit 1, vendor_wiring_unknown named, artifact + files untouched (base: exit 0, artifact deleted)	revert guard
e2e repair_reconstruction_flags_unrecoverable_gem_checksum — vendor_checksum_unrecoverable surfaced; revert leaves exactly the bare CHECKSUMS line (base: no event)	checksum degradation contract
e2e repair_gem_dir_tamper_matrix_and_vex_refusal — tampered/deleted/extra file each → VEX refusal (vendor_inventory_mismatch) + repair rebuild; heals byte-exact (base: fileInventory null, everything attested)	inventory record + verify
e2e repair_warns_on_legacy_gem_entry_without_inventory — no rebuild + vendor_inventory_missing (base: fails at fixture, no inventory exists)	backward tolerance + gap warning
unit gem::tests::reconstruction_reproduces_vendor_wiring_for_pinned_declaration / _for_managed_block — reconstruction == vendor's own records, revert byte-restores	reconstruction equivalence oracle
unit gem::tests::reconstruction_degrades_range_constraint_to_exact_pin, _preserves_trailing_options, _flags_unrecoverable_checksum, _refuses_foreign_or_mismatched_wiring	degradation + fail-closed contracts
unit gem::tests::revert_refuses_empty_wiring_entry (RED on base: revert succeeded)	revert guard
unit gem::tests::vendor_records_dir_file_inventory — hand-pinned sha256s	inventory recording
unit verify::tests::dir_inventory_detects_unpatched_tamper_missing_and_extra_files, dir_inventory_refuses_planted_symlink	inventory verification
unit state::tests::file_inventory_round_trips_sorted_camel_case	schema
unit lock_inventory::tests::gemfile_lock_multi_source_resolves_each_spec_against_its_own_remote (RED on base: second spec inherited the first remote)	per-section remotes
unit lock_inventory::tests::gemfile_lock_legacy_multi_remote_section_is_discovery_only (RED on base: guessed first remote)	multi-remote fail-closed
unit lock_inventory::recover_tests::gem_recovery_refuses_ambiguous_multi_source_lock (RED on base: guessed)	recovery ambiguity refusal

Verification

• core: 2474 passed / 0 failed (all targets); cli lib: 350/0
• repair_vendor_e2e: 20/20 (15 pre-existing npm rows untouched + 5 new gem rows)
• adjacent suites green: e2e_vex_vendor 8, in_process_vendor 27, repair_invariants 20, repair_vendor_flavors_e2e 28, scan_vendor_e2e 18, setup_contract_gaps 10, e2e_gem 8, in_process_gem_apply 7, in_process_gem_multi_platform 4, in_process_scan 24, ecosystem_dispatch_e2e 18, in_process_rollback_all_ecosystems 8
• docker_e2e_gem 2/2, docker_e2e_vendor_gem 1/1 (fresh image)
• e2e_vendor_gem_build capstone green under the real host bundler 4.0.15 (vendor → VEX → fresh-checkout frozen install → revert byte-restore)
• cargo fmt --check clean; cargo clippy --all-targets warning-clean on both crates
• RED verification: base sources checked out with the new tests kept → all 5 gem e2e rows + 4 behavior-level unit pins fail with exactly the defects described above

Reviewer notes

• The VendorArtifact schema addition forced a mechanical file_inventory: None, line in every literal constructor (~34 sites across backends/tests, outside the owned file list but compiler-forced); each is a single line, nothing functional.
• One-line re-export addition in vendor/mod.rs (artifact_is_file_shaped, compute_dir_inventory) — verify is pub(crate), the CLI needs the symbols.
• revert_gem's guard and vendor_gem's inventory recording go slightly beyond "export recognizers only" for gem.rs; both are additive, directly serve the mission items, and are covered by the equivalence/guard tests. The alternative placements (dispatch_revert_one / persist_vendor_entry) are outside the owned files.

🤖 Generated with Claude Code

---

Note

Medium Risk
Touches vendor ledger schema, repair/revert paths, and lockfile URL resolution—high test coverage but behavior changes affect gem repair and multi-source locks.

Overview
Closes three vendored-integrity gaps: ledger repair that could not restore gem wiring (revert stranded path: edits), dir-shaped artifacts that only checked patched files, and Gemfile.lock inventory that used the first GEM remote: for every spec.

Repair + gem revert. repair now routes per-ecosystem wiring reconstruction: gems call reconstruct_gem_wiring to rebuild revert-capable Gemfile / Gemfile.lock records from vendor-shaped live files (fail-closed on foreign shapes). Other ecosystems still get empty wiring with an explicit vendor_wiring_unknown warning. revert_gem refuses entries with empty wiring instead of deleting the artifact and leaving broken path: wiring.

Dir artifact integrity. Ledger entries add optional fileInventory (whole-tree path → sha256). vendor_gem records it; verify/repair/VEX compare the full directory (tamper on unpatched files → vendor_inventory_mismatch). Legacy entries without inventory keep member-only checks plus vendor_inventory_missing warnings.

Gem lock remotes. inventory_gemfile_lock resolves each spec against its own GEM section’s remote:; ambiguous multi-remote sections stay discovery-only. Ledger recovery refuses guessed downloads when multiple distinct GEM sources exist.

^{Reviewed by Cursor Bugbot for commit 0f24244. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{Bugbot Autofix is ON. A cloud agent has been kicked off to fix the reported issue.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0f24244. Configure here.}

[Mikola Lysenko (mikolalysenko)]

Review polish pushed as e5d975e — all 5 findings addressed, none skipped.

Major — service-provenance repair loop (repair_vendor.rs): confirmed and fixed at the post-verify seam. A member-verified dir-shaped rebuild of a NON-reconstructed entry that fails only the recorded-inventory comparison now recomputes the inventory from the verified rebuild, persists it, and names the source switch (vendor_inventory_refreshed + the vendor re-run remedy) instead of deleting the rebuild and re-failing every subsequent repair. All other post-verify failures (member hash mismatch, tampered ledger sha, unreadable tree) stay fail-closed and are re-checked after the refresh. Pinned by e2e repair_refreshes_stale_inventory_from_service_provenance (RED on the previous tip: exit 1 vendor_artifact_rebuild_failed, artifact deleted; the test also proves the loop is dead and the fixture's simulated service tree is healthy pre-corruption).

Minor — vendor_inventory_missing noise: gated to gem, the one backend that records inventories. Pinned inside repair_warns_on_legacy_gem_entry_without_inventory: a synthesized healthy inventory-less cargo dir entry now produces zero events (RED before: it warned every run), with an anti-vacuity phase proving the cargo entry is genuinely health-checked (its deleted artifact fails the same run).

Minor — existing empty-wiring entries never healed: pass 1 now backfills full revert-capable wiring via reconstruct_gem_wiring for healthy empty-wiring gem entries and persists it (warning on reconstruction failure, dry-run previewed). Pinned by e2e repair_backfills_wiring_for_empty_wiring_gem_entry — the backfilled state.json is byte-identical to the original ledger for the exact-pin fixture and the previously-refusing revert byte-restores the pair. Residual: an empty-wiring entry whose patch was also dropped from the manifest is skipped before the health check (reconcile owns it), so it is not backfilled — same population as before, now one repair run away from healable once the record returns.

Minor — reconstruction doc wrinkles: both documented on reconstruct_gem_wiring (now three degradations): trailing-comment loss on the pre-vendor gem line, and the deliberately-conservative vendor_checksum_unrecoverable on an already-bare pre-vendor CHECKSUMS entry (kept conservative rather than suppressed — a bare entry is indistinguishable from a lost token offline; empirically real, bundler 4.0.15 writes bare CHECKSUMS entries for file-sourced gems).

Minor — gem_remotes scheme filter: confirmed empirically against real bundler 4.0.15 (a Gemfile mixing an https source with a source "file://…" do block locks two GEM sections, one with a file:/// remote). gem_remotes now collects remotes scheme-agnostically before the ambiguity refusal, and a lone non-http survivor refuses instead of defaulting to rubygems.org (which would itself leak a private repo's gem name). Pinned by gem_recovery_counts_non_http_remotes_as_ambiguity (RED without the fix on both arms).

Verification: core 2474 green; cli lib 350; repair_vendor_e2e 22/22 (all 3 new pins RED-verified against the previous tip); adjacent suites green (e2e_vex_vendor 8, in_process_vendor 27, repair_invariants 20, repair_vendor_flavors_e2e 28, scan_vendor_e2e 18, setup_contract_gaps 10, e2e_gem 8, gem apply/multi-platform/dispatch/rollback/scan suites); docker_e2e_gem 2/2, docker_e2e_vendor_gem 1/1; gem_vendor_fresh_checkout_bundle_install_and_revert capstone green under host bundler 4.0.15; fmt + clippy clean on both crates.

🤖 Generated with Claude Code

[Mikola Lysenko (mikolalysenko)]

Bugbot finding (no-ledger restore TOFU hole) — CONFIRMED and fixed in bd4fae8.

Verified against the code first: pass 2's Healthy branch ran check_vendored_artifact on a synthesized entry (no fileInventory/sha256, so only patched members were verified), wired_vendor_integrity only reads npm-family locks (returns None for gems), and fill_artifact_fingerprint then hashed the live dir into the persisted entry. A tampered unpatched file became the canonical tree that later repairs enforce and VEX attests. Reproduced empirically: on the pre-fix branch, a tampered padlock.gemspec (require_paths = ["lib", "exfil"]) survived repair and its hash landed in the restored ledger, with the second repair reporting nothing to rebuild.

Fix (mirrors the PR's own vendor_inventory_refreshed philosophy — fingerprints only from verified rebuilds):

• Healthy-but-unanchored reconstructions (dir-shaped gem/cargo trees; file artifacts no npm-family lock records an integrity for) are queued as soft rebuild candidates instead of being blessed.
• The ledger entry is restored fingerprint-less first, so the restore survives any rebuild failure and the reconstructed gem wiring originals carry forward through persist_vendor_entry.
• The untrusted live tree is deleted only after patch sources + a pristine source are both secured; the backend's wired hot path then materialises a fresh copy (pristine + patch via dispatch_vendor_one), and the canonical fileInventory derives from that member-verified rebuild.
• If no trustworthy rebuild is possible (not installed, no recoverable registry fragment — reconstructed gem wiring never carries the CHECKSUMS sha256 —, --offline, fetch failure), the entry stays without a fingerprint plus a vendor_inventory_unverified warning; later repairs keep warning vendor_inventory_missing instead of enforcing a tampered tree.
• Anchored npm-family artifacts (wired lock integrity, verified intact) restore exactly as before.

Tests (both verified RED on the pre-fix code): repair_no_ledger_restore_never_canonizes_tampered_gem_tree (tamper heals via rebuild; restored state.json byte-identical to the pre-tamper original) and repair_no_ledger_restore_without_pristine_source_stays_unverified (fingerprint-less restore + warning + legacy-warn on later runs).

Runs: repair_vendor_e2e 24/24 (incl. G1's byte-identical + revert oracle), repair_vendor_flavors_e2e 28/28, core vendor::gem 56/56, repair_invariants + remove/repair lifecycle + in_process_vendor + scan_vendor_e2e + e2e_vex_vendor 87/87, docker_e2e_vendor_gem 1/1 (real bundler). fmt + clippy clean.

🤖 Generated with Claude Code

[Mikola Lysenko (mikolalysenko)]

Rebased fix/gem-vendor-integrity onto the freshly-rebased base fix/gem-bundler-audit (old base tip d063cb1 → new base tip 99f9cd0, which now carries main's 11 new commits: --vendor-source in vendor_flow, hosted JSON envelope changes, maven/nuget GA, etc.). Transplanted this PR's 3 own commits with git rebase --onto; new tip is 914004c.

Conflicts resolved (1):

• crates/socket-patch-core/src/vendor/lock_inventory.rs (recover_tests module) — purely additive test collision: the new base brought main's pypi-recovery tests (recover_pypi_pdm_static_urls_recovers_pure_wheel, recover_pypi_urlless_locks_report_no_fetchable_url) at the same insertion point as this PR's gem_recovery_refuses_ambiguous_multi_source_lock. Kept both sides verbatim; no behavior change or judgment call. git range-diff shows commits 2 and 3 replayed identically and commit 1 differing only in that test-context hunk.

Verification (all green):

• repair_vendor_e2e 24 passed, repair_vendor_flavors_e2e 28 passed, in_process_vendor 27 passed, scan_vendor_e2e 18 passed (cli, --features docker-e2e,setup-e2e)
• cargo test -p socket-patch-core vendor::gem 56 passed
• docker_e2e_vendor_gem 1 passed (fresh socket-patch-test-gem:latest)
• cargo fmt --check + clippy clean on every file this PR touches. Note: pre-existing (not from this rebase) fmt drift exists in 6 files inherited from main/base (scan/hosted.rs, scan/mod.rs, update_notifier.rs, e2e_vendored_production.rs, ecosystem_dispatch_e2e.rs, update_notifier_e2e.rs) plus 3 clippy doc-indent warnings in e2e_vendored_production.rs (from test(vendored): real-production e2e for scan --mode vendored across every package manager #169). Left untouched to keep this PR's diff scoped; there is no CI fmt gate.

🤖 Generated with Claude Code

[Mikola Lysenko (mikolalysenko)]

Rebased onto main — new tip bf55fa1 (was 914004c).

What happened: the CONFLICTING state was the stale stacked base — this PR sat on fix/gem-bundler-audit (old tip 99f9cd0), which squash-merged as #175. Transplanting only the PR's own 3 commits via rebase --onto produced zero textual conflicts, including against #177 (in_process_vendor/gem hosted tests), #193/#194 (cargo redirect + vendored fail-closed), #190 (berry lock carry-forward), #195 (composer), and #176 (gem/pypi/maven variant mismatch policy). Main moved twice mid-rebase (#195, #176); re-rebased onto each — final base is e2472c8. No commit content changed.

Verification on the final base:

• socket-patch-core vendor::gem: 56 passed
• repair_vendor_e2e 30 / repair_vendor_flavors_e2e 24 / in_process_vendor 28 / scan_vendor_e2e 18 — all passed
• docker_e2e_vendor_gem: 2 passed (real containers)
• cargo clippy --workspace --all-features -- -D warnings: clean
• rustfmt: every PR-touched file clean. (6 pre-existing fmt hunks live in main's vendor/cargo.rs/cargo_config.rs from fix(redirect): make the cargo hosted redirect transactional and fail-closed #193/fix(vendor): fail closed in the cargo vendored flow #194, surfaced only through mod-recursion; not this PR's code, and CI has no fmt gate.)

🤖 Generated with Claude Code

[Mikola Lysenko (mikolalysenko)]

Heads-up for reviewers: the red hosted-e2e check is unrelated to this PR — production unpublished the free pkg:gem/activestorage@7.0.2.2 patch the live suite is pinned to (preflight + canary + gem leg all fail on any branch, including main; all other ecosystem legs pass). This PR touches vendor repair/verify/lock-inventory only. A catalog-replacement PR is being prepared separately.

Otherwise green and MERGEABLE after the rebase onto current main. Awaiting an approving review to merge.

🤖 Generated with Claude Code

[Mikola Lysenko (mikolalysenko)]

Windows CI update: the test (windows-latest) failure was a confirmed flake in update_notifier_e2e::dead_endpoint_never_fails_the_command (code this PR does not touch) — the rerun passed with no changes. Only the known prod-catalog hosted-e2e red remains; the replacement-pin PR is in progress.

🤖 Generated with Claude Code

[Mikola Lysenko (mikolalysenko)]

The hosted-e2e fix is up: #199 (production publishes zero free gem patches — pin retired, gem canary-ized; live suite 15/0 on that branch). Once #199 merges, a rerun here goes fully green.

🤖 Generated with Claude Code