ci(release): promote v0 only after PyPI publish, verify the published action

[uipreliga]

Rewritten after a second review round (40345b6). The previous description
documented behavior that review changed — most importantly, preflight referenced a
step output that does not exist, which would have made it red on 100% of runs. This
body describes the branch as it now stands; the per-finding walk-through is in
the review-response comment.

Problem

Consumers pin uses: UiPath/coder_eval@v0. The composite action installs
coder-eval==<action.yml's version: default>, a pin the release commit bumps. In the OLD
release.yml the release job pushed main + the version tag, moved v0, then built the
wheel; PyPI publishing happens in a SEPARATE publish-pypi job (needs: release, behind
a pypi deployment environment for OIDC Trusted Publishing). So v0 moved to an
action.yml pinning version X before X existed on PyPI.

Two reachable paths:

1. publish-pypi fails, or waits on the environment gate.
2. The tag move sat before uv build, so a build failure stranded the pin with PyPI
   never involved.

Either way every @v0 consumer's uv tool install coder-eval==X 404s, and nothing
detected it. release.yml's own comment named this seam; the continue-on-error on the
Release step plus its "Flag missing GitHub Release" annotation were a workaround for it.

Separately, nothing verified the published composite: action-dogfood in
pr-checks.yml runs uses: ./ with version: local, which proves a PR's code works but
never touches v0 or PyPI.

Part 1 — prevention (release.yml)

The v0 tag move and the GitHub Release creation moved out of release into a new
promote job gated on needs: [release, publish-pypi]. Nothing a consumer can resolve
happens until the wheel is published. Marketplace listings are cut from a published
Release, so creating one also announces a version — hence both moved, not just the tag.

Supporting changes that make the recovery story actually hold:

• Monotonic, not merely idempotent. promote refuses to promote anything but the
  newest vX.Y.Z tag. Force-push is only self-idempotent and says nothing about
  ordering: GitHub keeps "Re-run failed jobs" live for 30 days, so replaying an older
  release's promote would walk v0 backwards and silently downgrade every consumer.
• skip-existing: true on publish-pypi. Without it, an upload that succeeds but
  whose step then fails (lost response, job timeout) gets 400 File already exists
  forever — so promote could never run for a version that is published, which is the
  stranded state from the other direction.
• …and an artifact-identity assertion to pay for it. skip-existing makes twine treat
  that 400 as success without comparing content, so on its own a green publish stops
  proving the wheel this run built is the one PyPI serves — and nothing downstream
  re-established it (promote moves v0 on job success alone; preflight checks
  reachability, not identity). A new step compares urls[].digests.sha256 from PyPI's
  version JSON against sha256sum dist/*. Mismatch is fatal — it must stop promote.
  An unreadable index is a warning, because the JSON API can lag an upload by seconds
  and a transient must not redden a publish that succeeded.
• publish-pypi carries no if:. It used to gate on
  if: needs.release.outputs.version != '', which was both dead (the release job already
  exit 1s on an empty version) and dangerous: on a partial "Re-run failed jobs" attempt a
  lost output resolved the job to SKIPPED-green, which — since promote needs it —
  also skipped the promotion, for a fully green run that published no wheel and never
  moved v0. The implicit success() on needs: release is the real gate; emptiness is
  asserted in-job, so a lost output is red.
• Prereleases discriminated on github.ref, the same signal "Determine release mode"
  already uses, rather than on a needs output — same hazard as above. Emptiness is
  enforced inside promote, so a lost output is red.
• Release creation normalizes an existing draft/prerelease
  (gh release edit --draft=false --prerelease=false --latest) instead of treating mere
  existence as done, which could report success while announcing nothing.
• Least privilege. Both create-github-app-token mints declare
  permission-contents: write (omitting permission-* yields a token holding every
  permission of the installation — and this is the app with the main-branch ruleset
  bypass); promote declares permissions: contents: read, dropping the workflow-level
  packages: write that only the GHCR steps need.
• The old continue-on-error + "Flag missing GitHub Release" scaffolding is removed.
  It existed only because a failure there would have skipped publish-pypi and stranded
  the tag; promote is strictly downstream, so a failure can no longer skip anything
  upstream, and the job is re-runnable — it can fail loudly instead.

Residuals, accepted and documented in-file:

• vX.Y.Z and main are still pushed by the release job, so if publish-pypi fails
  they briefly reference an unpublished version. Narrower than the @v0 window by design
  (@v0 is the documented pin; @vX.Y.Z/@main are opt-in) and cleared by re-running
  publish-pypi. Closing it entirely means publishing to PyPI before pushing any git ref,
  which requires carrying the bumped commit between jobs as an artifact — not worth the
  new failure modes.
• The GHCR agent image is deliberately not covered by the promote ordering: it is
  still pushed (and :latest still moved) inside the release job, best-effort. It is an
  internal convenience rather than a ref a stranger's pipeline resolves, and it must be
  built in the job holding the bumped pyproject. Stated in the promote header rather
  than left as an inconsistency.

Part 2 — detection (verify-published-action.yml)

For drift a release cannot cause: a PyPI yank, the pinned setup-uv SHA, runner-image
changes, the @anthropic-ai/claude-code npm package, model deprecation, or the listing
being renamed/delisted.

Tier 1 preflight — free, deterministic, gates tier 2. The hard gate is the
consumer contract: the version action.yml at the v0 tag pins must be installable
from PyPI, and that same pin is what the install/smoke step installs. Also asserts the pin
anchor is readable, that the major is still v0 (the uses: below can't be an
expression, so a 1.0.0 bump must fail loudly — and the error enumerates the doc surfaces
that hardcode the major, since CE026 checks the slug, not the major), and that the
Marketplace listing resolves.

Tag lag is classified, not failed:

v0 state	Verdict
At newest tag, pin matches	pass
At newest tag, pin ≠ that release	hard fail — release.yml's pin bump didn't take
Lags, newest version is on PyPI	hard fail — promote didn't run; re-run it
Lags, newest version absent from PyPI	warning — release incomplete, @v0 consumers healthy
Lags, PyPI gave no definitive answer	warning — inconclusive

Every HTTP probe in this tier — both PyPI ones and the Marketplace one — uses the same
transient split: only a definitive 4xx is a verdict; 000 / 403 / 429 / 5xx mean we
learned nothing. The hard gate stays red on an unproven pin, but it no longer says
"stranded pin, re-run publish-pypi" for a version PyPI already has.

Tier 2 e2e — cents. Consumes the action as a stranger would: uses: UiPath/coder_eval@v0, default version:, no repo checkout, task YAML written inline.
Doubles as a live proof that the documented Node + @anthropic-ai/claude-code
prerequisite steps still work. Skipped for branch-dispatched prereleases, which cannot
change the published artifact.

The gate is artifacts at the literal paths the workflow passes in with:, not the
step's exit code and not steps.run.outputs.* (the action step is continue-on-error,
and composite-output propagation through a failed step is undocumented). It requires
run.json, a JUnit report with at least as many <testcase> elements as task_results
rows (parse-alone let an empty report through), and non-zero tokens — plus assertions that
distinguish our breakage from the model's:

• Zero tokens branches on the error_category every run.json row already carries:
  agent_api_error / agent_rate_limit / agent_timeout / agent_crash warn as
  inconclusive; anything else (auth, billing, config, sandbox, or no category) is the hard
  "wiring is broken" error. A daily cron will eventually meet a transient, and sending the
  operator to audit credentials for an Anthropic outage is how a check earns being ignored.
• ERROR / BUILD_FAILED with a non-upstream category hard-fail rather than being
  tolerated as a model flake — those are exactly the harness failures this gate exists for.
• Output wiring must be exact when the step went green; when it went red, a missing
  output is ambiguous (runner behavior) and only warns, but a present-but-wrong one is
  not ambiguous and warns explicitly rather than falling through.
• The step must be green when every task reported SUCCESS, catching a regression in
  the action's own exit logic while still tolerating a model flake.
• The run dir uploads on always(), so the routinely-tolerated red step keeps the
  evidence that explains it.

Triggers on Release completion regardless of conclusion — a failed publish-pypi
makes the run's conclusion failure, so gating on success would skip the check exactly
when it matters. Plus a daily cron and workflow_dispatch.

Part 3 — guardrails, so this class cannot ship again

The first round of this PR shipped a reference to a step output that does not exist. It
was invisible to ruff, pyright, pytest and the CE runner, and actionlint models
steps.*.outputs as an open string map, so it passed clean there too. Two additions close
that:

• CE035 — workflow output-key parity (tests/lint/workflow_outputs.py, wired as
  tests/test_custom_lint.py::TestCE035WorkflowOutputParity). Resolves every
  ${{ steps.<id>.outputs.<key> }} and ${{ needs.<job>.outputs.<key> }} in
  .github/workflows/** + action.yml to a writer that actually produces it. Sound
  boundaries: third-party uses: are skipped (their metadata is not on disk), and a body
  whose writers are not statically readable is skipped rather than guessed at. Its negative
  test is the exact shape of the shipped bug.
• tests/test_verify_published_workflow.py (8 tests) binds the four couplings nothing
  asserted: the workflow_run: ["Release"] display-name link to release.yml's name:;
  Marketplace slug parity between the workflow's shell pipeline and the tested
  marketplace_slug() over a punctuation/whitespace table; all three
  # <-- kept in sync pin-anchor readers (both seds executed against the real
  action.yml); and the inline consumer task YAML loading through the real load_task.

The slug case was live, not hypothetical: the shipped tr ' ' '-' pipeline turned
Coder Eval (CI gate) into coder-eval-(ci-gate), which 404s, while marketplace_slug()
(and therefore the doc links CE026 pins) produced coder-eval-ci-gate. They agreed only
because action.yml's name: is the one input where both are the identity function.

Docs

CONTRIBUTING.md gains a § Releasing runbook — the three-job table, which jobs are
re-runnable, four named recovery flows, and a table of every annotation the nightly emits
with what it means and what to do. CLAUDE.md's tree line now names promote and the
verification workflow instead of describing the old single-job shape.

Verification

Every guard was exercised by reproducing the failure, not by reading:

• 4 parity breakages in a throwaway clone: stale major tag, stranded pin (amend step
  failed), detached # <-- kept in sync anchor, v1 bump that would rot the hardcoded
  @v0.
• Both lag-classification branches, incl. that the publish-pypi-failure case warns with
  the right diagnosis instead of hard-failing with the wrong one.
• 10 e2e-gate fixture paths against the extracted Python: happy path, upstream
  rate-limit tolerated, zero-tokens-no-category fatal, auth error fatal, harness error
  fatal, model flake tolerated, agent crash tolerated, all-SUCCESS-with-red-step fatal,
  empty JUnit fatal, no rows fatal.
• 5 digest-check fixture paths: digests match, foreign wheel pre-uploaded (fatal),
  filename absent (fatal), index unreachable (warning, publish stands), empty dist (fatal).
• The monotonicity refusal, both directions.
• CE035's negative test reproduces the shipped reference verbatim (a step echoing
  pin/newest read as outputs.version) and asserts the rule flags it, alongside the
  missing-step-id and undeclared-needs-output cases and the two skip boundaries.
• 000000 from || echo 000 confirmed empirically; gh release edit flags confirmed present.
• actionlint clean on both files; bash -n and ast.parse clean over every run: body
  and embedded Python block.
• make check clean, custom lint 175 passed, full suite 3894 passed.

make verify fails at pyright on 3 unresolved imports in codex_agent.py for the
optional [codex] extra, which isn't installed locally. Pre-existing and unrelated —
this PR contains no src/ changes.

Cannot be verified pre-merge

workflow_run and schedule only activate once the file is on main, and there is
deliberately no pull_request trigger — so this PR's own checks do not exercise the new
workflow at all. After merge, workflow_dispatch proves tier 1 immediately (free, no API
spend); CE035 means the failure that made that first dispatch mandatory can no longer be
the one that greets it.

Runner behaviors that remain assumptions, all now failing safe: whether needs.*.outputs
survive a partial re-run (both publish-pypi and promote fail loudly either way), and
whether composite outputs propagate through a failed step (the gate no longer depends on
it). One deliberate publish-pypi failure + re-run after merge would settle both.

Behavior changes worth a second opinion:

• If the pypi environment has required reviewers, promote now waits behind that
  approval before v0 moves. Correct, but the tag previously moved before the gate.
• permission-contents: write on the app-token mints first executes on a real release. A
  wrong scope key would 422 at mint time; contents is the correct GitHub App permission
  key, but it is worth watching on the next release.

Deliberately deferred

Booked in .claude/harness-candidates.md rather than silently dropped: extracting the two
oversized inline blocks into .github/scripts/ with fixture tests (CE040 — agreed in
principle, but it is a refactor of a workflow that cannot be exercised pre-merge, and
CE035 + the new tests close the specific classes); CE036 (ban the skipped-green job
gate) and CE037 (if: failure() in a job containing a continue-on-error step), both
shapes now hand-fixed twice; exercising the action's score gate in the failing direction
(needs a second paid agent run — belongs in action-dogfood, which already pays); and
extending CE026's REQUIRED_PREREQ_TOKENS anchor to the e2e job, now a third copy of the
prerequisite steps.

🤖 Generated with Claude Code

[github-actions]

Claude finished @uipreliga's task in 1m 40s —— View job

---

Code Review in Progress

Review Checklist:

• Read .github/code_review.md for review guidelines
• Read CLAUDE.md for project conventions
• Get full diff with git diff origin/main...HEAD
• Read .github/workflows/release.yml (full file)
• Read .github/workflows/verify-published-action.yml (full file)
• Analyze release workflow changes for correctness
• Review verification workflow design
• Check cross-file consistency
• Perform "what's missing" analysis
• Provide comprehensive review feedback

Starting code review now...

[akshaylive]

Nice work here — the release/promote redesign and the new verification tier are well thought through, and the review history shows a lot of care already went into hardening the failure modes. I double-checked the two blockers from the posted review directly against the checked-out branch, and both are real and still present:

1. steps.parity.outputs.version doesn't exist — preflight fails on every run

The parity step only emits pin, newest, and lagging (verify-published-action.yml:124-128):

{
  echo "pin=$PIN"
  echo "newest=$VERSION"
  echo "lagging=$LAGGING"
} >> "$GITHUB_OUTPUT"

But two later steps reference steps.parity.outputs.version, which resolves to empty:

• Line 202: TAG_REF: v${{ steps.parity.outputs.version }} → git show "v:action.yml" fails.
• Line 247: VERSION: ${{ steps.parity.outputs.version }} → uv tool install "coder-eval==" fails.

Since e2e depends on needs: preflight, this means preflight is red on every trigger and the paid e2e tier — the whole point of this PR — never runs. Easiest fix is probably swapping in steps.parity.outputs.pin at both sites, since the pin is the actual consumer contract this tier is meant to verify (and using newest instead would break under the legitimate lagging=true state).

2. publish-pypi's if: still has the skipped-green hazard that promote was just fixed to avoid

release.yml:321 still gates publish-pypi on:

if: needs.release.outputs.version != ''

promote's own comment right above it explains why this shape is risky — a lost output on a partial "Re-run failed jobs" resolves to skipped-green rather than failing loudly — and promote was switched to gate on github.ref instead. But publish-pypi is still on the old pattern, and since promote requires needs: [release, publish-pypi], a skipped publish-pypi silently skips promote too. That's the exact recovery scenario this whole PR is built around, so it'd be good to bring publish-pypi in line with promote's approach (or just drop the guard, since release's "Resolve published version" step at line 188 already hard-fails on an empty version, making this condition dead weight either way).

Everything else in the earlier review (rationale-comment drift, the PyPI probe's missing transient-code handling, GHCR image push not being gated behind publish-pypi, etc.) reads as solid non-blocking feedback for a follow-up pass. Happy to help with either fix if useful!

[uipreliga]

Thanks @akshaylive — both blockers were real, and both are fixed. Full walkthrough is in the comment above; the short version:

1. steps.parity.outputs.version. Fixed by keying each reader off a value that exists, rather than adding the missing output. The Marketplace step now uses a new major output — and your reasoning holds twice over there, because in the lagging=true state no Release was cut for the newest tag, so the live listing reflects the major tag's commit anyway. The install/smoke step uses pin, as you suggested: installing newest would fail during a legitimate lag while @v0 consumers are perfectly healthy. The false "same commit" comment at :200-201 went with it.

2. publish-pypi's if:. Dropped entirely — dead as you showed (Resolve published version already exit 1s on empty) and dangerous in the way promote's own header condemns. Emptiness is now asserted in-job, mirroring promote's "Validate version shape", so a lost output is red rather than a silent skip that also takes the promotion with it.

The thing your first finding really exposed is that nothing in the repo binds a steps.*.outputs.* reference to a step that writes the key — actionlint models it as an open string map, so the typo was invisible to every gate. That's now CE035 (tests/lint/workflow_outputs.py): every steps.<id>.outputs.<key> and needs.<job>.outputs.<key> in .github/workflows/** must resolve to a real writer, and its negative test is the exact shape of blocker #1.

Two commits have landed since that reply:

• 407b5ea — rebase onto main (plugin ship + the managed-runner migration). promote and preflight follow the pool; e2e deliberately stays on stock ubuntu-latest, with the reason recorded at its own runs-on:, because it exists to reproduce what the documented consumer snippet gets.
• 096485b — five gate-correctness fixes. The two that matter most: agent_crash no longer excuses a zero-token run (it's the categorizer's catch-all, so it's exactly what a missing or broken claude CLI produces — arriving under the one label that let the gate exit 0 with an unread ::warning::), and the PyPI artifact assertion is now set equality rather than containment (installers prefer a platform-specific wheel over our py3-none-any, so one planted …-cp313-manylinux_*.whl would be what uv tool install actually resolves while every file we built still matched byte-for-byte).

The branch was force-pushed with that rebased history just now. Still to do before merge: a rebase onto current main — one append conflict in .claude/harness-candidates.md. Everything from the review that wasn't fixed is booked in that file with reasons rather than dropped: CE036 (ban the skipped-green job gate), CE037, CE040 / extracting the oversized run: blocks into .github/scripts/, and exercising the Action's score gate in the failing direction.

Appreciate the offer to help — the fixes were small once you'd pinned them down. A re-review when you have a moment would be very welcome.