You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The bug-test agentic workflow (stage 3 of the assess → fix → test bug pipeline) cannot execute this repository's test suite. Runs come back inconclusive with an environment/setup failure rather than a real pass/fail, because the workflow environment can neither find pytest/uv preinstalled nor install them (PyPI is blocked by the egress firewall).
This is a CI/workflow-infrastructure bug, not a specify CLI bug. It was surfaced on issue #3990: the bug-test run (actions run 31021548784) for PR #3995 reported:
Setup failure (blocking):pytest is not installed in the CI runner. All pip install attempts failed with 403 Forbidden (PyPI access blocked by network policy). The venv was created successfully but dependency installation was not possible.
Net effect: the fix could only be verified by code inspection, and the pipeline's third stage provides no actual test signal for any Python change.
Observe the agent follow bug-test.md Step 3, which instructs it to run uv sync --extra test then uv run pytest for uv-based Python projects.
The install fails: pytest/uv are not provisioned and outbound requests to PyPI return 403 Forbidden.
The workflow posts a ⚠️ inconclusive report and applies tests-inconclusive.
Expected Behavior
The bug-test workflow provisions the Python test toolchain and runs pytest, producing a real passing/failing result for Python fixes.
Actual Behavior
Tests never run. The result is always inconclusive for Python changes because of two compounding gaps in .github/workflows/bug-test.lock.yml (and its source .github/workflows/bug-test.md):
No Python/test toolchain is provisioned. The job runs actions/setup-node and installs the Copilot CLI, but there is no actions/setup-python / astral-sh/setup-uv step and no uv sync / pip install before the agent starts. pytest and uv are listed as allowed tools but are not actually present. spec-kit requires them — pytest is under the test optional-dependencies extra in pyproject.toml.
The egress firewall blocks PyPI. The gh-aw squid firewall allowDomains list (the GH_AW_ALLOWED_DOMAINS / awf-config.jsonnetwork.allowDomains in the lock file) permits registry.npmjs.org, the Ubuntu/apt mirrors, github.com, Google APIs, etc., but does not include pypi.org or files.pythonhosted.org. So the agent's uv sync --extra test / pip install attempts return 403. Notably JS's npm registry is allowed while Python's PyPI is not — the environment appears tuned for JS ecosystems and was never fully wired for this Python repo.
The bug-test.md prompt itself (Step 3) tells the agent to run uv sync --extra test, a command guaranteed to fail behind this allowlist.
Proposed Fix
Either (or both) of:
Preinstall before the agent stage (preferred/most robust): add actions/setup-python + astral-sh/setup-uv and run uv sync --extra test (or pip install -e .[test]) as a normal workflow step while the network is open, so the venv is populated and the agent can run uv run pytest offline.
Allowlist PyPI: add pypi.org and files.pythonhosted.org to the firewall allowDomains so the agent's own uv sync / pip install succeeds at agent time.
Since these workflows are compiled, the change belongs in the source .github/workflows/bug-test.md (and the shared gh-aw config that produces the allowDomains list), followed by recompiling the .lock.yml. The same gap likely affects bug-fix and any other Python-executing agentic workflows and should be fixed consistently.
The bug-test agent behaved correctly — it located the fix, fell back to code inspection, and honestly reported inconclusive rather than fabricating a result. The defect is purely in the workflow's environment setup.
Bug Description
The
bug-testagentic workflow (stage 3 of the assess → fix → test bug pipeline) cannot execute this repository's test suite. Runs come back inconclusive with an environment/setup failure rather than a real pass/fail, because the workflow environment can neither findpytest/uvpreinstalled nor install them (PyPI is blocked by the egress firewall).This is a CI/workflow-infrastructure bug, not a
specifyCLI bug. It was surfaced on issue #3990: thebug-testrun (actions run 31021548784) for PR #3995 reported:Net effect: the fix could only be verified by code inspection, and the pipeline's third stage provides no actual test signal for any Python change.
Steps to Reproduce
bug-testso.github/workflows/bug-test.lock.ymlruns (e.g. issue [Bug]: specify init --here --force does not reapply installed presets, leaving preset registry out of sync with disk #3990 / PR [bug-fix] Fix init-force-preset-desync: reapply presets/extensions oninit --here --force#3995).bug-test.mdStep 3, which instructs it to runuv sync --extra testthenuv run pytestfor uv-based Python projects.pytest/uvare not provisioned and outbound requests to PyPI return403 Forbidden.⚠️ inconclusivereport and appliestests-inconclusive.Expected Behavior
The
bug-testworkflow provisions the Python test toolchain and runspytest, producing a real passing/failing result for Python fixes.Actual Behavior
Tests never run. The result is always
inconclusivefor Python changes because of two compounding gaps in.github/workflows/bug-test.lock.yml(and its source.github/workflows/bug-test.md):No Python/test toolchain is provisioned. The job runs
actions/setup-nodeand installs the Copilot CLI, but there is noactions/setup-python/astral-sh/setup-uvstep and nouv sync/pip installbefore the agent starts.pytestanduvare listed as allowed tools but are not actually present. spec-kit requires them —pytestis under thetestoptional-dependencies extra inpyproject.toml.The egress firewall blocks PyPI. The gh-aw squid firewall
allowDomainslist (theGH_AW_ALLOWED_DOMAINS/awf-config.jsonnetwork.allowDomainsin the lock file) permitsregistry.npmjs.org, the Ubuntu/apt mirrors,github.com, Google APIs, etc., but does not includepypi.orgorfiles.pythonhosted.org. So the agent'suv sync --extra test/pip installattempts return 403. Notably JS's npm registry is allowed while Python's PyPI is not — the environment appears tuned for JS ecosystems and was never fully wired for this Python repo.The
bug-test.mdprompt itself (Step 3) tells the agent to runuv sync --extra test, a command guaranteed to fail behind this allowlist.Proposed Fix
Either (or both) of:
actions/setup-python+astral-sh/setup-uvand runuv sync --extra test(orpip install -e .[test]) as a normal workflow step while the network is open, so the venv is populated and the agent can runuv run pytestoffline.pypi.organdfiles.pythonhosted.orgto the firewallallowDomainsso the agent's ownuv sync/pip installsucceeds at agent time.Since these workflows are compiled, the change belongs in the source
.github/workflows/bug-test.md(and the shared gh-aw config that produces theallowDomainslist), followed by recompiling the.lock.yml. The same gap likely affectsbug-fixand any other Python-executing agentic workflows and should be fixed consistently.Additional Context
.github/workflows/bug-test.md,.github/workflows/bug-test.lock.yml.bug-testagent behaved correctly — it located the fix, fell back to code inspection, and honestly reportedinconclusiverather than fabricating a result. The defect is purely in the workflow's environment setup.