Skip to content

[Bug]: bug-test agentic workflow cannot run tests — no Python/uv provisioning and PyPI blocked by egress firewall #3997

Description

@mnriem

Bug Description

The bug-test agentic workflow (stage 3 of the assess → fix → test bug pipeline) cannot execute this repository's test suite. Runs come back inconclusive with an environment/setup failure rather than a real pass/fail, because the workflow environment can neither find pytest/uv preinstalled nor install them (PyPI is blocked by the egress firewall).

This is a CI/workflow-infrastructure bug, not a specify CLI bug. It was surfaced on issue #3990: the bug-test run (actions run 31021548784) for PR #3995 reported:

Setup failure (blocking): pytest is not installed in the CI runner. All pip install attempts failed with 403 Forbidden (PyPI access blocked by network policy). The venv was created successfully but dependency installation was not possible.

Net effect: the fix could only be verified by code inspection, and the pipeline's third stage provides no actual test signal for any Python change.

Steps to Reproduce

  1. Label a Python-fix issue bug-test so .github/workflows/bug-test.lock.yml runs (e.g. issue [Bug]: specify init --here --force does not reapply installed presets, leaving preset registry out of sync with disk #3990 / PR [bug-fix] Fix init-force-preset-desync: reapply presets/extensions on init --here --force #3995).
  2. Observe the agent follow bug-test.md Step 3, which instructs it to run uv sync --extra test then uv run pytest for uv-based Python projects.
  3. The install fails: pytest/uv are not provisioned and outbound requests to PyPI return 403 Forbidden.
  4. The workflow posts a ⚠️ inconclusive report and applies tests-inconclusive.

Expected Behavior

The bug-test workflow provisions the Python test toolchain and runs pytest, producing a real passing/failing result for Python fixes.

Actual Behavior

Tests never run. The result is always inconclusive for Python changes because of two compounding gaps in .github/workflows/bug-test.lock.yml (and its source .github/workflows/bug-test.md):

  1. No Python/test toolchain is provisioned. The job runs actions/setup-node and installs the Copilot CLI, but there is no actions/setup-python / astral-sh/setup-uv step and no uv sync / pip install before the agent starts. pytest and uv are listed as allowed tools but are not actually present. spec-kit requires them — pytest is under the test optional-dependencies extra in pyproject.toml.

  2. The egress firewall blocks PyPI. The gh-aw squid firewall allowDomains list (the GH_AW_ALLOWED_DOMAINS / awf-config.json network.allowDomains in the lock file) permits registry.npmjs.org, the Ubuntu/apt mirrors, github.com, Google APIs, etc., but does not include pypi.org or files.pythonhosted.org. So the agent's uv sync --extra test / pip install attempts return 403. Notably JS's npm registry is allowed while Python's PyPI is not — the environment appears tuned for JS ecosystems and was never fully wired for this Python repo.

The bug-test.md prompt itself (Step 3) tells the agent to run uv sync --extra test, a command guaranteed to fail behind this allowlist.

Proposed Fix

Either (or both) of:

  • Preinstall before the agent stage (preferred/most robust): add actions/setup-python + astral-sh/setup-uv and run uv sync --extra test (or pip install -e .[test]) as a normal workflow step while the network is open, so the venv is populated and the agent can run uv run pytest offline.
  • Allowlist PyPI: add pypi.org and files.pythonhosted.org to the firewall allowDomains so the agent's own uv sync / pip install succeeds at agent time.

Since these workflows are compiled, the change belongs in the source .github/workflows/bug-test.md (and the shared gh-aw config that produces the allowDomains list), followed by recompiling the .lock.yml. The same gap likely affects bug-fix and any other Python-executing agentic workflows and should be fixed consistently.

Additional Context

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions