Skip to content

TST, CI: Skip platform dependent array API failures + adding windows and macOS env in CI #240

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
prady0t wants to merge 4 commits into
base: main
Choose a base branch
from
+17 −2
Open

TST, CI: Skip platform dependent array API failures + adding windows and macOS env in CI #240

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4318e5b
Skip platform dependent array API failures
prady0t Aug 10, 2026
cf9cd80
Apply suggestions from code review
ev-br Aug 10, 2026
fe6145d
Adding windows and macos CI environments
prady0t Aug 10, 2026
1864e72
Merge branch 'add-test-skips' of https://github.com/prady0t/array-api…
prady0t Aug 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .github/workflows/array-api-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,20 @@ env:

jobs:
array-api-tests:
runs-on: ubuntu-latest
runs-on: ${{ matrix.os }}
defaults:
run:
shell: bash
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: ['3.10', '3.13', '3.14']
numpy-version: ['1.26', 'latest', 'dev']
exclude:
- python-version: '3.10'
numpy-version: 'latest'
- python-version: '3.10'
numpy-version: 'dev'
numpy-version: 'dev'
- python-version: '3.13'
numpy-version: '1.26'
- python-version: '3.14'
Expand Down
11 changes: 11 additions & 0 deletions array-api-tests-xfails.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,14 @@ array_api_tests/test_dlpack.py::test_dunder_dlpack

# NumPy 1.26: TypeError: numpy.from_dlpack() takes no keyword arguments
array_api_tests/test_dlpack.py::test_from_dlpack

# https://github.com/numpy/numpy/issues/32234 : macOS specific failures
array_api_tests/test_special_cases.py::test_unary[atanh(real(x_i) is +0 and imag(x_i) is NaN) -> +0 + NaN j]

@betatim betatim Aug 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my education: how does the xfail machinery know that these tests should only be xfailed on macOS?

If possible, it would be good to not have a list of xfails that is "optional" (if it passes that is fine, if it fails that is also fine). In pytest there is a "strict" mode for xfails where something that is marked as failing but passes is a failure (🔴 ➡️ 🟢 ➡️ 🔴 - good luck not getting confused :D) . Maybe it is beyond the scope of this PR to get array-api-strict to run in strict mode, but we should aim for that ideal state. So we should briefly consider it here for OS dependent xfails.

@prady0t prady0t Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my education: how does the xfail machinery know that these tests should only be xfailed on macOS?

It doesn't; it loosely expects it to fall evrywhere. Yes, if we use --strict, it would raise an error if it passes.

Maybe we can modify test collection behaviour in the contest.py file? Or, use something like:

@pytest.mark.skipif(sys.platform == "the platform")

On the tests that are failing specifically on certain platforms.

@betatim betatim Aug 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah maybe pytest.mark.xfail is the way to go here. They even use platform as the example for that decorator :D

@ev-br ev-br Aug 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd have a rather strong desire to keep all xfails/skips in the same file. Tests themselves are in a different repository (array-api-tests), are reused across multiple array libraries, and themselves have zero pytest.mark.{skip, xfail} markers. All skipping/failing is done via these text files, and is specific to array libraries (compare to -{xfail,skip}.txt files in https://github.com/data-apis/array-api-compat).
Moreover, whether entries in a file are skips or xfails is controllable at the test suite invocation time by an environment variable [1].
IOW, whether to skip a test or not is a decision of whoever runs the test suite, when they run the test suite.

Therefore, if we are seriously looking to enable platform dependence, we need a DSL for the .txt files themselves. Currently, there isn't one, and the whole logic is here:
https://github.com/data-apis/array-api-tests/blob/master/conftest.py#L166-L200

[1] this is important from performance POV: with hypothesis, xfails are be very expensive, and needlessly so, cf data-apis/array-api-compat#321

@betatim betatim Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I didn't realise that the list of xfails is about tests from the array-api-tests repo, not tests that live in this repo.

How about extending the --xfails-file argument so that it can be passed multiple times? Then we can maintain a file that contains xfails that apply to all platforms and one each per platform that contains tests that only xfail on that platform. Something like pytest .. --xfails-file general-xfails.txt --xfails-file macos-only-xfails.txt ? I've not poked around pytest's arg handling enough to know if it support it. Otherwise we might need a recipe for combining two files into one on the fly. We could document this in this (array-api-strict) repo for those who want to invoke the array-api-tests tests on array-api-strict

@betatim betatim Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a hypothesis luddite, so I have nothing useful to say about the "use skips even if you mean xfail because xfails make things slow" :-/

@ev-br ev-br Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something like pytest .. --xfails-file general-xfails.txt --xfails-file macos-only-xfails.txt ?

Oh this is a neat idea! Definitely much better than inventing a DSL for a single xfails.txt file and parsing it out the comments or something.

To test-drive the idea, there are --skips-file=... and --xfails-file=... already, so they can be reused :-).
This would be a hack, of course, so longer term I agree it'd be better to be able to pass a semantically correct file multiple times. I haven't try to see how pytest handles these, but it should be very much doable I think.

I'm a hypothesis luddite, so I have nothing useful to say about the "use skips even if you mean xfail because xfails make things slow" :-/

Well, in the current setup the user keeps using semantically meaningful --xfails-file name. There's an additional switch to toggle the internal details of how pytest and hypothesis handle an expected failure. This switch is obscure enough (an environment variable) to be only used deliberately, when pressed hard enough. An "anecdotal evidence" from https://github.com/data-apis/array-api-tests#turning-xfails-into-skips is real: we did manage to increase the number of examples by a factor of 4-5 and keep a reasonable tun time for some backends. And with hypothesis the number of examples is the measure of reliability: at some point Dask CI had 10 examples, and that meant it was effectively untested.

array_api_tests/test_special_cases.py::test_unary[atanh(real(x_i) is 1 and imag(x_i) is +0) -> +infinity + 0j]
array_api_tests/test_special_cases.py::test_unary[atanh(real(x_i) is +infinity and imag(x_i) is NaN) -> +0 + NaN j]
array_api_tests/test_special_cases.py::test_unary[atanh(real(x_i) is NaN and imag(x_i) is +infinity) -> \xb10 + \u03c0j/2]
array_api_tests/test_special_cases.py::test_unary[tanh(real(x_i) is +0 and imag(x_i) is +infinity) -> +0 + NaN j]
array_api_tests/test_special_cases.py::test_unary[tanh(real(x_i) is +0 and imag(x_i) is NaN) -> +0 + NaN j]

# https://github.com/numpy/numpy/issues/32234 : macOS specific failure
array_api_tests/test_operators_and_elementwise_functions.py::test_sqrt
Comment thread
ev-br marked this conversation as resolved.
Loading