TST, CI: Skip platform dependent array API failures + adding windows and macOS env in CI

[prady0t]

Closes: #238

[ev-br]

Ah, you're adding the new CI runs here... because I did not press the green button. OK, let's carry on in this PR then.

[prady0t]

Two new failures in CI

macOS:

array_api_tests/test_special_cases.py::test_unary[atanh(real(x_i) is NaN and isfinite(imag(x_i))) -> NaN + NaN j] - AssertionError: out=(nan+1.5707963267948966j), but should be NaN + NaN j [atanh()]

Windows:

array_api_tests/test_creation_functions.py::test_arange - AssertionError: out.dtype=int64, should be default integer dtype int32 [arange()]

array_api_tests/test_creation_functions.py::test_full - AssertionError: out.dtype=int64, should be default integer dtype int32 [full()]

[prady0t]

On my machine (macOS 26.5.1) it passed

array_api_tests/test_special_cases.py::test_unary[atanh(real(x_i) is NaN and isfinite(imag(x_i))) -> NaN + NaN j] PASSED

For test on windows:

full((),0) and arange(0,0) are problematic.

[betatim]

For my education: how does the xfail machinery know that these tests should only be xfailed on macOS?

If possible, it would be good to not have a list of xfails that is "optional" (if it passes that is fine, if it fails that is also fine). In pytest there is a "strict" mode for xfails where something that is marked as failing but passes is a failure (🔴 ➡️ 🟢 ➡️ 🔴 - good luck not getting confused :D) . Maybe it is beyond the scope of this PR to get array-api-strict to run in strict mode, but we should aim for that ideal state. So we should briefly consider it here for OS dependent xfails.

[prady0t]

For my education: how does the xfail machinery know that these tests should only be xfailed on macOS?

It doesn't; it loosely expects it to fall evrywhere. Yes, if we use --strict, it would raise an error if it passes.

Maybe we can modify test collection behaviour in the contest.py file? Or, use something like:

@pytest.mark.skipif(sys.platform == "the platform")

On the tests that are failing specifically on certain platforms.

[betatim]

Yeah maybe pytest.mark.xfail is the way to go here. They even use platform as the example for that decorator :D

[ev-br]

I'd have a rather strong desire to keep all xfails/skips in the same file. Tests themselves are in a different repository (array-api-tests), are reused across multiple array libraries, and themselves have zero pytest.mark.{skip, xfail} markers. All skipping/failing is done via these text files, and is specific to array libraries (compare to -{xfail,skip}.txt files in https://github.com/data-apis/array-api-compat).
Moreover, whether entries in a file are skips or xfails is controllable at the test suite invocation time by an environment variable [1].
IOW, whether to skip a test or not is a decision of whoever runs the test suite, when they run the test suite.

Therefore, if we are seriously looking to enable platform dependence, we need a DSL for the .txt files themselves. Currently, there isn't one, and the whole logic is here:
https://github.com/data-apis/array-api-tests/blob/master/conftest.py#L166-L200

[1] this is important from performance POV: with hypothesis, xfails are be very expensive, and needlessly so, cf data-apis/array-api-compat#321

[betatim]

Ah, I didn't realise that the list of xfails is about tests from the array-api-tests repo, not tests that live in this repo.

How about extending the --xfails-file argument so that it can be passed multiple times? Then we can maintain a file that contains xfails that apply to all platforms and one each per platform that contains tests that only xfail on that platform. Something like pytest .. --xfails-file general-xfails.txt --xfails-file macos-only-xfails.txt ? I've not poked around pytest's arg handling enough to know if it support it. Otherwise we might need a recipe for combining two files into one on the fly. We could document this in this (array-api-strict) repo for those who want to invoke the array-api-tests tests on array-api-strict

[betatim]

I'm a hypothesis luddite, so I have nothing useful to say about the "use skips even if you mean xfail because xfails make things slow" :-/