Use `pytest-run-parallel` to test thread safety

[leofang]

It should work for both FT & non-FT builds.

• https://github.com/Quansight-Labs/pytest-run-parallel
• https://py-free-threading.github.io/testing/

[kkraus14]

I suspect this would need a non-trivial amount of test refactoring to avoid teardowns that interact with the CUDA driver in a stateful way? There's so much state in the CUDA driver that it's hard to imagine doing this in an automated way safely.

[leofang]

Maybe, maybe not. @seberg has invested a lot of efforts in CuPy and we're generally happy with us gaining more thread safety, following community standard, and having a CUDA project reaching FT-readiness. I am hopeful.

[seberg]

Yeah, I would propose to try it, after the CuPy release maybe I can have a look as well. CuPy had fun caches, the memory manager, and an awful lot of legacy unittest-style tests that were painful... I would think this is all much easier for cuda-python.

I am not sure what might be problematic, e.g. graph capture related tests, seems like one that one may need to just mark as thread-unsafe in a first iteration. (In CuPy they are largely fine, but not fully e.g. because module loading/unloading in multiple threads can be problematic.)

EDIT: I started to have a bit of a look.

I suspect this would need a non-trivial amount of test refactoring to avoid teardowns that interact with the CUDA driver in a stateful way?

Yeah, I now see what you mean. cuda-pathfinder tests seem fine with a few small thread-unsafe markers. But cuda-bindings/cuda-core would skip almost all tests without larger re-factor since it's fixtures need to run per-thread (e.g. due to cuInit()).
It may work to convert these to normal decorators, but without that the pytest-run-parallel run wouldn't add much.