Make it possible for faulthandler to terminate the pytest process on timeout by ogrisel · Pull Request #13679 · pytest-dev/pytest · GitHub

Closes #13678.

This would help projects avoid wasting their precious CI resources when running extensive thread-safety tests (which will become more popular in the new free-threading era).

I also changed the logic to mark the test as XFAIL conditionally on the presence of the "CI" environment variable instead of always skipping (related to #7022). This makes it possible to check that the test pass as expected when running locally.

If you think this is too risky, I can remove the condition and always mark the offending params with the xfail mark instead of skipping.

EDIT: I will do the later right away because I have already observed such a random failure on the CI: https://github.com/pytest-dev/pytest/actions/runs/17269674978/job/49010727146

EDIT 2: that does not work because xfail_strict = true is configured in pyproject.toml. Let me skip instead...

…timeout

At first glance this looks nice

There may be some nitpicks but they relate to preexisting patterns

I think I understand why the CI var was not properly detected: we probably need to add it to the passenv directive of tox. Let me try that quickly.

This reverts commit be4f0ba.

Ok so many other unrelated test started to fail because they are not written in a way that expect to be ever executed in the presence of the CI environment variable... Let me revert 000876e and 5066d98...

…ests on CI" This reverts commit 000876e.

This reverts commit 5066d98.

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>

… bool instance

I synced this PR with main as a follow-up on #13684. I think this PR is now ready for review. The matter of actually fixing #7022 is deferred to the exploratory #13695.

I changed the skip mark to a conditional mark to make it easy to check that this test actually passes locally.

Can it be an xfail, then?

I think I tried earlier, but this does not work because xfail_strict = true is configured in pyproject.toml and those tests would XPASS on most CI configurations except ubuntu-py314 that tends to fail for a reason I do not understand as investigated in #13695.

I synced this PR with main after the recent merge of #13695 to see if the new test also fails on ubuntu-py314 or not. If the new test fails here, I will add the same conditional skip as done for test_timeout in #13695

The new test did not fail, but let me trigger it a few more times to check that it's not (too) flaky.

EDIT: the new test is also flaky, selectively on ubuntu-py314. Let me push a fix to conditionally skip it in cfa2aa6.

I had a look at the codecov results and there is a single missing line:

https://app.codecov.io/gh/pytest-dev/pytest/pull/13679#f164a8d7864732ad2650984cfc20ed8e-R96

This line is actually covered in the new test, but the coverage data is not collected because the test uses a Python subprocess where coverage tracing is not enabled. I think collecting such coverage data is possible but out of the scope of the current PR.

One of the CI jobs for cfa2aa6 failed for an unrelated problem (a transient codecov upload error). All the non-skipped tests actually passed.

@RonnyPfannschmidt @webknjaz @nicoddemus I think this PR is ready for final review.

Thanks for delivering this as well as gracefully fixing prerequisites

Awesome, thanks for working on this! I've triggered hangs like this in CI and resorted to using pytest-timeout as a workaround. It didn't occur to me that pytest itself can handle this.

This week, we hit a random deadlock that caused a 6h CI timeout while releasing scikit-learn 1.7.2 on conda-forge. This is quite a waste of resources and slows down the release process by quite a lot when it happens. Having the option to configure pytest to early exit before the CI timeout would definitely help make our CI handling less painful.

Sorry this took so long im slowly catching up

ogrisel added 2 commits August 27, 2025 16:04

Make it possible for faulthandler to terminate the pytest process on …

a41b231

…timeout

Add myself to AUTHORS and add changelog entry.

91d600c

psf-chronographer Bot added the bot:chronographer:provided (automation) changelog entry is part of PR label Aug 27, 2025

ogrisel mentioned this pull request Aug 27, 2025

MAINT configure a default faulthandler timeout to help debug deadlocks joblib/loky#463

Merged

ogrisel added 2 commits August 27, 2025 16:50

Update the documentation

d3274b1

Skip flakky CI tests

be4f0ba

RonnyPfannschmidt approved these changes Aug 27, 2025

View reviewed changes

ogrisel added 2 commits August 27, 2025 17:49

Revert "Skip flakky CI tests"

5066d98

This reverts commit be4f0ba.

Pass the CI env in tox.ini and use it to conditionally skip tests on CI

000876e

ogrisel commented Aug 27, 2025

View reviewed changes

Comment thread src/_pytest/faulthandler.py Outdated

ogrisel added 2 commits August 27, 2025 18:09

Revert "Pass the CI env in tox.ini and use it to conditionally skip t…

dfc2207

…ests on CI" This reverts commit 000876e.

Revert "Revert "Skip flakky CI tests""

c7abf41

This reverts commit 5066d98.

ogrisel commented Aug 27, 2025

View reviewed changes

Comment thread testing/test_faulthandler.py

webknjaz reviewed Aug 28, 2025

View reviewed changes

Comment thread changelog/13678.feature.rst Outdated

Apply suggestions from code review

713166e

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>

ogrisel mentioned this pull request Aug 28, 2025

Fix passenv CI in tox ini and make tests insensitive to the presence of the CI env variable #13684

Merged

Better way to tell mypy that faulthandler_exit_on_timeout is always a…

da1adf3

… bool instance

patchback Bot mentioned this pull request Sep 2, 2025

[PR #13684/bf92debb backport][8.4.x] Fix passenv CI in tox ini and make tests insensitive to the presence of the CI env variable #13694

Merged

ogrisel added 2 commits September 3, 2025 09:43

Merge branch 'main' into faulthandler-exit-on-timeout

65ef53f

skip test only on CI

634ffcc

ogrisel commented Sep 3, 2025

View reviewed changes

Merge branch 'main' into faulthandler-exit-on-timeout

e1524d3

Trigger CI

fbce039

Skip the new test on ubuntu-py314

ea99722

ogrisel added 3 commits September 9, 2025 14:57

Actually skip the new test on ubuntu-py314

cfa2aa6

Trigger CI

136b515

Trigger CI

d752ce9

RonnyPfannschmidt approved these changes Sep 9, 2025

View reviewed changes

Merge branch 'main' into faulthandler-exit-on-timeout

5799868

ngoldbaum mentioned this pull request Sep 24, 2025

Add guidance suggesting using pytest-timeout to handle hangs Quansight-Labs/free-threaded-compatibility#241

Closed

Liam-DeVoe mentioned this pull request Oct 5, 2025

Thread safety and Pytest #13768

Open

12 tasks

RonnyPfannschmidt merged commit 83da343 into pytest-dev:main Oct 6, 2025
33 checks passed

ogrisel deleted the faulthandler-exit-on-timeout branch October 7, 2025 08:51

ngoldbaum mentioned this pull request Oct 8, 2025

Misc CI/Testing updates Quansight-Labs/free-threaded-compatibility#263

Merged

ogrisel mentioned this pull request Oct 28, 2025

⚠️ CI failed on Linux_free_threaded.pylatest_free_threaded (last failure: Jan 24, 2026) ⚠️ scikit-learn/scikit-learn#32430

Closed

+                          True,
+                          marks=pytest.mark.skipif(
+                              "CI" in os.environ, reason="sometimes crashes on CI (#7022)"
+                          ),

Make it possible for faulthandler to terminate the pytest process on timeout by ogrisel · Pull Request #13679 · pytest-dev/pytest · GitHub

Uh oh!

Uh oh!

Make it possible for faulthandler to terminate the pytest process on timeout - #13679

Make it possible for faulthandler to terminate the pytest process on timeout#13679RonnyPfannschmidt merged 19 commits intopytest-dev:mainpytest-dev/pytest:mainfrom ogrisel:faulthandler-exit-on-timeoutogrisel/pytest:faulthandler-exit-on-timeoutCopy head branch name to clipboard

Conversation

ogrisel commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RonnyPfannschmidt left a comment

Choose a reason for hiding this comment

Uh oh!

ogrisel commented Aug 27, 2025

Uh oh!

Uh oh!

ogrisel commented Aug 27, 2025

Uh oh!

Uh oh!

Uh oh!

ogrisel commented Sep 3, 2025

Uh oh!

ogrisel Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

webknjaz Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

ogrisel Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogrisel commented Sep 9, 2025

Uh oh!

ogrisel commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ogrisel commented Sep 9, 2025

Uh oh!

ogrisel commented Sep 9, 2025

Uh oh!

ogrisel commented Sep 9, 2025

Uh oh!

RonnyPfannschmidt left a comment

Choose a reason for hiding this comment

Uh oh!

ngoldbaum commented Sep 16, 2025

Uh oh!

ogrisel commented Sep 17, 2025

Uh oh!

Uh oh!

RonnyPfannschmidt commented Oct 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Make it possible for faulthandler to terminate the pytest process on timeout#13679
RonnyPfannschmidt merged 19 commits into
pytest-dev:mainfrom
ogrisel:faulthandler-exit-on-timeout

ogrisel commented Aug 27, 2025 •
edited
Loading

ogrisel Sep 4, 2025 •
edited
Loading

ogrisel commented Sep 9, 2025 •
edited
Loading