FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Work around potential Mono bug, that hangs the runtime when new threads start by lostmsu · Pull Request #1779 · pythonnet/pythonnet · GitHub

Work around potential Mono bug, that hangs the runtime when new threads start - #1779

Merged
filmor merged 1 commit into
pythonnet:masterfrom
losttech:bugs/1766-MacOS-Test-Hang
May 4, 2022
Merged

Work around potential Mono bug, that hangs the runtime when new threads start#1779
filmor merged 1 commit into
pythonnet:masterfrom
losttech:bugs/1766-MacOS-Test-Hang

Conversation

lostmsu commented May 2, 2022
edited
Loading

Copy link
Copy Markdown
Member

I suspect the issue is actually with Mono GC and the relatively new mode of suspending threads, which has a few (closed) bugs against it, complaining about hangs. In many of those bugs (you can find them here) switching to the old preemptive mode works around the issue. I tried it for our tests, and it seems to solve it too.

The issue also vaguely mentioned in https://www.mono-project.com/docs/advanced/runtime/docs/coop-suspend/#thread-startfinish-still-bad

So the fix is to simply set MONO_THREADS_SUSPEND to preemptive in CI before running tests in Mono.

Fixes #1766

This workaround seems to solve the issue on my local machine as well as in CI. I did 5+ CI full matrix reruns to confirm. Before the fix a few MacOS legs would time out in every run.

Below is some history of finding this workaround.

lostmsu force-pushed the bugs/1766-MacOS-Test-Hang branch 2 times, most recently from 88ecef6 to 64a04d3 Compare May 2, 2022 18:40

lostmsu commented May 2, 2022
edited
Loading

Copy link
Copy Markdown
Member Author

:/ it does not fail when the blame mode is on in dotnet test :/

On my own machine I was able to reproduce it on the first try, but then it never reproduced after that. :/

lostmsu commented May 2, 2022
edited
Loading

Copy link
Copy Markdown
Member Author

Looks like the issue may be with the TestThread function. Perhaps we are missing some synchronization there. I previously suspected that it needs memory barriers, because AFAIK acquiring GIL does not insert them. See this run passing on all MacOS versions with that method disabled: https://github.com/pythonnet/pythonnet/actions/runs/2260894105

lostmsu force-pushed the bugs/1766-MacOS-Test-Hang branch 3 times, most recently from 854f20b to 8d9d27e Compare May 3, 2022 20:59
lostmsu force-pushed the bugs/1766-MacOS-Test-Hang branch from 8d9d27e to f730963 Compare May 4, 2022 04:16
lostmsu changed the title WIP MacOS tests hang Work around potential Mono bug, that hangs the runtime when new threads start May 4, 2022
lostmsu requested a review from filmor May 4, 2022 04:33
lostmsu marked this pull request as ready for review May 4, 2022 04:35

filmor commented May 4, 2022

Copy link
Copy Markdown
Member

This is probably also something that we should include in our docs.

filmor merged commit bbfa252 into pythonnet:master May 4, 2022
lostmsu mentioned this pull request May 4, 2022
lostmsu deleted the bugs/1766-MacOS-Test-Hang branch May 4, 2022 17:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Embedding tests randomly hang in MacOS

2 participants


Back | FazBrowse Home | New Git URL