Repository navigation
PPC64LE RHEL8 3.x buildbot is timing out on test_gdb #104735
Description
Buildbot https://buildbot.python.org/all/#/builders/559 has been failing since May 12 with a failure like this:
test_pycfunction (test.test_gdb.PyBtTests.test_pycfunction) Timeout (0:15:00)! Thread 0x00007fff97a24d90 (most recent call first): File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/selectors.py", line 415 in select File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/subprocess.py", line 2075 in _communicate File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/subprocess.py", line 1207 in communicate File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 112 in run_gdb File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 229 in get_stack_trace File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 936 in test_pycfunction
It last passed on commit 718b132 and succeeded on a052be4, suggesting that the culprit is one of these commits:
- a052be4 PEP 688: Crash if __release_buffer__ is called while an exception is active #104371: check return value of calling mv.release (GH-104371: check return value of calling mv.release #104417)
- 5b8cd5a test_typing fails refleak tests #104415: Fix refleak tests for typing.ByteString deprecation (gh-104415: Fix refleak tests for typing.ByteString deprecation #104416)
- d50c37d Use hypothesis for testing the standard library, falling back to stubs #86275: Implementation of hypothesis stubs for property-based tests, with zoneinfo tests (GH-86275: Implementation of hypothesis stubs for property-based tests, with zoneinfo tests #22863)
- 45f5aa8 Implement and document PEP 669. #103082: Filter LINE events in VM, to simplify tool implementation. (GH-103082: Filter LINE events in VM, to simplify tool implementation. #104387)
- 19ee53d Split up _testcapimodule.c #93649: Split gc- and allocation tests from _testcapimodule.c (gh-93649: Split gc- and allocation tests from _testcapimodule.c #104403)
- b2c1b4d Make it possible to mark Argument Clinic args as unused #104389: Add 'unused' keyword to Argument Clinic C converters (gh-104389: Add 'unused' keyword to Argument Clinic C converters #104390)
- 15795b5 Isolate the _io extension module #101819: Prepare _io._IOBase for module state (gh-101819: Prepare _io._IOBase for module state #104386)
Activity
encukou commented on Jan 18, 2024
@vstinner is the PEP-11 powerpc64le maintainer.
Victor, do you need any help with this issue? I'm afraid I can't help too much, though -- I don't have a PPC64 box available.
encukou commented on Jan 18, 2024
Oh, sorry! It turns out that powerpc64le is listed twice in PEP-11, and in the other entry, I am listed as the maintainer.
Unfortunately I will need to step down from this role as I no longer have easy access to the hardware.
vstinner commented on Jan 18, 2024
Yeah, ppc64le with GCC is supported at Tier-2 level: it's the config of RHEL operating system. ppc64le with clang is only supported at Tier-3 level: best effort.
I made multiple test_gdb changes last year. Is the ppc64le buildbot still failing with timeout time to time? If yes, does someone have links to such recent failures?
vstinner commented on Jan 18, 2024
test_gdb.test_pycfunction() is running tons of tests in a single method. I splitted test_gdb.py into a test_gdb test package (multiple .py files): commit 8f324b7 of issue gh-109972.
If it still fails time to time, maybe we should simply increase the timeout. I noticed that some PPC64LE buildbot workers are way slower than other buildbot workers. I don't recall which ones.
vstinner commented on Jan 18, 2024
If you still care, you can get access to ppc64le machines from Fedora: https://fedoraproject.org/wiki/Test_Machine_Resources_For_Package_Maintainers I can give you a SSH access the ppc64 machines maintained by Red Hat, you should be able to easily reproduce the issue in a few minutes (need to clone the repo, build Python with similar build options than the worker, and run the test). Otherwise, we can just downgrade ppc64le+gcc to Tier-2, that's ok.
encukou commented on Jan 24, 2024
Let's do both: downgrade to tier 3, but I'll keep an eye on stable buildbots, and use Fedora infra when I around get to investigating this issue.
vstinner commented on Jan 24, 2024
I lost track of this issue. Does the issue still occur time to time?
encukou commented on Jan 25, 2024
No.
And the buildbot logs from May are gone, so it wouldn't be easy to investigate this. I don't think it's worth it to keep the issue open.
vstinner commented on Jan 25, 2024
Ok, thanks for keeping track of that issue.