I'm not exactly sure why test_importlib fails, but I don't want to push a dummy commit just to retry it.

Anyways, the TSAN checks for test_free_threading.test_slots are 🟢

I'm not exactly sure why test_importlib fails, but I don't want to push a dummy commit just to retry it.

I've relaunched the test manually (I can't relaunch the SSL test though, I don't know why)

I finally had some time to come back to this.

I guess that moving the stores after error checking does slightly change the externally-visible behavior.
Should we write a news entry?

!buildbot nogil

🤖 New build scheduled with the buildbot fleet by @colesbury for commit 6c5cec5 🤖

The command will test the builders whose names match following regular expression: nogil

The builders matched are:

• AMD64 Windows Server 2022 NoGIL PR
• AMD64 Fedora Rawhide NoGIL refleaks PR
• AMD64 CentOS9 NoGIL Refleaks PR
• x86-64 MacOS Intel ASAN NoGIL PR
• AMD64 Windows PGO NoGIL PR
• aarch64 Fedora Rawhide NoGIL PR
• PPC64LE Fedora Rawhide NoGIL refleaks PR
• aarch64 Fedora Rawhide NoGIL refleaks PR
• ARM64 MacOS M1 NoGIL PR
• AMD64 CentOS9 NoGIL PR
• ARM64 MacOS M1 Refleaks NoGIL PR
• AMD64 Fedora Rawhide NoGIL PR
• PPC64LE Fedora Rawhide NoGIL PR
• x86-64 MacOS Intel NoGIL PR

Thanks @dpdani!

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL9 3.x has failed when building commit 979bf24.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1565/builds/859) and take a look at the build logs.
4. Check if the failure is related to this commit (979bf24) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1565/builds/859

Failed tests:

• test_capi

Failed subtests:

• test_char - test.test_capi.test_structmembers.ReadWriteTests_NewAPI.test_char
• test_char - test.test_capi.test_structmembers.ReadWriteTests_OldAPI.test_char

Summary of the results of the build (if available):

==

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x/build/Lib/test/test_capi/test_structmembers.py", line 170, in test_char
    self.assertEqual(ts.T_CHAR, "c")
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
AssertionError: '\x00' != 'c'
- �
+ c

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL9 LTO + PGO 3.x has failed when building commit 979bf24.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1578/builds/862) and take a look at the build logs.
4. Check if the failure is related to this commit (979bf24) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1578/builds/862

Failed tests:

• test_capi

Failed subtests:

• test_char - test.test_capi.test_structmembers.ReadWriteTests_NewAPI.test_char
• test_char - test.test_capi.test_structmembers.ReadWriteTests_OldAPI.test_char

Summary of the results of the build (if available):

==

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.lto-pgo/build/Lib/test/test_capi/test_structmembers.py", line 170, in test_char
    self.assertEqual(ts.T_CHAR, "c")
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
AssertionError: '\x00' != 'c'
- �
+ c

that sounds like my fault, I'll take a look later today

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL9 LTO 3.x has failed when building commit 979bf24.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1587/builds/864) and take a look at the build logs.
4. Check if the failure is related to this commit (979bf24) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1587/builds/864

Failed tests:

• test_capi

Failed subtests:

• test_char - test.test_capi.test_structmembers.ReadWriteTests_NewAPI.test_char
• test_char - test.test_capi.test_structmembers.ReadWriteTests_OldAPI.test_char

Summary of the results of the build (if available):

==

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.lto/build/Lib/test/test_capi/test_structmembers.py", line 170, in test_char
    self.assertEqual(ts.T_CHAR, "c")
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
AssertionError: '\x00' != 'c'
- �
+ c

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL8 3.x has failed when building commit 979bf24.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/509/builds/7885) and take a look at the build logs.
4. Check if the failure is related to this commit (979bf24) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/509/builds/7885

Failed tests:

• test_capi

Failed subtests:

• test_char - test.test_capi.test_structmembers.ReadWriteTests_NewAPI.test_char
• test_char - test.test_capi.test_structmembers.ReadWriteTests_OldAPI.test_char

Summary of the results of the build (if available):

==

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-rhel8-s390x/build/Lib/test/test_capi/test_structmembers.py", line 170, in test_char
    self.assertEqual(ts.T_CHAR, "c")
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
AssertionError: '\x00' != 'c'
- �
+ c

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL8 LTO + PGO 3.x has failed when building commit 979bf24.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/442/builds/7968) and take a look at the build logs.
4. Check if the failure is related to this commit (979bf24) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/442/builds/7968

Failed tests:

• test_capi

Failed subtests:

• test_char - test.test_capi.test_structmembers.ReadWriteTests_NewAPI.test_char
• test_char - test.test_capi.test_structmembers.ReadWriteTests_OldAPI.test_char

Summary of the results of the build (if available):

==

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-rhel8-s390x.lto-pgo/build/Lib/test/test_capi/test_structmembers.py", line 170, in test_char
    self.assertEqual(ts.T_CHAR, "c")
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
AssertionError: '\x00' != 'c'
- �
+ c

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL8 LTO 3.x has failed when building commit 979bf24.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/567/builds/7887) and take a look at the build logs.
4. Check if the failure is related to this commit (979bf24) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/567/builds/7887

Failed tests:

• test_capi

Failed subtests:

• test_char - test.test_capi.test_structmembers.ReadWriteTests_NewAPI.test_char
• test_char - test.test_capi.test_structmembers.ReadWriteTests_OldAPI.test_char

Summary of the results of the build (if available):

==

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-rhel8-s390x.lto/build/Lib/test/test_capi/test_structmembers.py", line 170, in test_char
    self.assertEqual(ts.T_CHAR, "c")
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
AssertionError: '\x00' != 'c'
- �
+ c

@dpdani @colesbury: As you can see in previous comments, test_capi fails on multiple buildbots. I'm not sure why.

mm, I can't seem to reproduce it locally. maybe some platform compatibility issue?
bedevere reported here failures only for s390x, but I don't have easy access to that.

what do you usually do in these situations?

Let's skip the test on s390x for now while we investigate.

I'm not sure what's causing the failure. The only thing I can think of is that char is unsigned on s390x (vs. signed on GCC x86-64 and arm64).

#127577 should fix it. The problem was that the 'C' format code expects an int as the destination (not a char):

https://docs.python.org/3/c-api/arg.html#c.Py_BuildValue

This happens to work out okay on little endian systems because the non-zero part of the value is at the same place (the first byte), and it just writes some zeros to the struct's padding bytes. On big endian systems, like s390x, it would write zero to char_member and some non-zero value to parts of the padding bytes.

I followed the instructions on https://docs.gitlab.com/omnibus/development/s390x.html to debug s390x using Docker and QEMU. I had to slightly modify their commands: in particular, I had to specify --platform linux/s390x:

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker run --platform linux/s390x -v $(pwd):/cpython --rm -it s390x/ubuntu

oops, my bad

thank you for all the details 🙏