bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh by paulmon · Pull Request #13454 · python/cpython · GitHub

When running the buildbot for Windows ARM32 test.pythoninfo must be run remotely.
Also seperating deploy step from both pythoninfo and test.bat

Requires buildmaster config change to work:
python/buildmaster-config#91

@zooba @zware

https://bugs.python.org/issue36511

The debug part of the test pass should pass locally with these fixes, except for test_datetime.
See this PR for test_datetime: #13073
I am re-running test.bat now with these fixes, which takes more than an hour.
The last pass took 4 hours and 23 minutes on my secondary ARM test device, but on the buildbot ARM test device it's been taking about an hour and a half.
I still need to try the WindowsArm32ReleaseBuild steps, but expect the same results.

Would prefer a tighter skip on the tests, and deferring to @zware on the buildbot scripts (as with the build config change).

This is certainly not a general issue - can we scope it down to the platform(s) where it crashes?

I will scope it down.

I think it's limited to UCRT. The problem is that SSL has FILE* from the ucrtbase.dll table and python has a FILE* from the ucrtbased.dll table.

I think we could limit it to win32+arm or platform.win32_is_iot( ). Unless there's a way to check if python is using UCRT? I couldn't find one.

Oh, it's because of mixing FILE* across debug/release builds? In that case we scope it down to Windows and enhance the skip message to say "Avoid mixing debug/release CRT on Windows"

Done.

Enabling is_python_build( ) to return true on the test target uncovered more failing tests.
There is still one failure in test_pyexpat that I will be investigating next.

======================================================================
FAIL: test_exception (test.test_pyexpat.HandlerExceptionTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\py\lib\test\test_pyexpat.py", line 453, in test_exception
    parser.Parse(b"<a><b><c/></b></a>", 1)
RuntimeError: a

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\py\lib\test\test_pyexpat.py", line 469, in test_exception
    self.assertIn('call_with_frame("StartElement"', entries[1][3])
AssertionError: 'call_with_frame("StartElement"' not found in ''

----------------------------------------------------------------------

I simplified the deployment script to copy entire directories instead of picking files, except for PCBuild and Modules which have many files that aren't needed for the test run. This should make the script more resilient to test changes.

I think the Windows ARM32 test pass is green now when I run the scripts manually, if #13073 is included.

I'd rather fix this than skip it, but I haven't figured out where to start looking yet.

@tiran can you please review?
Some of these changes are required to enable the Windows arm32 buildbot to start working.
Thanks!

@paulmon I'm happy to merge without Christian's review once the updates I suggested in my last one are done.

Co-Authored-By: Steve Dower <steve.dower@microsoft.com>

@zooba I merged your suggested changes. I think it's ready

@paulmon The Py_DEBUG_WIN32 change needs the references to that variable updated as well. I didn't go through and suggest a change on each one.

@paulmon The Py_DEBUG_WIN32 change needs the references to that variable updated as well. I didn't go through and suggest a change on each one.

I'm not sure I'm following you. Are suggesting adding a check for arm/arm64 to Py_DEBUG_WIN32? That would be incorrect for Windows IoT Core x86/x64 which also use UCRT for everything.

@paulmon The Py_DEBUG_WIN32 change needs the references to that variable updated as well. I didn't go through and suggest a change on each one.

I'm not sure I'm following you. Are suggesting adding a check for arm/arm64 to Py_DEBUG_WIN32? That would be incorrect for Windows IoT Core x86/x64 which also use UCRT for everything.

No, I'm saying elsewhere you are referring to Py_DEBUG_Win32 which is now undefined. They need to be changed to Py_DEBUG_WIN32 as well

No, I'm saying elsewhere you are referring to Py_DEBUG_Win32 which is now undefined. They need to be changed to Py_DEBUG_WIN32 as well

Thanks. I noticed the whitespace change but somehow didn't see the uppercase change.

@zooba Thanks for the help
I fixed test_ssl and test_math so they pass on my dev machine.
All of the PR checks pass now.

Thanks @paulmon for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

…bot (pythonGH-13454) (cherry picked from commit f355069) Co-authored-by: Paul Monson <paulmon@users.noreply.github.com>

GH-14244 is a backport of this pull request to the 3.8 branch.

@paulmon Seems like it still fails: https://buildbot.python.org/all/#/builders/197/builds/338

@paulmon Seems like it still fails: https://buildbot.python.org/all/#/builders/197/builds/338

The raspberry pi was in a bad state. I have restarted it

Also, ssh.exe and scp.exe somehow don't exist when running under the buildbot, which is breaking the script.

@paulmon Different PATH setting? Or perhaps it's running with different file redirection settings (as a 32-bit app, maybe)?

@paulmon Different PATH setting? Or perhaps it's running with different file redirection settings (as a 32-bit app, maybe)?

The PATH is echoed at the top of the log and includes the PATH to ssh.
I hadn't thought of WoW as a possible cause. I installed from the link on python.org so I got the 32-bit version of python. I'll update to the 64-bit python and see if that helps. There isn't a SysWOW64 version of ssh.exe, so that could be it.

* master: (599 commits) Docs: Improved phrasing (pythonGH-14069) Remove redundant if check from optional argument function in argparse. (pythonGH-8766) bpo-37289: Add a test for if with ifexpr in the peephole optimiser to detect regressions (pythonGH-14127) Update What's New in Python 3.9 (pythonGH-14253) bpo-36511: Improve ARM32 buildbot scripts (pythonGH-14251) bpo-37151: remove _PyCFunction_FastCallDict (pythonGH-14269) Fix typo, 'widger' -> 'widget', in idlelib/tree.py (pythonGH-14263) Fix bpo number in News file. (pythonGH-14260) bpo-37342: Fix the incorrect nb_index's type in typeobj documentation (pythonGH-14241) Update What's New in Python 3.8 (pythonGH-14239) bpo-36710: Use tstate in pylifecycle.c (pythonGH-14249) Add missing single quote in io.TextIOWrapper.reconfigure documentation (pythonGH-14246) bpo-36511: Add buildbot scripts and fix tests for Windows ARM32 buildbot (pythonGH-13454) bpo-37333: Ensure IncludeTkinter has a value (pythonGH-14240) bpo-37331: Clarify format of socket handler messages in the documentation. (pythonGH-14234) bpo-37258: Not a bug, but added a unit test and updated documentation. (pythonGH-14229) bpo-36710: Remove PyImport_Cleanup() function (pythonGH-14221) Fix name of '\0'. (pythonGH-14222) bpo-36710: Add tstate parameter in import.c (pythonGH-14218) Document typing.ForwardRef (pythonGH-14216) ...

…bot (pythonGH-13454)

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh

1d33fe6

the-knights-who-say-ni added the CLA signed label May 20, 2019

bedevere-bot added the awaiting review label May 20, 2019

paulmon mentioned this pull request May 20, 2019

bpo-36511: Fix Windows arm32 buildbot, pythoninfo, scp, and ssh python/buildmaster-config#91

Merged

Paul Monson added 2 commits May 21, 2019 12:07

add +d to match test.bat

3aadf52

100% test pass when manually running test.bat

7f17d29

fix file destination

9aaa2c4

zooba reviewed May 22, 2019

View reviewed changes

Paul Monson added 6 commits May 23, 2019 11:22

only skip ssl tests on Windows IoT

3f38d18

Avoid mixing debug/release CRT on Windows

4e147de

test for win32

9819647

propagate exit code

365701c

fix 2 more exit 0 calls

69c40d7

fixes for test_regrtest

086a1b9

Paul Monson added 5 commits May 24, 2019 09:35

merge upstream/master

042e0e9

add tools/parser for test_tools

e230795

copy entire Tools directory

1a1f12d

skip check on Windows ARM32

29601d8

restore 64-bit check in test_regrtest

786194d

paulmon commented May 24, 2019

View reviewed changes

zooba added skip news needs backport to 3.8 labels Jun 11, 2019

merge upstream/master

cd2d5c5

paulmon requested a review from tiran as a code owner June 11, 2019 23:08

zooba reviewed Jun 12, 2019

View reviewed changes

Comment thread Lib/test/test_ssl.py Outdated

Comment thread Lib/test/test_math.py Outdated

paulmon and others added 2 commits June 19, 2019 11:16

Update Lib/test/test_math.py

f5c917a

Co-Authored-By: Steve Dower <steve.dower@microsoft.com>

Update Lib/test/test_ssl.py

e2fb14b

Co-Authored-By: Steve Dower <steve.dower@microsoft.com>

Merge remote-tracking branch 'upstream/master' into arm32_pythoninfo

ce3dfbc

Paul Monson added 2 commits June 19, 2019 11:45

fix mismatched paren

5c355ab

fix Py_DEBUG_WIN32 spelling

f1ce1bf

zooba merged commit f355069 into python:master Jun 19, 2019

bedevere-bot removed the awaiting review label Jun 19, 2019

bedevere-bot removed the needs backport to 3.8 label Jun 19, 2019

paulmon mentioned this pull request Jun 19, 2019

bpo-36511 add arm32 buildbot diagnostic messages #14251

Merged

paulmon deleted the arm32_pythoninfo branch July 17, 2019 20:30

lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019

bpo-36511: Add buildbot scripts and fix tests for Windows ARM32 build…

cd062ef

…bot (pythonGH-13454)

DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020

bpo-36511: Add buildbot scripts and fix tests for Windows ARM32 build…

a739c78

…bot (pythonGH-13454)

paulmon mannequin mentioned this pull request May 20, 2022

Add Windows ARM32 buildbot #80692

Closed

                           self.check_traceback_entry(entries[2],
                                                      "test_pyexpat.py", "StartElementHandler")
-                          if sysconfig.is_python_build():
+                          if sysconfig.is_python_build() and not (sys.platform == 'win32' and platform.machine() == 'ARM'):

		ctx.load_verify_locations(cadata=b"broken")


		@unittest.skipIf(Py_DEBUG, "Crashes on debug python builds")

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh by paulmon · Pull Request #13454 · python/cpython · GitHub

Uh oh!

Uh oh!

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh - #13454

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh#13454zooba merged 21 commits intopython:masterpython/cpython:masterfrom paulmon:arm32_pythoninfopaulmon/cpython:arm32_pythoninfoCopy head branch name to clipboard

Conversation

paulmon commented May 20, 2019 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

paulmon commented May 22, 2019

Uh oh!

zooba left a comment

Choose a reason for hiding this comment

Uh oh!

zooba May 22, 2019

Choose a reason for hiding this comment

Uh oh!

paulmon May 22, 2019

Choose a reason for hiding this comment

Uh oh!

zooba May 23, 2019

Choose a reason for hiding this comment

Uh oh!

paulmon May 23, 2019

Choose a reason for hiding this comment

Uh oh!

paulmon commented May 24, 2019

Uh oh!

paulmon commented May 24, 2019

Uh oh!

paulmon May 24, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

paulmon commented Jun 13, 2019

Uh oh!

zooba commented Jun 19, 2019

Uh oh!

paulmon commented Jun 19, 2019

Uh oh!

zooba commented Jun 19, 2019

Uh oh!

paulmon commented Jun 19, 2019

Uh oh!

zooba commented Jun 19, 2019

Uh oh!

paulmon commented Jun 19, 2019

Uh oh!

paulmon commented Jun 19, 2019

Uh oh!

miss-islington commented Jun 19, 2019

Uh oh!

bedevere-bot commented Jun 19, 2019

Uh oh!

zooba commented Jun 19, 2019

Uh oh!

paulmon commented Jun 19, 2019

Uh oh!

paulmon commented Jun 19, 2019

Uh oh!

zooba commented Jun 19, 2019

Uh oh!

paulmon commented Jun 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh#13454
zooba merged 21 commits into
python:masterfrom
paulmon:arm32_pythoninfo

paulmon commented May 20, 2019 •
edited by bedevere-bot
Loading

paulmon commented Jun 19, 2019 •
edited
Loading