FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh by paulmon · Pull Request #13454 · python/cpython · GitHub

/ cpython Public

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh #13454

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zooba merged 21 commits into from
Jun 19, 2019
Conversation 26 Commits 21 Checks 0 Files changed
Merged

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh #13454

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
1d33fe6
bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh
May 20, 2019
3aadf52
add +d to match test.bat
May 21, 2019
7f17d29
100% test pass when manually running test.bat
May 22, 2019
9aaa2c4
fix file destination
May 22, 2019
3f38d18
only skip ssl tests on Windows IoT
May 23, 2019
4e147de
Avoid mixing debug/release CRT on Windows
May 23, 2019
9819647
test for win32
May 23, 2019
365701c
propagate exit code
May 23, 2019
69c40d7
fix 2 more exit 0 calls
May 23, 2019
086a1b9
fixes for test_regrtest
May 24, 2019
042e0e9
merge upstream/master
May 24, 2019
e230795
add tools/parser for test_tools
May 24, 2019
1a1f12d
copy entire Tools directory
May 24, 2019
29601d8
skip check on Windows ARM32
May 24, 2019
786194d
restore 64-bit check in test_regrtest
May 24, 2019
cd2d5c5
merge upstream/master
Jun 11, 2019
f5c917a
Update Lib/test/test_math.py
paulmon Jun 19, 2019
e2fb14b
Update Lib/test/test_ssl.py
paulmon Jun 19, 2019
ce3dfbc
Merge remote-tracking branch 'upstream/master' into arm32_pythoninfo
Jun 19, 2019
5c355ab
fix mismatched paren
Jun 19, 2019
f1ce1bf
fix Py_DEBUG_WIN32 spelling
Jun 19, 2019
File filter

Filter by extension

Filter by extension .bat  (3) .py  (5) All 2 file types selected
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
4 changes: 3 additions & 1 deletion Lib/test/support/__init__.py
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -485,7 +485,9 @@ def _is_gui_available():
if hasattr(_is_gui_available, 'result'):
return _is_gui_available.result
reason = None
if sys.platform.startswith('win'):
if sys.platform.startswith('win') and platform.win32_is_iot():
reason = "gui is not available on Windows IoT Core"
elif sys.platform.startswith('win'):
# if Python is running as a service (such as the buildbot service),
# gui interaction may be disallowed
import ctypes
Expand Down
2 changes: 2 additions & 0 deletions Lib/test/test_math.py
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -468,6 +468,8 @@ def testCos(self):
self.assertRaises(ValueError, math.cos, NINF)
self.assertTrue(math.isnan(math.cos(NAN)))

@unittest.skipIf(sys.platform == 'win32' and platform.machine() in ('ARM', 'ARM64'),
"Windows UCRT is off by 2 ULP this test requires accuracy within 1 ULP")
def testCosh(self):
self.assertRaises(TypeError, math.cosh)
self.ftest('cosh(0)', math.cosh(0), 1)
Expand Down
3 changes: 2 additions & 1 deletion Lib/test/test_pyexpat.py
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

from io import BytesIO
import os
import platform
import sys
import sysconfig
import unittest
Expand Down Expand Up @@ -465,7 +466,7 @@ def test_exception(self):
"pyexpat.c", "StartElement")
self.check_traceback_entry(entries[2],
"test_pyexpat.py", "StartElementHandler")
if sysconfig.is_python_build():
if sysconfig.is_python_build() and not (sys.platform == 'win32' and platform.machine() == 'ARM'):

paulmon May 24, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I'd rather fix this than skip it, but I haven't figured out where to start looking yet.
self.assertIn('call_with_frame("StartElement"', entries[1][3])


Expand Down
4 changes: 4 additions & 0 deletions Lib/test/test_regrtest.py
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -618,6 +618,8 @@ def test_tools_buildbot_test(self):
test_args = ['--testdir=%s' % self.tmptestdir]
if platform.machine() == 'ARM64':
test_args.append('-arm64') # ARM 64-bit build
elif platform.machine() == 'ARM':
test_args.append('-arm32') # 32-bit ARM build
elif platform.architecture()[0] == '64bit':
test_args.append('-x64') # 64-bit build
if not Py_DEBUG:
Expand All @@ -633,6 +635,8 @@ def test_pcbuild_rt(self):
rt_args = ["-q"] # Quick, don't run tests twice
if platform.machine() == 'ARM64':
rt_args.append('-arm64') # ARM 64-bit build
elif platform.machine() == 'ARM':
rt_args.append('-arm32') # 32-bit ARM build
elif platform.architecture()[0] == '64bit':
rt_args.append('-x64') # 64-bit build
if Py_DEBUG:
Expand Down
9 changes: 9 additions & 0 deletions Lib/test/test_ssl.py
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@

from ssl import TLSVersion, _TLSContentType, _TLSMessageType, _TLSAlertType

Py_DEBUG = hasattr(sys, 'gettotalrefcount')
Py_DEBUG_WIN32 = Py_DEBUG and sys.platform == 'win32'

PROTOCOLS = sorted(ssl._PROTOCOL_NAMES)
HOST = support.HOST
IS_LIBRESSL = ssl.OPENSSL_VERSION.startswith('LibreSSL')
Expand Down Expand Up @@ -1347,6 +1350,7 @@ def test_load_verify_cadata(self):
ctx.load_verify_locations(cadata=b"broken")


@unittest.skipIf(Py_DEBUG_WIN32, "Avoid mixing debug/release CRT on Windows")
def test_load_dh_params(self):
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ctx.load_dh_params(DHFILE)
Expand Down Expand Up @@ -1665,6 +1669,7 @@ def test_str(self):
self.assertEqual(str(e), "foo")
self.assertEqual(e.errno, 1)

@unittest.skipIf(Py_DEBUG_WIN32, "Avoid mixing debug/release CRT on Windows")
def test_lib_reason(self):
# Test the library and reason attributes
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
Expand Down Expand Up @@ -3845,6 +3850,7 @@ def test_compression_disabled(self):
sni_name=hostname)
self.assertIs(stats['compression'], None)

@unittest.skipIf(Py_DEBUG_WIN32, "Avoid mixing debug/release CRT on Windows")
def test_dh_params(self):
# Check we can get a connection with ephemeral Diffie-Hellman
client_context, server_context, hostname = testing_context()
Expand Down Expand Up @@ -4440,6 +4446,7 @@ def keylog_lines(self, fname=support.TESTFN):
return len(list(f))

@requires_keylog
@unittest.skipIf(Py_DEBUG_WIN32, "Avoid mixing debug/release CRT on Windows")
def test_keylog_defaults(self):
self.addCleanup(support.unlink, support.TESTFN)
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
Expand All @@ -4463,6 +4470,7 @@ def test_keylog_defaults(self):
ctx.keylog_filename = 1

@requires_keylog
@unittest.skipIf(Py_DEBUG_WIN32, "Avoid mixing debug/release CRT on Windows")
def test_keylog_filename(self):
self.addCleanup(support.unlink, support.TESTFN)
client_context, server_context, hostname = testing_context()
Expand Down Expand Up @@ -4500,6 +4508,7 @@ def test_keylog_filename(self):
@requires_keylog
@unittest.skipIf(sys.flags.ignore_environment,
"test is not compatible with ignore_environment")
@unittest.skipIf(Py_DEBUG_WIN32, "Avoid mixing debug/release CRT on Windows")
def test_keylog_env(self):
self.addCleanup(support.unlink, support.TESTFN)
with unittest.mock.patch.dict(os.environ):
Expand Down
48 changes: 48 additions & 0 deletions Tools/buildbot/remoteDeploy.bat
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
@echo off
rem Used by the buildbot "remotedeploy" step.
setlocal

set here=%~dp0
set arm32_ssh=

:CheckOpts
if "%1"=="-arm32" (set arm32_ssh=true) & shift & goto CheckOpts
if NOT "%1"=="" (echo unrecognized option %1) & goto Arm32SshHelp

if "%arm32_ssh%"=="true" goto :Arm32Ssh

:Arm32Ssh
if "%SSH_SERVER%"=="" goto :Arm32SshHelp
if "%SSH%"=="" if EXIST %WINDIR%\System32\OpenSSH\ssh.exe (set SSH=%WINDIR%\System32\OpenSSH\ssh.exe)
if "%SCP%"=="" if EXIST %WINDIR%\System32\OpenSSH\scp.exe (set SCP=%WINDIR%\System32\OpenSSH\scp.exe)
echo SSH = %SSH%
echo SCP = %SCP%
if "%PYTHON_SOURCE%"=="" (set PYTHON_SOURCE=%here%..\..\)
if "%REMOTE_PYTHON_DIR%"=="" (set REMOTE_PYTHON_DIR=C:\python\)
if NOT "%REMOTE_PYTHON_DIR:~-1,1%"=="\" (set REMOTE_PYTHON_DIR=%REMOTE_PYTHON_DIR%\)
%SSH% %SSH_SERVER% "if EXIST %REMOTE_PYTHON_DIR% (rd %REMOTE_PYTHON_DIR% /s/q)"
%SSH% %SSH_SERVER% "md %REMOTE_PYTHON_DIR%PCBuild\arm32"
%SSH% %SSH_SERVER% "md %REMOTE_PYTHON_DIR%temp"
%SSH% %SSH_SERVER% "md %REMOTE_PYTHON_DIR%Modules"
%SSH% %SSH_SERVER% "md %REMOTE_PYTHON_DIR%PC"
for /f "USEBACKQ" %%i in (`dir PCbuild\*.bat /b`) do @%SCP% PCBuild\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild"
for /f "USEBACKQ" %%i in (`dir PCbuild\*.py /b`) do @%SCP% PCBuild\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild"
for /f "USEBACKQ" %%i in (`dir PCbuild\arm32\*.exe /b`) do @%SCP% PCBuild\arm32\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild\arm32"
for /f "USEBACKQ" %%i in (`dir PCbuild\arm32\*.pyd /b`) do @%SCP% PCBuild\arm32\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild\arm32"
for /f "USEBACKQ" %%i in (`dir PCbuild\arm32\*.dll /b`) do @%SCP% PCBuild\arm32\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild\arm32"
%SCP% -r "%PYTHON_SOURCE%Include" "%SSH_SERVER%:%REMOTE_PYTHON_DIR%Include"
%SCP% -r "%PYTHON_SOURCE%Lib" "%SSH_SERVER%:%REMOTE_PYTHON_DIR%Lib"
%SCP% -r "%PYTHON_SOURCE%Tools" "%SSH_SERVER%:%REMOTE_PYTHON_DIR%Tools"
%SCP% "%PYTHON_SOURCE%Modules\Setup" "%SSH_SERVER%:%REMOTE_PYTHON_DIR%Modules"
%SCP% "%PYTHON_SOURCE%PC\pyconfig.h" "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PC"

exit /b %ERRORLEVEL%

:Arm32SshHelp
echo SSH_SERVER environment variable must be set to administrator@[ip address]
echo where [ip address] is the address of a Windows IoT Core ARM32 device.
echo.
echo The test worker should have the SSH agent running.
echo Also a key must be created with ssh-keygen and added to both the buildbot worker machine
echo and the ARM32 worker device: see https://docs.microsoft.com/en-us/windows/iot-core/connect-your-device/ssh
exit /b 127
34 changes: 34 additions & 0 deletions Tools/buildbot/remotePythonInfo.bat
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
@echo off
rem Used by the buildbot "remotedeploy" step.
setlocal

set here=%~dp0
set arm32_ssh=
set suffix=_d
if "%REMOTE_PYTHON_DIR%"=="" (set REMOTE_PYTHON_DIR=C:\python\)
if NOT "%REMOTE_PYTHON_DIR:~-1,1%"=="\" (set REMOTE_PYTHON_DIR=%REMOTE_PYTHON_DIR%\)

:CheckOpts
if "%1"=="-arm32" (set arm32_ssh=true) & (set prefix=%REMOTE_PYTHON_DIR%pcbuild\arm32) & shift & goto CheckOpts
if "%1"=="-d" (set suffix=_d) & shift & goto CheckOpts
if "%1"=="+d" (set suffix=) & shift & goto CheckOpts
if NOT "%1"=="" (echo unrecognized option %1) & goto Arm32SshHelp

if "%arm32_ssh%"=="true" goto :Arm32Ssh

:Arm32Ssh
if "%SSH_SERVER%"=="" goto :Arm32SshHelp
if "%SSH%"=="" if EXIST %WINDIR%\System32\OpenSSH\ssh.exe (set SSH=%WINDIR%\System32\OpenSSH\ssh.exe)
set PYTHON_EXE=%prefix%\python%suffix%.exe
echo on
%SSH% %SSH_SERVER% %PYTHON_EXE% -m test.pythoninfo
exit /b %ERRORLEVEL%

:Arm32SshHelp
echo SSH_SERVER environment variable must be set to administrator@[ip address]
echo where [ip address] is the address of a Windows IoT Core ARM32 device.
echo.
echo The test worker should have the SSH agent running.
echo Also a key must be created with ssh-keygen and added to both the buildbot worker machine
echo and the ARM32 worker device: see https://docs.microsoft.com/en-us/windows/iot-core/connect-your-device/ssh
exit /b 127
16 changes: 5 additions & 11 deletions Tools/buildbot/test.bat
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,10 @@ if "%1"=="+d" (set rt_opts=%rt_opts:-d=%) & shift & goto CheckOpts
if "%1"=="+q" (set rt_opts=%rt_opts:-q=%) & shift & goto CheckOpts
if NOT "%1"=="" (set regrtest_args=%regrtest_args% %1) & shift & goto CheckOpts

if "%PROCESSOR_ARCHITECTURE%"=="ARM" if "%arm32_ssh%"=="true" goto NativeExecution
if "%arm32_ssh%"=="true" goto :Arm32Ssh

:NativeExecution
call "%here%..\..\PCbuild\rt.bat" %rt_opts% -uall -rwW --slowest --timeout=1200 --fail-env-changed %regrtest_args%
exit /b %ERRORLEVEL%

Expand All @@ -28,20 +30,12 @@ set dashU=-unetwork,decimal,subprocess,urlfetch,tzdata
if "%SSH_SERVER%"=="" goto :Arm32SshHelp
if "%PYTHON_SOURCE%"=="" (set PYTHON_SOURCE=%here%..\..\)
if "%REMOTE_PYTHON_DIR%"=="" (set REMOTE_PYTHON_DIR=C:\python\)
if NOT "%REMOTE_PYTHON_DIR:~-1,1%"=="\" (set REMOTE_PYTHON_DIR=%REMOTE_PYTHON_DIR%\)
if "%SSH%"=="" if EXIST %WINDIR%\System32\OpenSSH\ssh.exe (set SSH=%WINDIR%\System32\OpenSSH\ssh.exe)
set TEMP_ARGS=--temp %REMOTE_PYTHON_DIR%temp
ssh %SSH_SERVER% "if EXIST %REMOTE_PYTHON_DIR% (rd %REMOTE_PYTHON_DIR% /s/q)"
ssh %SSH_SERVER% "md %REMOTE_PYTHON_DIR%PCBuild\arm32"
ssh %SSH_SERVER% "md %REMOTE_PYTHON_DIR%temp"
for /f "USEBACKQ" %%i in (`dir PCbuild\*.bat /b`) do @scp PCBuild\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild"
for /f "USEBACKQ" %%i in (`dir PCbuild\*.py /b`) do @scp PCBuild\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild"
for /f "USEBACKQ" %%i in (`dir PCbuild\arm32\*.exe /b`) do @scp PCBuild\arm32\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild\arm32"
for /f "USEBACKQ" %%i in (`dir PCbuild\arm32\*.pyd /b`) do @scp PCBuild\arm32\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild\arm32"
for /f "USEBACKQ" %%i in (`dir PCbuild\arm32\*.dll /b`) do @scp PCBuild\arm32\%%i "%SSH_SERVER%:%REMOTE_PYTHON_DIR%PCBuild\arm32"
scp -r "%PYTHON_SOURCE%Include" "%SSH_SERVER%:%REMOTE_PYTHON_DIR%Include"
scp -r "%PYTHON_SOURCE%Lib" "%SSH_SERVER%:%REMOTE_PYTHON_DIR%Lib"

set rt_args=%rt_opts% %dashU% -rwW --slowest --timeout=1200 --fail-env-changed %regrtest_args% %TEMP_ARGS%
ssh %SSH_SERVER% "set TEMP=%REMOTE_PYTHON_DIR%temp& %REMOTE_PYTHON_DIR%PCbuild\rt.bat" %rt_args%
%SSH% %SSH_SERVER% "set TEMP=%REMOTE_PYTHON_DIR%temp& %REMOTE_PYTHON_DIR%PCbuild\rt.bat" %rt_args%
exit /b %ERRORLEVEL%

:Arm32SshHelp
Expand Down
Back | FazBrowse Home | New Git URL