gh-109276: libregrtest: WASM use filename for JSON - #109326
Closed
vstinner wants to merge 5 commits into
Closed
Conversation
Copy link
Copy Markdown
Member
Sorry, something went wrong.
vstinner
force-pushed
the
regrtest_json_file2
branch
from
359d425 to
c45529b
Compare
September 12, 2023 13:34
This was referenced Sep 12, 2023
Copy link
Copy Markdown
Member
Author
|
!buildbot wasm32 |
Sorry, something went wrong.
Copy link
Copy Markdown
|
🤖 New build scheduled with the buildbot fleet by @vstinner for commit c45529b 🤖 The command will test the builders whose names match following regular expression: wasm32 The builders matched are:
|
Sorry, something went wrong.
Copy link
Copy Markdown
Member
Author
|
Emscripten/WASI still use a file descriptor. It makes no sense to me :-( I'm not sure if buildbots got the right commit. |
Sorry, something went wrong.
On Emscripten and WASI platforms, libregrtest now uses a filename for the JSON file. Passing a file descriptor to a child process doesn't work on these platforms.
vstinner
force-pushed
the
regrtest_json_file2
branch
from
ee0cac8 to
9c052ba
Compare
September 12, 2023 13:57
Copy link
Copy Markdown
Member
Author
|
!buildbot wasm32 |
Sorry, something went wrong.
Copy link
Copy Markdown
|
🤖 New build scheduled with the buildbot fleet by @vstinner for commit 9c052ba 🤖 The command will test the builders whose names match following regular expression: wasm32 The builders matched are:
|
Sorry, something went wrong.
vstinner
marked this pull request as draft
September 12, 2023 14:08
Copy link
Copy Markdown
Member
Author
|
!buildbot wasm32 |
Sorry, something went wrong.
Copy link
Copy Markdown
|
🤖 New build scheduled with the buildbot fleet by @vstinner for commit 9c052ba 🤖 The command will test the builders whose names match following regular expression: wasm32 The builders matched are:
|
Sorry, something went wrong.
Copy link
Copy Markdown
Member
Author
|
wasm32-emscripten node (dynamic linking) PR failed with: _PYTHON_HOSTRUNNER='/opt/emsdk/node/current/bin/node --experimental-wasm-bigint' _PYTHON_PROJECT_BASE=/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.emscripten-node-dl/build/build_oot/host _PYTHON_HOST_PLATFORM=emscripten-wasm32 PYTHONPATH=/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.emscripten-node-dl/build/build_oot/host/build/lib.emscripten-wasm32-3.13:../../Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__emscripten_wasm32-emscripten ../build/python ../../Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=900 -j2 --junit-xml test-results.xml -j2
== CPython 3.13.0a0 (heads/refs/pull/109326/merge-dirty:a70b894cdf, Sep 12 2023, 15:11:12) [GCC 9.4.0]
== Linux-5.15.0-1019-azure-x86_64-with-glibc2.31 little-endian
== Python build: release
== cwd: /opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.emscripten-node-dl/build/build_oot/host/build/test_python_worker_432395æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Using random seed 96764921
0:00:00 load avg: 2.54 Run 463 tests in parallel using 2 worker processes (timeout: 15 min, worker timeout: 20 min)
main process is_emscripten: False
main process is_wasi: False
main process JSON_FILE_USE_FILENAME: False
main process is_emscripten: False
main process is_wasi: False
main process JSON_FILE_USE_FILENAME: False
main process: create TemporaryFile
main process json_type file: <class 'int'>
main process json_type: 5
main process: create TemporaryFile
main process json_type file: <class 'int'>
main process json_type: 6
0:00:03 load avg: 2.54 [ 1/463/1] test_optparse process crashed (Exit code 1)
worker: json_file type: <class 'int'>
worker: json_file: 5
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.emscripten-node-dl/build/Lib/test/libregrtest/worker.py", line 131, in <module>
main()
File "/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.emscripten-node-dl/build/Lib/test/libregrtest/worker.py", line 127, in main
worker_process(worker_json)
File "/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.emscripten-node-dl/build/Lib/test/libregrtest/worker.py", line 110, in worker_process
with open(json_file, 'w', encoding='utf-8') as fp:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 8] Bad file descriptor
wasm32-wasi PR failed with: _PYTHON_HOSTRUNNER='wasmtime run --env PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib --mapdir /::../.. --' _PYTHON_PROJECT_BASE=/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host _PYTHON_HOST_PLATFORM=wasi-wasm32 PYTHONPATH=/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/lib.wasi-wasm32-3.13:../../Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__wasi_wasm32-wasi ../build/python ../../Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=900 -j2 --junit-xml test-results.xml -j2
== CPython 3.13.0a0 (heads/refs/pull/109326/merge-dirty:a70b894cdf, Sep 12 2023, 15:14:49) [GCC 9.4.0]
== Linux-5.15.0-1019-azure-x86_64-with-glibc2.31 little-endian
== Python build: release
== cwd: /opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_worker_458151æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Using random seed 76879218
0:00:00 load avg: 2.51 Run 463 tests in parallel using 2 worker processes (timeout: 15 min, worker timeout: 20 min)
main process is_emscripten: False
main process is_wasi: False
main process JSON_FILE_USE_FILENAME: False
main process is_emscripten: False
main process is_wasi: False
main process JSON_FILE_USE_FILENAME: False
main process: create TemporaryFile
main process json_type file: <class 'int'>
main process json_type: 5
main process: create TemporaryFile
main process json_type file: <class 'int'>
main process json_type: 6
0:00:00 load avg: 2.47 [ 1/463/1] test_embed process crashed (Exit code 1)
/Lib/test/support/os_helper.py:531: RuntimeWarning: tests may fail, unable to create temporary directory '/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_48104æ': [Errno 44] No such file or directory: '/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_48104æ'
with temp_dir(path=name, quiet=quiet) as temp_path:
/Lib/test/support/os_helper.py:532: RuntimeWarning: tests may fail, unable to change the current working directory to '/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_48104æ': [Errno 44] No such file or directory: '/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_48104æ'
with change_cwd(temp_path, quiet=quiet) as cwd_dir:
worker: json_file type: <class 'int'>
worker: json_file: 6
test_embed skipped -- test module requires subprocess
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/Lib/test/libregrtest/worker.py", line 131, in <module>
main()
File "/Lib/test/libregrtest/worker.py", line 127, in main
worker_process(worker_json)
File "/Lib/test/libregrtest/worker.py", line 110, in worker_process
with open(json_file, 'w', encoding='utf-8') as fp:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 8] Bad file descriptor
|
Sorry, something went wrong.
Copy link
Copy Markdown
Member
Author
|
I just merged PR #109337 which collect more data in test.support and test.support sub-modules. |
Sorry, something went wrong.
Copy link
Copy Markdown
Member
Author
wasm32-emscripten node (dynamic linking) 3.x: test.pythoninfo, support.is_emscripten: True and sys.platform: emscripten as expected. support.has_fork_support: False support.has_socket_support: False support.has_strftime_extensions: False support.has_subprocess_support: False support.is_android: False support.is_emscripten: True support.is_jython: False support.is_wasi: False support_threading_helper.can_start_thread: False sys.platform: emscripten wasm32-emscripten node (pthreads) 3.x: test.pythoninfo, support.is_emscripten: True and sys.platform: emscripten as expected support.has_fork_support: False support.has_socket_support: False support.has_strftime_extensions: False support.has_subprocess_support: False support.is_android: False support.is_emscripten: True support.is_jython: False support.is_wasi: False support_threading_helper.can_start_thread: True sys.platform: emscripten wasm32-wasi 3.x: test.pythoninfo, support.is_wasi: True and sys.platform: wasi as expected support.has_fork_support: False support.has_socket_support: False support.has_strftime_extensions: True support.has_subprocess_support: False support.is_android: False support.is_emscripten: False support.is_jython: False support.is_wasi: True support_threading_helper.can_start_thread: False sys.platform: wasi |
Sorry, something went wrong.
Copy link
Copy Markdown
Member
Author
|
!buildbot wasm32 |
Sorry, something went wrong.
Copy link
Copy Markdown
|
🤖 New build scheduled with the buildbot fleet by @vstinner for commit c436370 🤖 The command will test the builders whose names match following regular expression: wasm32 The builders matched are:
|
Sorry, something went wrong.
Copy link
Copy Markdown
Member
Author
|
!buildbot wasm32 |
Sorry, something went wrong.
Copy link
Copy Markdown
|
🤖 New build scheduled with the buildbot fleet by @vstinner for commit 4c8d5c8 🤖 The command will test the builders whose names match following regular expression: wasm32 The builders matched are:
|
Sorry, something went wrong.
Copy link
Copy Markdown
Member
Author
|
!buildbot wasm32 |
Sorry, something went wrong.
Copy link
Copy Markdown
|
🤖 New build scheduled with the buildbot fleet by @vstinner for commit 4e2cfab 🤖 The command will test the builders whose names match following regular expression: wasm32 The builders matched are:
|
Sorry, something went wrong.
Copy link
Copy Markdown
Member
Author
|
Ok, I wrote a clean fix from this debug PR: PR #109340. |
Sorry, something went wrong.
Copy link
Copy Markdown
Member
Author
|
wasm32-wasi PR failed: the WASI test worker process doesn't seem to have access to /tmp/test_python_json_7sbyggem of the Linuxparent process :-( _PYTHON_HOSTRUNNER='wasmtime run --env PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib --mapdir /::../.. --' _PYTHON_PROJECT_BASE=/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host _PYTHON_HOST_PLATFORM=wasi-wasm32 PYTHONPATH=/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/lib.wasi-wasm32-3.13:../../Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__wasi_wasm32-wasi ../build/python ../../Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=900 -j2 --junit-xml test-results.xml -j2
1118966 JsonFileType: int | str
main process start sys.platform: linux
main process start is_emscripten: False
main process start is_wasi: False
main process python_cmd: ('wasmtime', 'run', '--env', 'PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib', '--mapdir', '/::../..', '--', 'python.wasm')
== CPython 3.13.0a0 (heads/refs/pull/109326/merge-dirty:0540fe293c, Sep 12 2023, 17:39:37) [GCC 9.4.0]
== Linux-5.15.0-1019-azure-x86_64-with-glibc2.31 little-endian
== Python build: release
== cwd: /opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_worker_1118966æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Using random seed 30180538
0:00:00 load avg: 1.66 Run 463 tests in parallel using 2 worker processes (timeout: 15 min, worker timeout: 20 min)
main process: json_file_use_filename? ('wasmtime', 'run', '--env', 'PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib', '--mapdir', '/::../..', '--', 'python.wasm')
main process: json_file_use_filename? ('wasmtime', 'run', '--env', 'PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib', '--mapdir', '/::../..', '--', 'python.wasm')
main process: create NamedTemporaryFile /tmp/test_python_json_7sbyggem
main process json_file type: <class 'str'>
main process json_file: /tmp/test_python_json_7sbyggem
main process: create NamedTemporaryFile /tmp/test_python_json_yq7jvz76
main process json_file type: <class 'str'>
main process json_file: /tmp/test_python_json_yq7jvz76
main process executable: ('wasmtime', 'run', '--env', 'PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib', '--mapdir', '/::../..', '--', 'python.wasm')
main process worker cmd: ['wasmtime', 'run', '--env', 'PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib', '--mapdir', '/::../..', '--', 'python.wasm', '-bb', '-Wdefault', '-u', '-m', 'test.libregrtest.worker', '{"tests": ["test_bigaddrspace"], "fail_fast": false, "fail_env_changed": true, "match_tests": null, "ignore_tests": null, "match_tests_dict": null, "rerun": null, "forever": false, "pgo": false, "pgo_extended": false, "output_on_failure": true, "timeout": 900.0, "verbose": 0, "quiet": false, "hunt_refleak": null, "test_dir": null, "use_junit": true, "memory_limit": null, "gc_threshold": null, "use_resources": ["audio", "curses", "largefile", "network", "decimal", "cpu", "subprocess", "urlfetch", "gui", "walltime"], "python_cmd": ["wasmtime", "run", "--env", "PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib", "--mapdir", "/::../..", "--", "python.wasm"], "randomize": true, "random_seed": 30180538, "json_file": "/tmp/test_python_json_7sbyggem", "__runtests__": true}']
create_worker_process() json_file: filename
main process executable: ('wasmtime', 'run', '--env', 'PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib', '--mapdir', '/::../..', '--', 'python.wasm')
main process worker cmd: ['wasmtime', 'run', '--env', 'PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib', '--mapdir', '/::../..', '--', 'python.wasm', '-bb', '-Wdefault', '-u', '-m', 'test.libregrtest.worker', '{"tests": ["test_venv"], "fail_fast": false, "fail_env_changed": true, "match_tests": null, "ignore_tests": null, "match_tests_dict": null, "rerun": null, "forever": false, "pgo": false, "pgo_extended": false, "output_on_failure": true, "timeout": 900.0, "verbose": 0, "quiet": false, "hunt_refleak": null, "test_dir": null, "use_junit": true, "memory_limit": null, "gc_threshold": null, "use_resources": ["audio", "curses", "largefile", "network", "decimal", "cpu", "subprocess", "urlfetch", "gui", "walltime"], "python_cmd": ["wasmtime", "run", "--env", "PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib", "--mapdir", "/::../..", "--", "python.wasm"], "randomize": true, "random_seed": 30180538, "json_file": "/tmp/test_python_json_yq7jvz76", "__runtests__": true}']
create_worker_process() json_file: filename
0:00:00 load avg: 1.66 [ 1/463/1] test_bigaddrspace process crashed (Exit code 1)
42 JsonFileType: int | str
worker process sys.platform: wasi
worker process is_emscripten: False
worker process is_wasi: True
/Lib/test/support/os_helper.py:531: RuntimeWarning: tests may fail, unable to create temporary directory '/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_230432æ': [Errno 44] No such file or directory: '/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_230432æ'
with temp_dir(path=name, quiet=quiet) as temp_path:
/Lib/test/support/os_helper.py:532: RuntimeWarning: tests may fail, unable to change the current working directory to '/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_230432æ': [Errno 44] No such file or directory: '/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_230432æ'
with change_cwd(temp_path, quiet=quiet) as cwd_dir:
worker: json_file type: <class 'str'>
worker: json_file: /tmp/test_python_json_7sbyggem
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/Lib/test/libregrtest/worker.py", line 143, in <module>
main()
File "/Lib/test/libregrtest/worker.py", line 139, in main
worker_process(worker_json)
File "/Lib/test/libregrtest/worker.py", line 118, in worker_process
with open(json_file, 'w', encoding='utf-8') as fp:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 44] No such file or directory: '/tmp/test_python_json_7sbyggem'
Kill <WorkerThread #1 running test=test_venv pid=1118986 time=423 ms> process group
|
Sorry, something went wrong.
Copy link
Copy Markdown
Member
Author
I merged this PR instead of this DEBUG PR. |
Sorry, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On Emscripten and WASI platforms, libregrtest now uses a filename for the JSON file. Passing a file descriptor to a child process doesn't work on these platforms.