gh-126925: Modify how iOS test results are gathered - #127592
Merged
freakboy3742 merged 12 commits intoDec 9, 2024
Merged
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
freakboy3742
requested review from
FFY00,
ericsnowcurrently and
erlend-aasland
as code owners
December 4, 2024 07:17
Copy link
Copy Markdown
Contributor
Author
|
!buildbot iOS |
Sorry, something went wrong.
Copy link
Copy Markdown
|
🤖 New build scheduled with the buildbot fleet by @freakboy3742 for commit 60584e0 🤖 The command will test the builders whose names match following regular expression: iOS The builders matched are:
|
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
Author
|
Moving to draft because of the test failures on macOS buils |
Sorry, something went wrong.
freakboy3742
marked this pull request as draft
December 4, 2024 07:34
Copy link
Copy Markdown
Contributor
Author
|
!buildbot iOS |
Sorry, something went wrong.
Copy link
Copy Markdown
|
🤖 New build scheduled with the buildbot fleet by @freakboy3742 for commit 89bb435 🤖 The command will test the builders whose names match following regular expression: iOS The builders matched are:
|
Sorry, something went wrong.
freakboy3742
marked this pull request as ready for review
December 5, 2024 06:38
Copy link
Copy Markdown
Contributor
Author
|
I've now resolved the testing issue; I've also added some documentation for the new capabilities. |
Sorry, something went wrong.
Copy link
Copy Markdown
|
Thanks @freakboy3742 for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13. |
Sorry, something went wrong.
Copy link
Copy Markdown
|
Sorry, @freakboy3742, I could not cleanly backport this to 3.13 due to a conflict. cherry_picker 2041a95e68ebf6d13f867e214ada28affa830669 3.13 |
Sorry, something went wrong.
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Dec 9, 2024
…honGH-127592) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95) Co-authored-by: Russell Keith-Magee <russell@keith-magee.com>
Copy link
Copy Markdown
|
GH-127754 is a backport of this pull request to the 3.13 branch. |
Sorry, something went wrong.
freakboy3742
added a commit
that referenced
this pull request
Dec 9, 2024
#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Dec 13, 2024
…onGH-127592) (python#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Dec 13, 2024
…honGH-127592) (python#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Dec 13, 2024
…honGH-127592) (python#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Dec 13, 2024
…honGH-127592) (python#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Dec 13, 2024
) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite.
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Dec 13, 2024
…honGH-127592) (python#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
srinivasreddy
pushed a commit
to srinivasreddy/cpython
that referenced
this pull request
Jan 8, 2025
) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite.
This was referenced Mar 12, 2025
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Feb 3, 2026
…honGH-127592) (python#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Feb 3, 2026
…honGH-127592) (python#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Feb 3, 2026
…honGH-127592) (python#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Jun 12, 2026
…honGH-127592) (python#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Jun 12, 2026
…honGH-127592) (python#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reviewers
Assignees
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Xcode 16 deprecated the xcresultool tool we were using to gather test results after an iOS test run.
This tool was less than ideal anyway, as it didn't provide a way to see test results as the test suite was running. As a result, we had intermittent CI test failures that manifested as "test suite didn't finish", with no runtime debugging possible.
This PR adds a new configuration item (config->use_system_logger) for Apple platforms, providing the option to redirect stdout and stderr to the system log. This mirrors how Android operates - the implementation is strongly influenced by the Android implementation, but is just different enough that refactoring into a common base class is more trouble than it's worth IMHO.
This PR also adds a new test runner for iOS projects. This test runner uses the iOS testbed as a template, installing the compiled framework into copy generated from the template; and then building and running the test project in parallel with a log streamer that captures the system log from the iOS simulator as it runs. This allows us to observe test output as the test suite runs.
The test runner also includes options to allows the installation of app code. This means the testbed project can be used to test arbitrary Python code, in addition to the core CPython test suite.
Although this feature is primarily useful for iOS apps, the same code can be used for macOS apps - something that is useful when embedding Python in to a GUI app (where stdout/stderr isn't visible).
This PR resolves both #126925 and #126821; Resolving #126821 was a pre-requisite for fixing #126925 (and is the first commit in this PR), but by itself, there's very little opportunity to evaluate that it's working. You need an updated test runner to be able to observe that stdout/err redirection is working.
Fixes #126925
Fixes #126821
📚 Documentation preview 📚: https://cpython-previews--127592.org.readthedocs.build/