Add patch for https://bugs.python.org/issue42015 (which affects pybind11) by mbargull · Pull Request #402 · conda-forge/python-feedstock · GitHub

Checklist

Used a personal fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

This adds python/cpython#22674 which is a backport of python/cpython#22670 addressing https://bugs.python.org/issue42015 which was reported by the pybind11 maintainer @YannickJadoul in reference to pybind/pybind11#2558 and their workaround PR pybind/pybind11#2576.

Given the small changeset and scope of the patch, it could make sense to include it in our python build even before the Python 3.9.1 release (schedules for December).

cc @isuruf, @conda-forge/pybind11, @henryiii

…da-forge-pinning 2020.10.13.19.09.25

This reverts commit f2cc154.

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Without this conda-smithy=3.8.2 failed with

Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Adding in variants from /tmp/tmp600bzq31/conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from /tmp/tmp600bzq31/conda_build_config.yaml
Adding in variants from /home/mbargull/code/src/conda-forge/git/python-feedstock/recipe/conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from /home/mbargull/code/src/conda-forge/git/python-feedstock/recipe/conda_build_config.yaml
Traceback (most recent call last):
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/bin/conda-smithy", line 10, in <module>
    sys.exit(main())
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/lib/python3.7/site-packages/conda_smithy/cli.py", line 613, in main
    args.subcommand_func(args)
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/lib/python3.7/site-packages/conda_smithy/cli.py", line 419, in __call__
    self._call(args, tmpdir)
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/lib/python3.7/site-packages/conda_smithy/cli.py", line 430, in _call
    temporary_directory=temporary_directory,
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/lib/python3.7/site-packages/conda_smithy/configure_feedstock.py", line 1938, in main
    render_travis(env, config, forge_dir, return_metadata=True)
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/lib/python3.7/site-packages/conda_smithy/configure_feedstock.py", line 1082, in render_travis
    return_metadata=return_metadata,
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/lib/python3.7/site-packages/conda_smithy/configure_feedstock.py", line 602, in _render_ci_provider
    os.path.join(forge_dir, forge_config["recipe_dir"]), config=config
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/lib/python3.7/site-packages/conda_build/variants.py", line 525, in get_package_combined_spec
    combined_spec = combine_specs(specs, log_output=config.verbose)
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/lib/python3.7/site-packages/conda_build/variants.py", line 277, in combine_specs
    log_output=log_output)
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/lib/python3.7/site-packages/conda_build/variants.py", line 252, in _combine_spec_dictionaries
    spec_source))
ValueError: variant config in /home/mbargull/code/src/conda-forge/git/python-feedstock/recipe/conda_build_config.yaml is ambiguous because it does not fully implement all zipped keys, or specifies a subspace that is not fully implemented.

osx-arm64 failed with

+++ echo 'Downloading 11.0 sdk'
Downloading 11.0 sdk
+++ curl -L -O https://github.com/phracker/MacOSX-SDKs/releases/download/10.15/MacOSX11.0.sdk.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100     9  100     9    0     0     20      0 --:--:-- --:--:-- --:--:--    20
++++ dirname /Applications/Xcode_12_beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk
+++ tar -xf MacOSX11.0.sdk.tar.xz -C /Applications/Xcode_12_beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs
tar: Error opening archive: Unrecognized archive format
+++ '[' '!' -z osx_arm64_target_platformosx-arm64 ']'
+++ echo ''
+++ echo CONDA_BUILD_SYSROOT:
+++ echo '- /Applications/Xcode_12_beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk'
+++ echo ''
+++ echo 'export CONDA_BUILD_SYSROOT='\''/Applications/Xcode_12_beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk'\'''
+++ echo 'export MACOSX_DEPLOYMENT_TARGET='\''11.0'\'''
+++ [[ -d /Applications/Xcode_12_beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk ]]
+++ echo 'Missing CONDA_BUILD_SYSROOT: /Applications/Xcode_12_beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk'
+++ exit 1
Missing CONDA_BUILD_SYSROOT: /Applications/Xcode_12_beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk

due to non-existing https://github.com/phracker/MacOSX-SDKs/releases/download/10.15/MacOSX11.0.sdk.tar.xz

Thanks, @mbargull! It's really great for us to see this getting patched so soon :-)

@YannickJadoul, I saw your work at pybind/pybind11#2576 in which you added the workaround depending on the exact Python version (which makes sense giving that there's probably nothing else to go off of to see if the workaround is need or not).
That made me think that you should be made aware of the possibility that distributors (like conda-forge, but possibly others) might also include your patch in their 3.9.0 builds!

The inclusion of this patch in our 3.9.0 is a suggestion from my part, but I'd like members of @conda-forge/python to decide on this (which also includes @mingwandroid who's working on packaging CPython for Anaconda).

osx-arm64 failed with
...
due to non-existing https://github.com/phracker/MacOSX-SDKs/releases/download/10.15/MacOSX11.0.sdk.tar.xz

Looking at the logs, we used the 11.0 SDK that the CI provided before. But Azure https://github.com/microsoft/azure-pipelines-image-generation/commit/1a6b857e3408a2318dec81dbc332ff2abecaab62#diff-7a1606bd717fc0cf55f9419157117d9ca306f91bd2fdfc294720687d7be1b2c7R5 their image and the Xcode 12 beta alongside it -- and apparently the SDK isn't available under the same path anymore.
Maybe the SDK is under /Applications/Xcode_12.2.app? We'll have to check.

In a previous (successful) build we had

2020-10-10T20:25:12.5804880Z ##[section]Starting: osx osx_arm64_target_platformosx-arm64
2020-10-10T20:25:12.8170180Z ##[section]Starting: Initialize job
2020-10-10T20:25:12.8171630Z Agent name: 'Azure Pipelines 8'
2020-10-10T20:25:12.8172060Z Agent machine name: 'Mac-1602357544119'
2020-10-10T20:25:12.8172350Z Current agent version: '2.175.2'
2020-10-10T20:25:12.8218120Z ##[group]Operating System
2020-10-10T20:25:12.8218400Z Mac OS X
2020-10-10T20:25:12.8218560Z 10.15.7
2020-10-10T20:25:12.8218740Z 19H2
2020-10-10T20:25:12.8218910Z ##[endgroup]
2020-10-10T20:25:12.8219130Z ##[group]Virtual Environment
2020-10-10T20:25:12.8219370Z Environment: macos-10.15
2020-10-10T20:25:12.8219600Z Version: 20201003.1
2020-10-10T20:25:12.8219980Z Included Software: https://github.com/actions/virtual-environments/blob/macos-10.15/20201003.1/images/macos/macos-10.15-Readme.md
2020-10-10T20:25:12.8220360Z ##[endgroup]

Now we have

2020-10-14T10:49:11.6471610Z ##[section]Starting: osx osx_arm64_target_platformosx-arm64
2020-10-14T10:49:11.9642240Z ##[section]Starting: Initialize job
2020-10-14T10:49:11.9643740Z Agent name: 'Azure Pipelines 140'
2020-10-14T10:49:11.9644150Z Agent machine name: 'Mac-1602671764673'
2020-10-14T10:49:11.9644420Z Current agent version: '2.175.2'
2020-10-14T10:49:11.9695340Z ##[group]Operating System
2020-10-14T10:49:11.9695640Z Mac OS X
2020-10-14T10:49:11.9695800Z 10.15.7
2020-10-14T10:49:11.9695930Z 19H2
2020-10-14T10:49:11.9696080Z ##[endgroup]
2020-10-14T10:49:11.9696260Z ##[group]Virtual Environment
2020-10-14T10:49:11.9696470Z Environment: macos-10.15
2020-10-14T10:49:11.9696660Z Version: 20201011.1
2020-10-14T10:49:11.9697010Z Included Software: https://github.com/actions/virtual-environments/blob/macos-10.15/20201011.1/images/macos/macos-10.15-Readme.md
2020-10-14T10:49:11.9697350Z ##[endgroup]

I didn't read actions/runner-images#1646 yet, but there seems to be more information for this.

@mbargull, right, that ís actually something to be discussed. Let's also bring in @henryiii into this, then.
It might be hard to actually have a reliable way of checking whether a patched version of Python is running?

Whatever we decide, this patch will still be very useful for users stuck with older, pre-2.6.0 version of pybind11 that want to support 3.9!

Maybe the SDK is under /Applications/Xcode_12.2.app? We'll have to check.

Yep, from conda-forge/conda-forge-ci-setup-feedstock#125

lrwxr-xr-x  1 runner  wheel   10 Oct 11 13:15 /Applications/Xcode_12.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk -> MacOSX.sdk

I'd vote for putting it in. It is hard to control the usage of pybind11, in part because a lot of software uses it directly from the headers, for example, via submodule, rather than using conda-forge's pybind11 package. SciPy, PyTorch, and more will segfault until they update unless we put this in. We can "un-patch" the conda-forge pybind11 package to make it slightly faster and avoid leaking on 3.9.0 if we want to, but I believe the leak is very small and the performance difference is minimal (a few clock cycles per callable destruction).

Hi! This is the friendly conda-forge automerge bot!

I considered the following status checks when analyzing this PR:

linter: passed
drone: passed
travis: passed
azure: passed

Thus the PR was passing and merged! Have a great day!

osx-arm64 build failed at

python-feedstock/recipe/build_base.sh

Lines 294 to 297 in c5af9fa

    
           ${SRC_DIR}/configure "${_common_configure_args[@]}" \ 
        
                                "${_dbg_opts[@]}" \ 
        
                                --oldincludedir=${BUILD_PREFIX}/${HOST}/sysroot/usr/include \ 
        
                                --enable-shared

with

2020-10-14T14:45:22.2281860Z + /Users/runner/miniforge3/conda-bld/python-split_1602686242521/work/configure --prefix=/Users/runner/miniforge3/conda-bld/python-split_1602686242521/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh --build=arm64-apple-darwin20.0.0 --host=arm64-apple-darwin20.0.0 --enable-ipv6 --with-ensurepip=no --with-tzpath=/Users/runner/miniforge3/conda-bld/python-split_1602686242521/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/share/zoneinfo --with-computed-gotos --with-system-ffi --enable-loadable-sqlite-extensions --with-tcltk-includes=-I/Users/runner/miniforge3/conda-bld/python-split_1602686242521/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/include '--with-tcltk-libs=-L/Users/runner/miniforge3/conda-bld/python-split_1602686242521/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib -ltcl8.6 -ltk8.6' --with-platlibdir=lib --with-lto --oldincludedir=/Users/runner/miniforge3/conda-bld/python-split_1602686242521/_build_env/arm64-apple-darwin20.0.0/sysroot/usr/include --enable-shared
2020-10-14T14:45:22.6268860Z configure: loading site script /Users/runner/miniforge3/conda-bld/python-split_1602686242521/work/config.site
2020-10-14T14:45:22.6714460Z checking build system type... arm64-apple-darwin20.0.0
2020-10-14T14:45:22.6926740Z checking host system type... arm64-apple-darwin20.0.0
2020-10-14T14:45:22.6955740Z checking for python3.9... python3.9
2020-10-14T14:45:22.7525340Z checking for --enable-universalsdk... no
2020-10-14T14:45:22.7558570Z checking for --with-universal-archs... no
2020-10-14T14:45:22.7602690Z checking MACHDEP... "darwin"
2020-10-14T14:45:22.7615590Z checking for arm64-apple-darwin20.0.0-gcc... arm64-apple-darwin20.0.0-clang
2020-10-14T14:45:23.3849150Z checking whether the C compiler works... yes
2020-10-14T14:45:23.3850070Z checking for C compiler default output file name... a.out
2020-10-14T14:45:23.5250860Z checking for suffix of executables... 
2020-10-14T14:45:23.6797900Z checking whether we are cross compiling... configure: error: in `/Users/runner/miniforge3/conda-bld/python-split_1602686242521/work/build-shared':
2020-10-14T14:45:23.6798860Z configure: error: cannot run C compiled programs.
2020-10-14T14:45:23.6799720Z If you meant to cross compile, use `--host'.
2020-10-14T14:45:23.6800560Z See `config.log' for more details

I made a bad PR for latest conda-build. Fixed in conda/conda-build#4094. I'll add a workaround in the compiler activation recipe

Thanks for this patch!!

mbargull added 7 commits October 14, 2020 10:27

Make version in cbc.yaml, build_base.* consistent

2e6df79

Add patch for https://bugs.python.org/issue42015

ae86d01

Bump build number

c953720

Add mbargull to maintainers

f776b9f

Add python_impl+numpy to cbc.yaml for rerender

f2cc154

MNT: Re-rendered with conda-build 3.20.3, conda-smithy 3.8.2, and con…

7548664

…da-forge-pinning 2020.10.13.19.09.25

Revert "Add python_impl+numpy to cbc.yaml for rerender"

c54fc58

This reverts commit f2cc154.

mbargull requested review from isuruf, jakirkham, jjhelmus, mingwandroid, msarahan, ocefpaf, pelson and scopatz as code owners October 14, 2020 08:53

mbargull commented Oct 14, 2020

View reviewed changes

mbargull added 4 commits October 14, 2020 11:11

Fix typo in build_base.bat

5bf1300

build_base.bat: Move comments out of for-block

20c4a58

Use consistent naming of non-dotted version

c79e58b

Fix another typo in build_base.bat

8d48a5e

mbargull mentioned this pull request Oct 14, 2020

Add an additional OSX_SDK_DIR for newer macOS images on Azure conda-forge/conda-forge-ci-setup-feedstock#126

Merged

5 tasks

mingwandroid approved these changes Oct 14, 2020

View reviewed changes

isuruf added the automerge Merge the PR when CI passes label Oct 14, 2020

maintainers: insert into alphabetical order

c5af9fa

github-actions Bot merged commit 19c3ddf into conda-forge:master Oct 14, 2020

henryiii mentioned this pull request Oct 15, 2020

[chore] Test pybind11 2.6.0 (RC 2?) pytorch/pytorch#46367

Open

tdegeus mentioned this pull request Oct 15, 2020

Rebuild for python39 conda-forge/python-goosefem-feedstock#7

Merged

+              python_impl:
+                - cpython
+              numpy:
+                - 1.19

Add patch for https://bugs.python.org/issue42015 (which affects pybind11) by mbargull · Pull Request #402 · conda-forge/python-feedstock · GitHub

Uh oh!

Add patch for https://bugs.python.org/issue42015 (which affects pybind11) - #402

Conversation

mbargull commented Oct 14, 2020

Uh oh!

conda-forge-linter commented Oct 14, 2020

Uh oh!

mbargull Oct 14, 2020

Choose a reason for hiding this comment

Uh oh!

mbargull commented Oct 14, 2020

Uh oh!

YannickJadoul commented Oct 14, 2020

Uh oh!

mbargull commented Oct 14, 2020

Uh oh!

mbargull commented Oct 14, 2020

Uh oh!

YannickJadoul commented Oct 14, 2020

Uh oh!

mbargull commented Oct 14, 2020

Uh oh!

henryiii commented Oct 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mbargull commented Oct 14, 2020

Uh oh!

isuruf commented Oct 14, 2020

Uh oh!

tdegeus commented Oct 15, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

henryiii commented Oct 14, 2020 •
edited
Loading

github-actions Bot commented Oct 14, 2020 •
edited
Loading