FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

--disable-gil & --enable-experimental-jit fails to build on M1 Mac · Issue #118844 · python/cpython · GitHub

  1. python
  2. cpython

Repository navigation

--disable-gil & --enable-experimental-jit fails to build on M1 Mac #118844

Closed
Closed
--disable-gil & --enable-experimental-jit fails to build on M1 Mac#118844
Assignees
Labels
OS-macbuildThe build process and cross-buildtopic-JITtype-bugAn unexpected behavior, bug, or error

Description

smontanaro
opened on May 9, 2024

Bug report

Bug description:

On my M1 Mac, if I configure like so: ./configure --disable-gil --enable-experimental-jit I get a compilation error:

gcc -c -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall    -D_Py_TIER2=1 -D_Py_JIT -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal -I./Include/internal/mimalloc  -I. -I./Include    -DPy_BUILD_CORE -o Python/jit.o Python/jit.c
In file included from Python/jit.c:392:
./jit_stencils.h:13419:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16a(code + 0x198, (uintptr_t)&_PyCriticalSection_Resume);

(followed by many more of the same error).

I don't recall if I've tried this configure combination before or not. I'm using Apple's clang on an up-to-date MacOS (14.4.1):

% gcc --version
Apple clang version 15.0.0 (clang-1500.3.9.4)
Target: arm64-apple-darwin23.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

Activity

  1. smontanaro
    added
    type-bugAn unexpected behavior, bug, or error
    on May 9, 2024
  2. Eclips4
    added
    OS-mac
    buildThe build process and cross-build
    on May 9, 2024
  3. Eclips4
    changed the title [-]--disable-gil & --wirh-experimental-jit fails to build on M1 Mac[/-] [+]--disable-gil & --enable-experimental-jit fails to build on M1 Mac[/+] on May 9, 2024

  4. Eclips4 commented on May 9, 2024

    Eclips4
    on May 9, 2024
    Member

    Full compilation error:

    Details 
    gcc -c -fno-strict-overflow -Wsign-compare -g -Og -Wall    -D_Py_TIER2=1 -D_Py_JIT -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal -I./Include/internal/mimalloc  -I. -I./Include    -DPy_BUILD_CORE -o Python/jit.o Python/jit.c
In file included from Python/jit.c:392:
./jit_stencils.h:24576:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16a(code + 0x230, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:24577:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16b(code + 0x234, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:24578:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16c(code + 0x238, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:24579:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16d(code + 0x23c, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:24596:49: error: use of undeclared identifier '_PyCriticalSection_BeginSlow'
    patch_aarch64_16a(code + 0x2a8, (uintptr_t)&_PyCriticalSection_BeginSlow);
                                                ^
./jit_stencils.h:24597:49: error: use of undeclared identifier '_PyCriticalSection_BeginSlow'
    patch_aarch64_16b(code + 0x2ac, (uintptr_t)&_PyCriticalSection_BeginSlow);
                                                ^
./jit_stencils.h:24598:49: error: use of undeclared identifier '_PyCriticalSection_BeginSlow'
    patch_aarch64_16c(code + 0x2b0, (uintptr_t)&_PyCriticalSection_BeginSlow);
                                                ^
./jit_stencils.h:24599:49: error: use of undeclared identifier '_PyCriticalSection_BeginSlow'
    patch_aarch64_16d(code + 0x2b4, (uintptr_t)&_PyCriticalSection_BeginSlow);
                                                ^
./jit_stencils.h:45384:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16a(code + 0x1c0, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:45385:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16b(code + 0x1c4, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:45386:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16c(code + 0x1c8, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:45387:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16d(code + 0x1cc, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:45404:49: error: use of undeclared identifier '_PyCriticalSection_BeginSlow'
    patch_aarch64_16a(code + 0x238, (uintptr_t)&_PyCriticalSection_BeginSlow);
                                                ^
./jit_stencils.h:45405:49: error: use of undeclared identifier '_PyCriticalSection_BeginSlow'
    patch_aarch64_16b(code + 0x23c, (uintptr_t)&_PyCriticalSection_BeginSlow);
                                                ^
./jit_stencils.h:45406:49: error: use of undeclared identifier '_PyCriticalSection_BeginSlow'
    patch_aarch64_16c(code + 0x240, (uintptr_t)&_PyCriticalSection_BeginSlow);
                                                ^
./jit_stencils.h:45407:49: error: use of undeclared identifier '_PyCriticalSection_BeginSlow'
    patch_aarch64_16d(code + 0x244, (uintptr_t)&_PyCriticalSection_BeginSlow);
                                                ^
./jit_stencils.h:48060:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16a(code + 0x358, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:48061:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16b(code + 0x35c, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:48062:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16c(code + 0x360, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make: *** [Python/jit.o] Error 1

  5. brandtbucher commented on May 9, 2024

    brandtbucher
    on May 9, 2024
    Member

    Thanks! I’m not sure if this is a build configuration that’s been given any thought yet... or if it even makes sense right now (I believe our tier 2 optimization pipeline is disabled on free-threaded builds, but I’d need to check). Regardless, we’re headed towards a future with both of these features and should probably make sure that they at least build together.

    @savannahostrowski, interested in taking this too? I’m on my phone now, but I think we probably just need to #include the header where this is defined in jit.c.

  6. brandtbucher
    self-assigned this
    on May 9, 2024

  7. sobolevn commented on May 9, 2024

    sobolevn
    on May 9, 2024
    Member

    Here' the code sample that is generated:

    memcpy(data, data_body, sizeof(data_body));
    patch_64(data + 0x0, instruction->oparg);
    memcpy(code, code_body, sizeof(code_body));
    patch_aarch64_33rx(code + 0x30, (uintptr_t)data);
    patch_aarch64_26r(code + 0x16c, (uintptr_t)code + sizeof(code_body));
    patch_aarch64_33rx(code + 0x190, (uintptr_t)data);
    patch_aarch64_26r(code + 0x1c8, instruction_starts[instruction->error_target]);
    patch_aarch64_16a(code + 0x1d0, (uintptr_t)&_PyEval_FormatExcUnbound);
    patch_aarch64_16b(code + 0x1d4, (uintptr_t)&_PyEval_FormatExcUnbound);
    patch_aarch64_16c(code + 0x1d8, (uintptr_t)&_PyEval_FormatExcUnbound);
    patch_aarch64_16d(code + 0x1dc, (uintptr_t)&_PyEval_FormatExcUnbound);
    patch_aarch64_16a(code + 0x1e8, (uintptr_t)&_PyCriticalSection_Resume);
    patch_aarch64_16b(code + 0x1ec, (uintptr_t)&_PyCriticalSection_Resume);
    patch_aarch64_16c(code + 0x1f0, (uintptr_t)&_PyCriticalSection_Resume);
    patch_aarch64_16d(code + 0x1f4, (uintptr_t)&_PyCriticalSection_Resume);
    patch_aarch64_16a(code + 0x200, (uintptr_t)&_PyThreadState_GetCurrent);
    patch_aarch64_16b(code + 0x204, (uintptr_t)&_PyThreadState_GetCurrent);
    patch_aarch64_16c(code + 0x208, (uintptr_t)&_PyThreadState_GetCurrent);
    patch_aarch64_16d(code + 0x20c, (uintptr_t)&_PyThreadState_GetCurrent);

  8. savannahostrowski commented on May 9, 2024

    savannahostrowski
    on May 9, 2024
    Member

    Yep - feel free to assign to me @brandtbucher! I'll take a look tonight or tomorrow!

  9. brandtbucher
    assigned
    savannahostrowski
    and unassigned
    brandtbucher
    on May 9, 2024

  10. smontanaro commented on May 9, 2024

    smontanaro
    on May 9, 2024
    ContributorAuthor

    FWIW, I'm not sure this is M1-specific. It compiled just fine on my Dell laptop running XUbuntu 22.04. OTOH, on an old MacBook Air I have laying around (dual core Intel i5, MacOS 12.7.4), it failed to compile Python/jit.c in the same fashion as on my M1 MBP. Maybe it's a MacOS issue?

  11. brandtbucher
    added a commit that references this issue on May 11, 2024

    GH-118844: Fix build failures when combining --disable-gil with --ena…

    5b941e5
  12. miss-islington
    added a commit that references this issue on May 11, 2024

    pythonGH-118844: Fix build failures when combining --disable-gil with…

    f3612d8
  13. brandtbucher
    closed this as completedon May 11, 2024

  14. smontanaro commented on May 13, 2024

    smontanaro
    on May 13, 2024
    ContributorAuthor

    @brandtbucher @savannahostrowski

    I saw the close-out of the PR and apparent backport to 3.13. I can build fine on main, but compilation still fails for me on 3.13. On both main and 3.13 I used this configure command (grepped from config.status):

    /bin/sh './configure'  '--enable-experimental-jit=yes' '--disable-gil' '--with-openssl=/opt/homebrew' 'GDBM_CFLAGS=-I/opt/homebrew/opt/gdbm/include' 'GDBM_LIBS=-L/opt/homebrew/opt/gdbm/lib -lgdbm' 'LIBLZMA_CFLAGS=-I/opt/homebrew/include' 'LIBLZMA_LIBS=-L/opt/homebrew/lib -llzma' $ac_configure_extra_args --no-create --no-recursion

    I thought the PR was backported to 3.13, but maybe not? Diffing the main and 3.13 versions of Python/jit.c I see this line is missing from the 3.13 version:

    #include "pycore_critical_section.h"

    Adding it allows the compilation to succeed. I don't see it in the jit.c source on Github either.

  15. brandtbucher commented on May 13, 2024

    brandtbucher
    on May 13, 2024
    Member

    I saw the close-out of the PR and apparent backport to 3.13. I can build fine on main, but compilation still fails for me on 3.13.

    Ah, sorry, looks like the backport failed! Re-running it now...

  16. brandtbucher
    added a commit that references this issue on May 13, 2024

    [3.13] GH-118844: Fix build failures when combining --disable-gil wit…

    9d2c10b

  17. brandtbucher commented on May 13, 2024

    brandtbucher
    on May 13, 2024
    Member

    ...done!

  18. brandtbucher
    added
    topic-JIT
    on May 31, 2024
  19. estyxx
    added a commit that references this issue on Jul 17, 2024

    pythonGH-118844: Fix build failures when combining --disable-gil with…

    7652b6e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

OS-macbuildThe build process and cross-buildtopic-JITtype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    Back | FazBrowse Home | New Git URL