gh-157384: Unify `_Py_ext_module_loader_info` initialization by encukou · Pull Request #157758 · python/cpython · GitHub

This builds on #157387 and #157390 with a larger refactor.
#157387 should be merged first, but I'd like to get this one reviewed before that, in case we find something else to backport.

Unify _Py_ext_module_loader_info_init* functions. The “core” one was entirely unused, and the difference between “builtin” and “extension” can be expressed in arguments.
Consolidate getting & checking the module name from spec. (This improves error message wording for imp.create_builtin with empty name.)
Allow non-ASCII names.

Core modules (sys and builtins) still don't go through the routine.

Issue: PyImport_CreateModuleFromInitfunc(): wrong __name__ for submodules, non-ASCII names rejected, inittab name clashes #157384

…func() A single-phase init function that creates its module using only the last component of a dotted name (as pybind11 does for submodules) got a module whose __name__ was the short name. Set the package context while calling the init function, as the dynamic loader does, so the module gets the full name from the spec. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HoZE3WdRWQshvinfBZ137N

…nitfunc() The builtin loader info encoded the module name as ASCII, so a non-ASCII spec name failed with UnicodeEncodeError even for multi-phase init modules, which support such names when loaded dynamically. Fall back to UTF-8 and mark the name as non-ASCII, so that multi-phase init works and single-phase init is rejected with the same error as for dynamically loaded extensions.

This private function was added in 291cfa4 but never called.

It is always set to info->name.

…lready

PyUnicode_AsUTF8 might malloc.

Documentation build overview

📚 cpython-previews | 🛠️ Build #34638263 | 📁 Comparing d9ca232 against main (c0bc6fa)

🔍 Preview build

2 files changed
± c-api/import.html
± whatsnew/changelog.html

unification and refactor looks good to me!

requesting changes for the bug (inline comment), and also flagging that there's a user-visible behavior change with dotted single-phase init modules in inittab -- this changes their __name__ to be the full dotted name. I think this is a good change, but since it's potentially a breaking change, it should probably be called out in the 3.16 release notes.

I think this is a bug.

when the spec has no origin attribute, this would null-deref

repro (that you can turn into a regression test)

class Spec:
    name = "abc"

_imp.create_dynamic(Spec())

on main, this raises AttributeError, on this PR it segfaults. works with the suggested fix:

Suggested change

      
                    info.filename = PyObject_GetAttr(spec, &_Py_ID(origin));
          
                    if (info.filename == NULL) {
          
                        Py_DECREF(name);
          
                        return -1;
          
                    info.filename = PyObject_GetAttr(spec, &_Py_ID(origin));
          
                    if (info.filename == NULL) {
          
                        goto error;

Thanks for the catch!

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

there's a user-visible behavior change with dotted single-phase init modules in inittab -- this changes their name to be the full dotted name. I think this is a good change, but since it's potentially a breaking change, it should probably be called out in the 3.16 release notes.

Well, that's for #159316, which is now in 3.15 as well...
I'm not convinced it's visible enough for What's New, TBH.

why is this normalization happening only in the non-ASCII branch now?

iiuc, it should remain shared for both ASCII and non-ASCII.

example that worked before and would fail after:

import _heapq
import importlib.util

spec = importlib.util.spec_from_file_location("-heapq", _heapq.__file__)
module = importlib.util.module_from_spec(spec)

is this entire block now redundant? (because _Py_ext_module_loader_info_init would already set info.newcontext unconditionally)

itamaro and others added 12 commits September 12, 2026 18:37

Remove _Py_ext_module_loader_info_init_for_core

63eb6bf

This private function was added in 291cfa4 but never called.

Replace the newcontext string with a boolean

fb0b332

It is always set to info->name.

Unify module loader info initialization

d226a2c

Remove bool needs_swapcontext entirely

76c73aa

Gotta have a spec if there's no name & if we need origin

cc5b73f

Leave getattr(spec, "name") to info_init, if we don't have the name a…

1243838

…lready

Use _Py_ID for static string objects

b797ab6

Fix check for empty name

1e4bb6c

Pedantic fix for condition for setting info.origin

3a97ca3

Put newcontext back, for subinterpreter safety

902073b

PyUnicode_AsUTF8 might malloc.

encukou requested review from FFY00, brettcannon, ericsnowcurrently, kumaraditya303, ncoghlan and warsaw as code owners September 18, 2026 15:01

bedevere-app Bot mentioned this pull request Sep 18, 2026

Expose a C-API function to allow custom importers to create a module using an init function #116146

Closed

bedevere-app Bot added the awaiting core review label Sep 18, 2026

encukou mentioned this pull request Sep 18, 2026

gh-157384: Set package context in PyImport_CreateModuleFromInitfunc() #157387

Merged

encukou changed the title gh-116146: Unify _Py_ext_module_loader_info initialization gh-157384: Unify _Py_ext_module_loader_info initialization Sep 18, 2026

bedevere-app Bot mentioned this pull request Sep 18, 2026

PyImport_CreateModuleFromInitfunc(): wrong __name__ for submodules, non-ASCII names rejected, inittab name clashes #157384

Open

4 tasks

encukou added 5 commits September 18, 2026 17:36

Fix error message in tests

b290f31

Use PyUnicode_IS_ASCII in get_encoded_name

96621eb

Merge in the main branch to refresh generated code

18b96bb

Refcount fix

ac44362

Remove unnecessary check

d9ca232

Use PyObject_VectorcallMethod to call replace

fd07130

itamaro requested changes Sep 18, 2026

View reviewed changes

bedevere-app Bot added awaiting changes and removed awaiting core review labels Sep 18, 2026

encukou added 5 commits October 19, 2026 17:47

Merge in the main branch

a72f9b8

Fixups

502e569

Use nicer names for nonascii test modules

689b9b5

Merge in the main branch

4fd51b8

Handle missing origin

dbc56b0

encukou marked this pull request as draft October 20, 2026 23:38

bedevere-app Bot removed the awaiting changes label Oct 20, 2026

encukou marked this pull request as ready for review October 20, 2026 23:43

bedevere-app Bot added the awaiting core review label Oct 20, 2026

itamaro reviewed Oct 20, 2026

View reviewed changes

-                  modname = _PyObject_CallMethod(encoded, &_Py_ID(replace), "cc", '-', '_');
-                  if (modname == NULL)
-                      goto error;
+                      /* Call encoded.replace('-', '_') */

                   if (initfunc != NULL) {
                       /* An explicitly provided init function (see
                        * PyImport_CreateModuleFromInitfunc()) may belong to a submodule
                        * whose single-phase init creates the module using only the last
                        * component of the name (as e.g. pybind11 does).  Set the package
                        * context so that PyModule_Create() resolves the full name, the
                        * same as for dynamically loaded extensions. */
                       info.newcontext = PyUnicode_AsUTF8(info.name);
                       if (info.newcontext == NULL) {
                           _Py_ext_module_loader_info_clear(&info);
                           return NULL;
                       }
                   }

gh-157384: Unify `_Py_ext_module_loader_info` initialization by encukou · Pull Request #157758 · python/cpython · GitHub

Repository navigation

gh-157384: Unify _Py_ext_module_loader_info initialization - #157758

gh-157384: Unify _Py_ext_module_loader_info initialization#157758
encukou wants to merge 23 commits into
python:mainfrom
encukou:gh-116146-unify-info-init

encukou commented Sep 18, 2026 •
edited
Loading

Uh oh!

Uh oh!

read-the-docs-community Bot commented Sep 18, 2026 •
edited
Loading

Uh oh!

Uh oh!

itamaro left a comment

Uh oh!

itamaro Sep 18, 2026

Uh oh!

encukou Oct 20, 2026

Uh oh!

bedevere-app Bot commented Sep 18, 2026

Uh oh!

encukou commented Oct 20, 2026

Uh oh!

itamaro Oct 20, 2026

Uh oh!

itamaro Oct 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gh-157384: Unify `_Py_ext_module_loader_info` initialization by encukou · Pull Request #157758 · python/cpython · GitHub

Repository navigation

gh-157384: Unify _Py_ext_module_loader_info initialization - #157758

gh-157384: Unify _Py_ext_module_loader_info initialization#157758encukou wants to merge 23 commits intopython:mainpython/cpython:mainfrom encukou:gh-116146-unify-info-initencukou/cpython:gh-116146-unify-info-initCopy head branch name to clipboard

Conversation

encukou commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

read-the-docs-community Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

itamaro left a comment

Choose a reason for hiding this comment

Uh oh!

itamaro Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

encukou Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

bedevere-app Bot commented Sep 18, 2026

Uh oh!

encukou commented Oct 20, 2026

Uh oh!

itamaro Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

itamaro Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gh-157384: Unify _Py_ext_module_loader_info initialization#157758
encukou wants to merge 23 commits into
python:mainfrom
encukou:gh-116146-unify-info-init

encukou commented Sep 18, 2026 •
edited
Loading

read-the-docs-community Bot commented Sep 18, 2026 •
edited
Loading