FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

bpo-45020: Freeze the modules imported during startup. by ericsnowcurrently · Pull Request #28107 · python/cpython · GitHub

Repository navigation

Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
a8bd3fc
Fix freeze_module() in freeze_modules.py.
ericsnowcurrently Sep 2, 2021
6555ae4
Add FrozenSource and FrozenModule to freeze_modules.py.
ericsnowcurrently Sep 1, 2021
3a63f92
Leave all non-required frozen modules uncommitted.
ericsnowcurrently Sep 2, 2021
0587328
Leave *all* frozen modules uncommitted.
ericsnowcurrently Sep 2, 2021
9881456
Generate the list of frozen modules used by test_ctypes.
ericsnowcurrently Sep 1, 2021
3df3e23
Freeze the stdlib modules loaded during startup.
ericsnowcurrently Aug 12, 2021
bfd733c
Add the -X frozen_modules CLI option and PyConfig.use_frozen_modules.
ericsnowcurrently Aug 31, 2021
ca952fc
Ignore frozen modules depending on "-X frozen_modules".
ericsnowcurrently Aug 31, 2021
b4ca34c
Explicitly control PyConfig.use_frozen_modules during tests.
ericsnowcurrently Sep 1, 2021
ef5f3f7
Remember the stdlib dir during startup (adding PyConfig.stdlib_dir).
ericsnowcurrently Jun 12, 2021
4b630b8
Add _Py_GetMainConfig().
ericsnowcurrently Aug 31, 2021
dd1a498
Let _Py_GetStdlibDir() fall back to a config.
ericsnowcurrently Aug 31, 2021
af07489
Add _PyConfig_InitImportConfig().
ericsnowcurrently Aug 31, 2021
4e4baeb
Identify whether or not the executable is running installed (adding _…
ericsnowcurrently Aug 31, 2021
344a5ec
Default to "off" if in development.
ericsnowcurrently Aug 31, 2021
8710d80
Remember if built with --enable-optimizations (adding the _Py_OPT mac…
ericsnowcurrently Aug 31, 2021
77cf801
Default to "on" for PGO builds.
ericsnowcurrently Aug 31, 2021
2702554
Add a NEWS entry.
ericsnowcurrently Aug 31, 2021
becae8f
Fix test_embed for PyConfig.stdlib_dir.
ericsnowcurrently Sep 7, 2021
7f0c8eb
In _Py_GetStdlibDir(), treat "" as though it were NULL.
ericsnowcurrently Sep 7, 2021
d07f63c
Fix test_embed for "Default to "off" if in development.".
ericsnowcurrently Sep 7, 2021
4787a70
Treat __main__ as an essential frozen module.
ericsnowcurrently Sep 7, 2021
a75b808
Fix test_cmd_line_script for "Default to "off" if in development.".
ericsnowcurrently Sep 7, 2021
2050793
Ignore frozen submodules in generate_stdlib_module_names.py.
ericsnowcurrently Sep 7, 2021
e16771d
Add a comment to the frozen modules manifest file.
ericsnowcurrently Sep 7, 2021
e5b4da7
_Py_IsInstalled -> _Py_IsDevelopmentEnv.
ericsnowcurrently Sep 7, 2021
8b54908
Look up $_PYTHONTESTFROZENMODULES using getenv() instead of in os.env…
ericsnowcurrently Sep 7, 2021
a82b06a
Remove zipimport.h from the repo.
ericsnowcurrently Sep 7, 2021
ead34a1
Add zipimport to the list of essential frozen modules.
ericsnowcurrently Sep 7, 2021
58ccaff
Show how the frozen manifest changed.
ericsnowcurrently Sep 7, 2021
6d068b1
Stop using the frozen helper in test_cmd_line_script.
ericsnowcurrently Sep 7, 2021
1021f53
Go back to keeping frozen modules in the repo.
ericsnowcurrently Sep 7, 2021
c01b532
Also stop tracking the frozen manifest.
ericsnowcurrently Sep 8, 2021
0c1061d
Mark the frozen manifest as a generated file.
ericsnowcurrently Sep 8, 2021
fa23009
Drop a superfluous prefix on makefile rule dependencies.
ericsnowcurrently Sep 8, 2021
03b9571
Do not generate test code.
ericsnowcurrently Sep 8, 2021
b3a7aaf
Drop unused code from freeze_modules.py.
ericsnowcurrently Sep 8, 2021
6a896f1
Flip around the arg to the "frozen_modules" test helper.
ericsnowcurrently Sep 8, 2021
211febe
Disable frozen modules if $_PYTHONTESTFROZENMODULES is 0.
ericsnowcurrently Sep 8, 2021
fbaf15c
Add the "usefrozen" arg to CleanImport.__init__().
ericsnowcurrently Sep 8, 2021
6243c54
Undo the changes to tests that were using the os module.
ericsnowcurrently Sep 8, 2021
da978a2
Undo the changes to tests that were using the os module.
ericsnowcurrently Sep 8, 2021
46d488f
Ignore decode errors in find_frozen().
ericsnowcurrently Sep 8, 2021
93fc5c0
Fix test_ctypes.
ericsnowcurrently Sep 8, 2021
dd36ba5
Allow for a .exe suffix on the executable.
ericsnowcurrently Sep 9, 2021
45e7509
Do not clear the frozen .h files with "make distclean", now that they…
ericsnowcurrently Sep 9, 2021
a7d607f
Update the frozen modules using a debug build.
ericsnowcurrently Sep 9, 2021
75556af
Always flush the printed "title" when freezing modules.
ericsnowcurrently Sep 10, 2021
37b79f1
Add the frozen manifest back into the repo.
ericsnowcurrently Sep 10, 2021
d93c1e7
Drop the PGO check (for now).
ericsnowcurrently Sep 10, 2021
25b60de
Drop _Py_IsDevelopmentEnv().
ericsnowcurrently Sep 10, 2021
358ab69
Drop -_Py_GetStdlibDir() and PyConfig.stdlib_dir.
ericsnowcurrently Sep 10, 2021
1fc1198
Default to "-X frozen_modules=off" if built with --with-debug.
ericsnowcurrently Sep 10, 2021
37eb1f8
Fix test_embed.
ericsnowcurrently Sep 10, 2021
2e05771
Clean up the marshal code.
ericsnowcurrently Sep 10, 2021
82b4f85
Add WFILE.refs.nonref.
ericsnowcurrently Sep 10, 2021
097fc40
Add _PyMarshal_WriteForFreezing().
ericsnowcurrently Sep 10, 2021
a333fa3
Fix the ASAN job.
ericsnowcurrently Sep 11, 2021
8abf4a0
Fix the Windows builds.
ericsnowcurrently Sep 11, 2021
1005321
Use a simpler change for test_idle.
ericsnowcurrently Sep 11, 2021
7d62747
Fix test_4_daemon_threads.
ericsnowcurrently Sep 13, 2021
File filter

Filter by extension

Filter by extension .c  (1) .py  (1) All 2 file types selected
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
Prev Previous commit
Next Next commit
Disable frozen modules if $_PYTHONTESTFROZENMODULES is 0.
  • Loading branch information
ericsnowcurrently committed Sep 8, 2021
commit 211febe4d28748278bbfe364ac4639865a452758
4 changes: 1 addition & 3 deletions Lib/test/support/import_helper.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -111,10 +111,8 @@ def _save_and_block_module(name, orig_modules):

@contextlib.contextmanager
def frozen_modules(enabled=True):
# No other code should be setting this env var.
assert os.getenv('_PYTHONTESTFROZENMODULES') is None
# FYI: the env var will never show up in os.environ.
os.putenv('_PYTHONTESTFROZENMODULES', '1' if enabled else '')
os.putenv('_PYTHONTESTFROZENMODULES', '1' if enabled else '0')
try:
yield
finally:
Expand Down
120 changes: 67 additions & 53 deletions Python/import.c
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -886,7 +886,7 @@ _imp__fix_co_filename_impl(PyObject *module, PyCodeObject *code,


/* Forward */
static const struct _frozen * find_frozen(PyObject *, bool);
static const struct _frozen * find_frozen(PyObject *);


/* Helper to test for built-in module */
Expand Down Expand Up @@ -1051,49 +1051,82 @@ _imp_create_builtin(PyObject *module, PyObject *spec)
/* Frozen modules */

static bool
is_essential_frozen_module(PyObject *name)
parse_env_var_flag(const char *rawvalue, bool *isset)
{
/* It isn't set or it is set to an empty string. */
if (rawvalue == NULL || strlen(rawvalue) == 0) {
if (isset != NULL) {
*isset = false;
}
return false;
}
if (isset != NULL) {
*isset = true;
}
if (strcmp(rawvalue, "0") == 0) {
return false;
}
/* For now we treat all other non-empty strings as true. */
return true;
}

static bool
is_essential_frozen_module(const char *name)
{
/* These modules are necessary to bootstrap the import system. */
if (_PyUnicode_EqualToASCIIString(name, "_frozen_importlib")) {
if (strcmp(name, "_frozen_importlib") == 0) {
return true;
}
if (_PyUnicode_EqualToASCIIString(name, "_frozen_importlib_external")) {
if (strcmp(name, "_frozen_importlib_external") == 0) {
return true;
}
if (_PyUnicode_EqualToASCIIString(name, "zipimport")) {
if (strcmp(name, "zipimport") == 0) {
return true;
}
/* This doesn't otherwise have anywhere to find the module.
See frozenmain.c. */
if (_PyUnicode_EqualToASCIIString(name, "__main__")) {
if (strcmp(name, "__main__") == 0) {
return true;
}
return false;
}

static bool
use_frozen(const char *modname)
{
bool isset;
/* Note that we don't bother with os.environ. */
bool use = parse_env_var_flag(getenv("_PYTHONTESTFROZENMODULES"), &isset);
if (!isset) {
const PyConfig *config = _Py_GetConfig();
use = config->use_frozen_modules;
}
if (!use && modname != NULL) {
use = is_essential_frozen_module(modname);
}
return use;
}

static PyObject *
list_frozen_module_names(bool force)
list_frozen_module_names()
{
const PyConfig *config = _Py_GetConfig();
PyObject *names = PyList_New(0);
if (names == NULL) {
return NULL;
}
bool enabled = use_frozen(NULL);
for (const struct _frozen *p = PyImport_FrozenModules; ; p++) {
if (p->name == NULL) {
break;
}
if (!enabled && !is_essential_frozen_module(p->name)) {
continue;
}
PyObject *name = PyUnicode_FromString(p->name);
if (name == NULL) {
Py_DECREF(names);
return NULL;
}
if (!config->use_frozen_modules && !force) {
if (!is_essential_frozen_module(name)) {
Py_DECREF(name);
continue;
}
}
int res = PyList_Append(names, name);
Py_DECREF(name);
if (res != 0) {
Expand All @@ -1105,33 +1138,31 @@ list_frozen_module_names(bool force)
}

static const struct _frozen *
find_frozen(PyObject *name, bool force)
find_frozen(PyObject *modname)
{
const struct _frozen *p;

if (name == NULL)
if (modname == NULL) {
return NULL;

const PyConfig *config = _Py_GetConfig();
if (!config->use_frozen_modules && !force) {
if (!is_essential_frozen_module(name)) {
return NULL;
}
}

const char *name = PyUnicode_AsUTF8(modname);
if (!use_frozen(name)) {
return NULL;
}
const struct _frozen *p;
for (p = PyImport_FrozenModules; ; p++) {
if (p->name == NULL)
if (p->name == NULL) {
return NULL;
if (_PyUnicode_EqualToASCIIString(name, p->name))
}
if (strcmp(name, p->name) == 0) {
break;
}
}
return p;
}

static PyObject *
get_frozen_object(PyObject *name, bool force)
get_frozen_object(PyObject *name)
{
const struct _frozen *p = find_frozen(name, force);
const struct _frozen *p = find_frozen(name);
int size;

if (p == NULL) {
Expand All @@ -1153,9 +1184,9 @@ get_frozen_object(PyObject *name, bool force)
}

static PyObject *
is_frozen_package(PyObject *name, bool force)
is_frozen_package(PyObject *name)
{
const struct _frozen *p = find_frozen(name, force);
const struct _frozen *p = find_frozen(name);
int size;

if (p == NULL) {
Expand Down Expand Up @@ -1188,7 +1219,7 @@ PyImport_ImportFrozenModuleObject(PyObject *name)
int ispackage;
int size;

p = find_frozen(name, false);
p = find_frozen(name);

if (p == NULL)
return 0;
Expand Down Expand Up @@ -1949,17 +1980,6 @@ _imp_init_frozen_impl(PyObject *module, PyObject *name)
return import_add_module(tstate, name);
}

static bool
parse_env_var_flag(const char *value)
{
/* It isn't set or it is set to an empty string. */
if (value == NULL || strlen(value) == 0) {
return false;
}
/* For now we treat all non-empty strings as true. */
return true;
}

/*[clinic input]
_imp.get_frozen_object

Expand All @@ -1973,9 +1993,7 @@ static PyObject *
_imp_get_frozen_object_impl(PyObject *module, PyObject *name)
/*[clinic end generated code: output=2568cc5b7aa0da63 input=ed689bc05358fdbd]*/
{
/* Note that we don't bother with os.environ. */
bool force = parse_env_var_flag(getenv("_PYTHONTESTFROZENMODULES"));
return get_frozen_object(name, force);
return get_frozen_object(name);
}

/*[clinic input]
Expand All @@ -1991,9 +2009,7 @@ static PyObject *
_imp_is_frozen_package_impl(PyObject *module, PyObject *name)
/*[clinic end generated code: output=e70cbdb45784a1c9 input=81b6cdecd080fbb8]*/
{
/* Note that we don't bother with os.environ. */
bool force = parse_env_var_flag(getenv("_PYTHONTESTFROZENMODULES"));
return is_frozen_package(name, force);
return is_frozen_package(name);
}

/*[clinic input]
Expand Down Expand Up @@ -2025,11 +2041,9 @@ static PyObject *
_imp_is_frozen_impl(PyObject *module, PyObject *name)
/*[clinic end generated code: output=01f408f5ec0f2577 input=7301dbca1897d66b]*/
{
/* Note that we don't bother with os.environ. */
bool force = parse_env_var_flag(getenv("_PYTHONTESTFROZENMODULES"));
const struct _frozen *p;

p = find_frozen(name, force);
p = find_frozen(name);
return PyBool_FromLong((long) (p == NULL ? 0 : p->size));
}

Expand All @@ -2043,7 +2057,7 @@ static PyObject *
_imp__frozen_module_names_impl(PyObject *module)
/*[clinic end generated code: output=80609ef6256310a8 input=76237fbfa94460d2]*/
{
return list_frozen_module_names(true);
return list_frozen_module_names();
}

/* Common implementation for _imp.exec_dynamic and _imp.exec_builtin */
Expand Down

Back | FazBrowse Home | New Git URL