| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
Fix a crash caused by immortal interned strings being shared between sub-interpreters that use basic single-phase init. In that case, the string can be used by an interpreter that outlives the interpeter that created and interned it. For interpreters that share obmalloc state, also share the interned dict with the main interpreter.
|
This is a small change code-wise but I'm not sure about the full implications of it. It does fix a bug but might create other bugs or problems, I'm not confident. One downside to this change is that if many sub-interpreters are created and they create a lot of immortal interned strings, those strings will remain alive in the main interpreter state. So, it could cause a lot more memory to be used. My gut feeling is that's okay since we shouldn't be immortalizing strings unless we are fairly sure they won't cause memory bloat, even in a single interpreter case. E.g. it happens to variable names appearing in Python source code. |
Sorry, something went wrong.
There was a problem hiding this comment.
Mostly LGTM
I think there are just a couple spots where we need to clear the PyInterpreterState field for the shared interned dict case.
Sorry, something went wrong.
|
When you're done making the requested changes, leave the comment: I have made the requested changes; please review again. |
Sorry, something went wrong.
|
I have made the requested changes; please review again |
Sorry, something went wrong.
|
Thanks for making the requested changes! @ericsnowcurrently: please review the changes made to this pull request. |
Sorry, something went wrong.
There was a problem hiding this comment.
LGTM
Sorry, something went wrong.
⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️Hi! The buildbot AMD64 Arch Linux TraceRefs 3.12 has failed when building commit 5dd07eb. What do you need to do:
You can take a look at the buildbot page here: https://buildbot.python.org/#/builders/1197/builds/893 Failed tests:
Failed subtests:
Summary of the results of the build (if available): == Click to see traceback logsTracebackTests.test_broken_parent_from) ... ok
TracebackTests.test_broken_submodule) ... ok
TracebackTests.test_import_bug) ... ok
TracebackTests.test_syntax_error) ... ok
Traceback (most recent call last):
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 174, in test_subinterps_main
for run in self.run_repeated_init_and_subinterpreters():
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address : 0x7f22e05ffe80\nobject refcount : 0\nobject type : 0x55b39aff42c0\nobject type name: str\nobject repr : <refcnt 0 at 0x7f22e05ffe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x000055b39b168908)\n\nCurrent thread 0x00007f22e0989740 (most recent call first):\n <no Python frame>\n'
Traceback (most recent call last):
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 180, in test_subinterps_different_ids
for run in self.run_repeated_init_and_subinterpreters():
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address : 0x7f48cf32fe80\nobject refcount : 0\nobject type : 0x557a9680d2c0\nobject type name: str\nobject repr : <refcnt 0 at 0x7f48cf32fe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x0000557a96981908)\n\nCurrent thread 0x00007f48cf6b9740 (most recent call first):\n <no Python frame>\n'
TracebackTests.test_exec_failure_nested) ... ok
TracebackTests.test_nonexistent_module_nested) ... ok
Traceback (most recent call last):
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1897, in test_multi_init_extension_compat
self.check_compatible_fresh(module, strict=True)
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1817, in check_compatible_fresh
_, out, err = script_helper.assert_python_ok('-c', textwrap.dedent(f'''
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 166, in assert_python_ok
return _assert_python(True, *args, **env_vars)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 151, in _assert_python
res.fail(cmd_line)
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 76, in fail
raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is -11
command line: ['/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/python', '-X', 'faulthandler', '-I', '-c', '\nimport _testcapi, sys\nassert (\n \'_testmultiphase\' in sys.builtin_module_names or\n \'_testmultiphase\' not in sys.modules\n), repr(\'_testmultiphase\')\nret = _testcapi.run_in_subinterp_with_config(\n "\\nimport os, sys\\n\\ntry:\\n import _testmultiphase\\nexcept ImportError as exc:\\n text = \'ImportError: \' + str(exc)\\nelse:\\n text = \'okay\'\\nos.write(sys.stdout.fileno(), text.encode(\'utf-8\'))\\n",\n **{\'allow_fork\': False, \'allow_exec\': False, \'allow_threads\': True, \'allow_daemon_threads\': False, \'use_main_obmalloc\': True, \'gil\': 1, \'check_multi_interp_extensions\': True},\n)\nassert ret == 0, ret\n']
Traceback (most recent call last):
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1886, in test_single_init_extension_compat
self.check_incompatible_fresh(module)
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1841, in check_incompatible_fresh
_, out, err = script_helper.assert_python_ok('-c', textwrap.dedent(f'''
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 166, in assert_python_ok
return _assert_python(True, *args, **env_vars)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 151, in _assert_python
res.fail(cmd_line)
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 76, in fail
raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is -11
command line: ['/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/python', '-X', 'faulthandler', '-I', '-c', '\nimport _testcapi, sys\nassert \'_testsinglephase\' not in sys.modules, \'_testsinglephase\'\nret = _testcapi.run_in_subinterp_with_config(\n "\\nimport os, sys\\n\\ntry:\\n import _testsinglephase\\nexcept ImportError as exc:\\n text = \'ImportError: \' + str(exc)\\nelse:\\n text = \'okay\'\\nos.write(sys.stdout.fileno(), text.encode(\'utf-8\'))\\n",\n **{\'allow_fork\': False, \'allow_exec\': False, \'allow_threads\': True, \'allow_daemon_threads\': False, \'use_main_obmalloc\': True, \'gil\': 1, \'check_multi_interp_extensions\': True},\n)\nassert ret == 0, ret\n']
Traceback (most recent call last):
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 188, in test_subinterps_distinct_state
for run in self.run_repeated_init_and_subinterpreters():
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address : 0x7fef8cfbfe80\nobject refcount : 0\nobject type : 0x55da79f132c0\nobject type name: str\nobject repr : <refcnt 0 at 0x7fef8cfbfe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x000055da7a087908)\n\nCurrent thread 0x00007fef8d34b740 (most recent call first):\n <no Python frame>\n'
TracebackTests.test_broken_from) ... ok
TracebackTests.test_nonexistent_module) ... ok
Traceback (most recent call last):
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 174, in test_subinterps_main
for run in self.run_repeated_init_and_subinterpreters():
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address : 0x7fc0394bfe80\nobject refcount : 0\nobject type : 0x55c76866b2c0\nobject type name: str\nobject repr : <refcnt 0 at 0x7fc0394bfe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x000055c7687df908)\n\nCurrent thread 0x00007fc03984b740 (most recent call first):\n <no Python frame>\n'
Traceback (most recent call last):
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1945, in test_python_compat
self.check_compatible_fresh(module, strict=True)
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1817, in check_compatible_fresh
_, out, err = script_helper.assert_python_ok('-c', textwrap.dedent(f'''
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 166, in assert_python_ok
return _assert_python(True, *args, **env_vars)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 151, in _assert_python
res.fail(cmd_line)
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 76, in fail
raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is -11
command line: ['/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/python', '-X', 'faulthandler', '-I', '-c', '\nimport _testcapi, sys\nassert (\n \'threading\' in sys.builtin_module_names or\n \'threading\' not in sys.modules\n), repr(\'threading\')\nret = _testcapi.run_in_subinterp_with_config(\n "\\nimport os, sys\\n\\ntry:\\n import threading\\nexcept ImportError as exc:\\n text = \'ImportError: \' + str(exc)\\nelse:\\n text = \'okay\'\\nos.write(sys.stdout.fileno(), text.encode(\'utf-8\'))\\n",\n **{\'allow_fork\': False, \'allow_exec\': False, \'allow_threads\': True, \'allow_daemon_threads\': False, \'use_main_obmalloc\': True, \'gil\': 1, \'check_multi_interp_extensions\': True},\n)\nassert ret == 0, ret\n']
TracebackTests.test_unencodable_filename) ... ok
Traceback (most recent call last):
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 188, in test_subinterps_distinct_state
for run in self.run_repeated_init_and_subinterpreters():
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address : 0x7f14a6de80a0\nobject refcount : 0\nobject type : 0x5567decca2c0\nobject type name: str\nobject repr : <refcnt 0 at 0x7f14a6de80a0>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x00005567dee3e908)\n\nCurrent thread 0x00007f14a7168740 (most recent call first):\n <no Python frame>\n'
Traceback (most recent call last):
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 180, in test_subinterps_different_ids
for run in self.run_repeated_init_and_subinterpreters():
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address : 0x7f06ef537e80\nobject refcount : 0\nobject type : 0x5610d3a722c0\nobject type name: str\nobject repr : <refcnt 0 at 0x7f06ef537e80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x00005610d3be6908)\n\nCurrent thread 0x00007f06ef8be740 (most recent call first):\n <no Python frame>\n'
TracebackTests.test_broken_parent) ... ok
TracebackTests.test_exec_failure) ... ok
|
Sorry, something went wrong.
…terned strings. (pythongh-124541)" This reverts commit 5dd07eb.
|
GH-124814 is a backport of this pull request to the 3.12 branch. |
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
Fix a crash caused by immortal interned strings being shared between sub-interpreters that use basic single-phase init. In that case, the string can be used by an interpreter that outlives the interpeter that created and
interned it. For interpreters that share obmalloc state, also share the interned dict with the main interpreter.