| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
|
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead. |
Sorry, something went wrong.
After PR pythongh-149583 (Fix double evaluation of .pth and .site files in venvs), .pth files are no longer loaded in subprocesses started with subprocess.run([sys.executable, ...]). The root cause: main() seeds known_paths from removeduppaths() with all sys.path entries inherited from the parent process. addsitedir() then skips .pth processing for every directory already in known_paths. Fix: - main(): call removeduppaths() for dedup but start known_paths as a fresh empty set, so that addsitedir() processes .pth files in every site-packages directory regardless of inherited sys.path. - addsitedir(): move known_paths.add() before the sys.path.append and guard the append with 'sitedir not in sys.path' to avoid creating duplicate entries when called with a fresh known_paths. This preserves the pythongh-75723 dedup guarantee while allowing subprocesses to load .pth files.
|
Before I review this, I think we still need a reproducer. See this comment where it works for me. |
Sorry, something went wrong.
Why the venv script missed the bugThe original bug (gh-149819) requires the parent directory of a .pth file to be inherited via the PYTHONPATH environment variable. When a child process boots up in that environment, site.main() automatically ingests PYTHONPATH into its baseline tracking set (known_paths). Because the directory is already marked as "known," the unpatched addsitedir() function mistakenly skips parsing its .pth files. ReproductionReproduction scriptimport os
import sys
import tempfile
import subprocess
import shutil
def main():
print(f'Running on Python version: {sys.version}')
# *** Create a temporary 'site-packages' layout
temp_dir = tempfile.mkdtemp(prefix='repro_site_')
try:
# A mock package directory that our .pth file will dynamically add to sys.path
mock_pkg_dir = os.path.join(temp_dir, 'my_hidden_package')
os.makedirs(mock_pkg_dir)
# Create a simple module inside the hidden package directory
with open(os.path.join(mock_pkg_dir, 'triggered.py'), 'w') as f:
f.write("def test(): return 'Success: Subprocess loaded .pth file!'\n")
# Create the .pth file inside our temp_dir
# It must contain code or paths to alter sys.path when site.py executes
pth_file_path = os.path.join(temp_dir, 'test_loader.pth')
with open(pth_file_path, 'w') as f:
# .pth files can execute code if prefixed with 'import '
f.write(f'import sys; sys.path.append({repr(mock_pkg_dir)})\n')
print(f'Created temporary site-dir at: {temp_dir}\nCreated .pth file executing code injection pointing to hidden package.')
# *** Prepare the environment for the subprocess
# We need to prepend our temporary directory to PYTHONPATH so the
# subprocess inherits this directory in its initial sys.path.
env = os.environ.copy()
existing_pythonpath = env.get('PYTHONPATH', '')
if existing_pythonpath:
env['PYTHONPATH'] = f'{temp_dir}{os.pathsep}{existing_pythonpath}'
else:
env['PYTHONPATH'] = temp_dir
# Inline code for the subprocess to execute.
# It will try to import the module that only exists via the .pth file injection.
subprocess_code = """if True:
import sys
try:
import triggered
print(triggered.test())
sys.exit(0)
except ModuleNotFoundError as e:
print('BUG TRIGGERED: ModuleNotFoundError - .pth file was NOT executed!', file=sys.stderr)
sys.exit(1)
"""
print('\n--- Spawning Python Subprocess ---')
# *** Spawn the subprocess using the current executable
result = subprocess.run(
[sys.executable, '-c', subprocess_code],
env=env,
capture_output=True,
text=True,
)
# *** Evaluate the outcome
print(f'Subprocess exit code: {result.returncode}')
if result.stdout:
print(f'Subprocess STDOUT:\n{result.stdout.strip()}')
if result.stderr:
print(f'Subprocess STDERR:\n{result.stderr.strip()}')
# Assertions to cleanly flag bug vs fixed behavior
if 1 == result.returncode:
print('\n❌ [RESULT] Bug reproduced successfully! The subprocess failed to process the .pth file.')
sys.exit(1)
elif 0 == result.returncode and 'Success' in result.stdout:
print('\n✅ [RESULT] Test passed! The .pth file was loaded properly (The patch is likely active).')
sys.exit(0)
else:
print('\n⚠️ [RESULT] Unexpected subprocess outcome.')
sys.exit(result.returncode)
finally:
# Cleanup temporary files
shutil.rmtree(temp_dir)
if __name__ == '__main__':
main()Regression testTest caseimport os
import sys
import unittest
from test import support
from test.support import os_helper, script_helper
class PthSubprocessTests(unittest.TestCase):
def setUp(self):
super().setUp()
# Establish an isolated target directory for test files
self.site_dir = os.path.abspath(os_helper.TESTFN_ASCII)
self.addCleanup(os_helper.rmtree, self.site_dir)
os.makedirs(self.site_dir)
def test_pth_file_loaded_in_subprocess(self):
# gh-149819: Ensure .pth files evaluate inside child processes
# when their containing directory is passed down via PYTHONPATH.
hidden_dir = os.path.join(self.site_dir, 'hidden_package')
os.makedirs(hidden_dir)
module_path = os.path.join(hidden_dir, 'triggered_module.py')
with open(module_path, 'w', encoding='utf-8') as f:
f.write('PTH_WAS_PROCESSED = True\n')
pth_path = os.path.join(self.site_dir, 'test_gh149819.pth')
with open(pth_path, 'w', encoding='utf-8') as f:
f.write(f'import sys; sys.path.append({repr(hidden_dir)})\n')
script = """if True:
import sys
try:
import triggered_module
sys.exit(0 if triggered_module.PTH_WAS_PROCESSED else 1)
except ImportError:
sys.exit(2)
"""
env = os.environ.copy()
pythonpath = env.get('PYTHONPATH', '')
if pythonpath:
env['PYTHONPATH'] = f'{self.site_dir}{os.pathsep}{pythonpath}'
else:
env['PYTHONPATH'] = self.site_dir
rc, out, err = script_helper.assert_python_ok('-c', script, **env)
self.assertEqual(rc, 0, msg=f'Stderr: {err.decode()}')
def tearDownModule():
support.reap_children()
if __name__ == '__main__':
unittest.main()Referenced venv scriptvenv sh script#!/bin/sh
#
# Usage:
# ./149819.sh # uses ./python.exe
# ./149819.sh /usr/bin/python3 # uses a specific interpreter
PYTHON=${1:-./python.exe}
VENV=$(mktemp -d)
"$PYTHON" -m venv "$VENV"
SITE=$("$VENV"/bin/python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')
cat > "$SITE"/pp.pth <<EOF
import sys; print('pp.pth')
EOF
"$VENV/bin/python" -c "import sys, subprocess; print('parent'); subprocess.run([sys.executable, '-c', 'print(\'child\')'])"
Footnotes
|
Sorry, something went wrong.
|
Thanks for the details @tcely - I've reproduced the problem, and I think it's legit. I'm working on a refinement of your branch to include a test case, etc. Please stay tuned. |
Sorry, something went wrong.
I put together a test case, but it might not be exactly what's needed. By the way, I'm not the owner of this branch. I simply responded to your request for a reproduction. Thanks for looking into this! |
Sorry, something went wrong.
Oops! Thanks go to @IntentBug. |
Sorry, something went wrong.
|
Please review the latest linked PR |
Sorry, something went wrong.
|
#150177 was merged and backported, so this should now be fixed in main and 3.15. You should have been attributed in that merge. Thanks again! |
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
After PR gh-149583 (Fix double evaluation of .pth and .site files in venvs), .pth files are no longer loaded in subprocesses started with subprocess.run([sys.executable, ...]).
Root cause: main() seeds known_paths from removeduppaths() with all sys.path entries inherited from the parent process. addsitedir() then skips .pth processing for every directory already in known_paths.
Fix (2 changes in Lib/site.py, +10/−6 lines):
Verification:
Closes gh-149819