gh-109653: Fix py312 regression in the import time of `random` by AlexWaygood · Pull Request #110221 · python/cpython · GitHub

As an optimisation to reduce the import time of the module, random first tries to import sha512 from the internal _sha512 module before falling back to hashlib. The problem, however, is that Python no longer has a _sha512 module! It was removed in 0b13575, by @gpshead. That means we're currently always falling back to the slow path in random.py, leading to the import time of random being far slower than it should be.

Importing sha512 from the correct module in the fast path cuts 60% off the import time of random.

Issue: Improve import time of various stdlib modules #109653

Given that this is a regression, should we backport it into 3.12?

Given that this is a regression, should we backport it into 3.12?

I was wondering that. I'd vote in favour of doing so, since it doesn't seem particularly high-risk to me. But I'd like to hear Raymond's and/or Greg's thoughts.

A backport to 3.12 would be reasonable.

…B0peK.rst

A backport to 3.12 would be reasonable.

Great, I've scheduled the backport. Thanks for the review!

Thanks @AlexWaygood for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

…110221) (cherry picked from commit 21a6263) Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

GH-110247 is a backport of this pull request to the 3.12 branch.

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Debian Clang LTO + PGO 3.x has failed when building commit 21a6263.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1084/builds/2183) and take a look at the build logs.
Check if the failure is related to this commit (21a6263) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1084/builds/2183

Failed tests:

test.test_concurrent_futures.test_shutdown

Failed subtests:

test_interpreter_shutdown - test.test_concurrent_futures.test_shutdown.ProcessPoolSpawnProcessPoolShutdownTest.test_interpreter_shutdown

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/var/lib/buildbot/workers/arm64-clang/3.x.gps-arm64-debian.clang.lto-pgo/build/Lib/test/test_concurrent_futures/test_shutdown.py", line 50, in test_interpreter_shutdown
    self.assertEqual(out.strip(), b"apple")
AssertionError: b'' != b'apple'

… (#110247) gh-109653: Fix regression in the import time of `random` in Python 3.12 (GH-110221) (cherry picked from commit 21a6263) Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

This code is technically awkward... It tried to speed up import time and did so by circumventing hashlib which means that it is loading and using the slowest possible sha512 implementation by default (hashlib will pick up openssl 3's accelerated sha512 support by default on most platforms and not use our builtin). so faster startup time for a slower runtime computation? thankfully this is only ever used by seed() which is a single/constant number of calls for most programs on tiny data so there is zero reason to care about sha512 performance for its purposes. The slower implementation may still be faster on small seed data anyways due to less setup overhead.

Nothing to do here. This works for random's purposes & thanks for the fixup. But this being a regression in the first place demonstrates how fragile direct use of internal details can be. (and indirectly how much in need of an overhaul hashlib.py could use)

I'd file an issue rather than leaving this comment in the merged PR void if there were anything concrete to describe and tackle, there isn't. :)

…0221)

Reduce the import time of random by 60%

a9e3194

AlexWaygood added the performance Performance or resource usage label Oct 2, 2023

AlexWaygood requested a review from rhettinger as a code owner October 2, 2023 14:45

bedevere-app Bot added the awaiting review label Oct 2, 2023

AlexWaygood changed the title Reduce the import time of random by 60% gh-109653: Reduce the import time of random by 60% Oct 2, 2023

bedevere-app Bot mentioned this pull request Oct 2, 2023

Improve import time of various stdlib modules #109653

Closed

AlexWaygood commented Oct 2, 2023

View reviewed changes

Comment thread Misc/NEWS.d/next/Library/2023-10-02-15-40-10.gh-issue-109653.iB0peK.rst Outdated

AlexWaygood requested a review from gpshead October 2, 2023 14:53

AlexWaygood added the stdlib Standard Library Python modules in the Lib/ directory label Oct 2, 2023

rhettinger approved these changes Oct 2, 2023

View reviewed changes

bedevere-app Bot added awaiting core review and removed awaiting review labels Oct 2, 2023

Update Misc/NEWS.d/next/Library/2023-10-02-15-40-10.gh-issue-109653.i…

e39e4dd

…B0peK.rst

AlexWaygood added the needs backport to 3.12 only security fixes label Oct 2, 2023

AlexWaygood enabled auto-merge (squash) October 2, 2023 22:30

AlexWaygood changed the title gh-109653: Reduce the import time of random by 60% gh-109653: Fix py312 regression in the import time of random Oct 2, 2023

AA-Turner approved these changes Oct 2, 2023

View reviewed changes

AlexWaygood merged commit 21a6263 into python:main Oct 2, 2023

AlexWaygood deleted the random-import-time branch October 2, 2023 22:56

bedevere-app Bot removed the awaiting core review label Oct 2, 2023

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Oct 2, 2023

pythongh-109653 : Reduce the import time of random by 60% (pythonGH-…

6277f18

…110221) (cherry picked from commit 21a6263) Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

bedevere-app Bot removed the needs backport to 3.12 only security fixes label Oct 2, 2023

This comment was marked as duplicate.

Sign in to view

gpshead reviewed Oct 3, 2023

View reviewed changes

Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024

pythongh-109653 : Reduce the import time of random by 60% (python#11…

c35196c

…0221)

gh-109653: Fix py312 regression in the import time of `random` by AlexWaygood · Pull Request #110221 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-109653: Fix py312 regression in the import time of random - #110221

gh-109653: Fix py312 regression in the import time of random#110221
AlexWaygood merged 2 commits into
python:mainfrom
AlexWaygood:random-import-time

AlexWaygood commented Oct 2, 2023 •
edited by bedevere-app Bot
Loading

Uh oh!

Uh oh!

Uh oh!

JelleZijlstra commented Oct 2, 2023

Uh oh!

AlexWaygood commented Oct 2, 2023 •
edited
Loading

Uh oh!

Uh oh!

rhettinger commented Oct 2, 2023

Uh oh!

AlexWaygood commented Oct 2, 2023

Uh oh!

miss-islington commented Oct 2, 2023

Uh oh!

bedevere-app Bot commented Oct 2, 2023

Uh oh!

This comment was marked as duplicate.

bedevere-bot commented Oct 2, 2023

Uh oh!

gpshead Oct 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

		@@ -65,7 +65,7 @@

		try:
		# hashlib is pretty heavy to load, try lean internal module first

gh-109653: Fix py312 regression in the import time of `random` by AlexWaygood · Pull Request #110221 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-109653: Fix py312 regression in the import time of random - #110221

gh-109653: Fix py312 regression in the import time of random#110221AlexWaygood merged 2 commits intopython:mainpython/cpython:mainfrom AlexWaygood:random-import-timeAlexWaygood/cpython:random-import-timeCopy head branch name to clipboard

Conversation

AlexWaygood commented Oct 2, 2023 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

JelleZijlstra commented Oct 2, 2023

Uh oh!

AlexWaygood commented Oct 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rhettinger commented Oct 2, 2023

Uh oh!

AlexWaygood commented Oct 2, 2023

Uh oh!

miss-islington commented Oct 2, 2023

Uh oh!

bedevere-app Bot commented Oct 2, 2023

Uh oh!

This comment was marked as duplicate.

bedevere-bot commented Oct 2, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

gpshead Oct 3, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

gh-109653: Fix py312 regression in the import time of random#110221
AlexWaygood merged 2 commits into
python:mainfrom
AlexWaygood:random-import-time

AlexWaygood commented Oct 2, 2023 •
edited by bedevere-app Bot
Loading

AlexWaygood commented Oct 2, 2023 •
edited
Loading