bpo-29514: Check magic number for bugfix release by appeltel · Pull Request #54 · python/cpython · GitHub

Add a test to check the current MAGIC_NUMBER against the
expected number for the release if the current release is
at candidate or final level. On test failure, describe to
the developer the procedure for changing the magic number.

https://bugs.python.org/issue29514

Add a dict importlib.util.EXPECTED_MAGIC_NUMBERS which details the initial and expected pyc magic number for each minor release. This gives a mechanism for users to check if the magic number has changed within a release and for a test to ensure procedure is followed if a change is necessary. Add a test to check the current MAGIC_NUMBER against the expected number for the release if the current release is at candidate or final level. On test failure, describe to the developer the procedure for changing the magic number.

The general approach in the test case looks like it will give affected maintainers useful guidance, but the impact can be reduced by moving the EXPECTED_MAGIC_NUMBERS table into the test case itself (the test case will complain when it needs updating, so it's OK to have it separated from the list of actual magic numbers in the importlib code)

I'd put this table in the test suite rather than in the code - there's no need to have it other than the bytecode stability test case, and if it hasn't been updated when it needs to be we'll get a test failure anyway.

Simplifed to a sigle entry in the test case in 9ea7c52

With the table moved to the test suite, this won't be needed any more.

removed in 9ea7c52

dev-python -> python-dev

fixed in 9ea7c52

Does this mean we're reverting the magic number bump from 3.5.3?

No, but the current special case from the test should just be moved directly into the table of expected magic numbers with a suitable comment.

Simplify the magic number release test by removing EXPECTED_MAGIC_NUMBERS table and making the expected magic number self-contained within the test. BPO: 29514

This is looking pretty close now, just a couple of suggestions around details of the test execution.

Inherit from unittest.TestCase here

I initially avoided this and followed the test machinery as I wasn't sure about directly importing importlib.util within this test module, but after looking at the way the test specialization works I don't see how this is an issue. Changed in 1e32a1b.

Yeah, this module does some interesting things to create "real" test classes in different configurations, but we don't need those kinds of tricks for the new test.

This can change to a unittest.skipUnless decorator on the testmethod:

@unittest.skipUnless(sys.version_info.release in ('final', 'release'), 'Only check magic number stability for release candidates and later')

Done in 1e32a1b

I'd start this message with "To avoid breaking backwards compatibility with cached bytecode files that can't be automatically regenerated by the current user, ..."

Added in 1e32a1b

Improve the execution of the magic number test by using skipUnless for alpha and beta releases, and directly inheriting from unittest.TestCase rather than using the machinery for the other tests. Also improve the error message to explain the reason for caution in changing the magic number. BPO: 29514

Codecov Report

Merging #54 into master will decrease coverage by <.01%.
The diff coverage is 50%.

@@            Coverage Diff             @@
##           master      #54      +/-   ##
==========================================
- Coverage   82.37%   82.37%   -0.01%     
==========================================
  Files        1427     1428       +1     
  Lines      350948   350968      +20     
==========================================
+ Hits       289093   289102       +9     
- Misses      61855    61866      +11

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c22bfaa...1e32a1b. Read the comment docs.

Wouldn't be better to use a multiline string literal if you want such verbose error message?

But I think that "broken *.pyc files compatibility" would be enough.

Then you'd have to dedent it, etc. This message is perfectly legible to me.

I think it would be better to write this as a comment rather than a docstring. If you want, add a one-line docstring, but most test methods don't have docstrings.

🐍🍒⛏🤖 Thanks @appeltel for the PR, and @ambv for merging it 🌮🎉.I'm working now to backport this PR to: 2.7, 3.6.

Sorry @appeltel and @ambv, I had trouble checking out the 3.6 backport branch.
Please backport using cherry_picker on command line.

Sorry, @appeltel and @ambv, I could not cleanly backport this to 2.7 due to a conflict.
Please backport using cherry_picker on command line.

This was backported a while back, but I missed updating the labels:

2.7:#2160
3.5: #2158
3.6: #2157

The versions checked here are 20 years old. Also dllwrap has started to emit a deprecation warning in the latest release spamming the build logs. Fixes python#54

the-knights-who-say-ni added the CLA signed label Feb 12, 2017

ncoghlan requested changes Feb 12, 2017

View reviewed changes

ambv reviewed Feb 13, 2017

View reviewed changes

Simplify magic number release test

9ea7c52

Simplify the magic number release test by removing EXPECTED_MAGIC_NUMBERS table and making the expected magic number self-contained within the test. BPO: 29514

ncoghlan requested changes Mar 12, 2017

View reviewed changes

ncoghlan approved these changes Mar 13, 2017

View reviewed changes

ncoghlan added needs backport to 2.7 labels Mar 13, 2017

serhiy-storchaka reviewed Apr 2, 2017

View reviewed changes

ambv merged commit d6d344d into python:master Apr 17, 2017

Mariatta added needs backport to 3.6 and removed needs backport to 3.6 labels Sep 7, 2017

ncoghlan removed needs backport to 2.7 labels Sep 7, 2017

mgmacias95 pushed a commit to mgmacias95/cpython that referenced this pull request Mar 27, 2023

Merge pull request python#54 from sunmy2019/handle-invalid-expressions

30ea09f

oraluben pushed a commit to oraluben/cpython that referenced this pull request Jul 12, 2023

feat(ci): cache llvm installs across CI (python#54 )

454b349

gvanrossum mentioned this pull request Aug 22, 2023

heap-use-after-free in _PyFunction_LookupByVersion #108253

Closed

LinanV mentioned this pull request Oct 23, 2023

Objects/typeobject.c: No such file or directory. #111203

Closed

kcatss mentioned this pull request Feb 20, 2024

Use After Free at _heapreplace_max #115706

Closed

ngoldbaum mentioned this pull request Sep 23, 2024

Crash running PyO3 tests with --test-threads=1000 #124375

Closed

This was referenced Feb 11, 2025

segmentfault when pip installing setuptools #129992

Closed

segmentfault when pip installing setuptools #129993

Closed

sergey-miryanov mentioned this pull request Dec 18, 2025

Observed memory leak in ssl library: Python 3.14 GC issue #142516

Closed

ngoldbaum mentioned this pull request Mar 6, 2026

Remove deprecated and undocumented function ctypes.SetPointerType #133866

Closed

               from ._bootstrap import _resolve_name
               from ._bootstrap import spec_from_loader
               from ._bootstrap import _find_spec
+              from ._bootstrap_external import EXPECTED_MAGIC_NUMBERS

+                      adjustment to this test case.
+                      """
+                      if sys.version_info.releaselevel not in ('final', 'candidate'):
+                          return

+                  (2, 7): 62211,
+                  (3, 5): 3350,
+                  (3, 6): 3379
+              }

+                      In exceptional cases, it may be required to change the MAGIC_NUMBER
+                      for a maintenance release. In this case the change should be
+                      discussed in dev-python. If a change is required, community

		) = util.test_both(PEP3147Tests, util=importlib_util)


		class MagicNumberTests:

+                      actual = int.from_bytes(self.util.MAGIC_NUMBER[:2], 'little')
+                      msg = (
+                          "Candidate and final releases require the current  "

+                      actual = int.from_bytes(importlib.util.MAGIC_NUMBER[:2], 'little')
+                      msg = (
+                          "To avoid breaking backwards compatibility with cached bytecode "

+                  )
+                  def test_magic_number(self):
+                      """
+                      Each python minor release should generally have a MAGIC_NUMBER

bpo-29514: Check magic number for bugfix release by appeltel · Pull Request #54 · python/cpython · GitHub

Uh oh!

Uh oh!

bpo-29514: Check magic number for bugfix release - #54

bpo-29514: Check magic number for bugfix release#54ambv merged 3 commits intopython:masterpython/cpython:masterfrom appeltel:bpo-29514-magic-numbersappeltel/cpython:bpo-29514-magic-numbersCopy head branch name to clipboard

Conversation

appeltel commented Feb 12, 2017 • edited by ncoghlan Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ncoghlan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ncoghlan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Mar 12, 2017

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miss-islington commented Sep 7, 2017

Uh oh!

miss-islington commented Sep 7, 2017

Uh oh!

miss-islington commented Sep 7, 2017

Uh oh!

ncoghlan commented Sep 7, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

bpo-29514: Check magic number for bugfix release#54
ambv merged 3 commits into
python:masterfrom
appeltel:bpo-29514-magic-numbers

appeltel commented Feb 12, 2017 •
edited by ncoghlan
Loading