gh-156995: Fix `bytearray.take_bytes()` corrupting shared single-byte bytes objects by StanFromIreland · Pull Request #156996 · python/cpython · GitHub

Issue: bytearray.take_bytes() can corrupt shared single-byte bytes objects #156995

Double checking against implementation this change looks correct.

I think should add an assert as a safety net to help catch this bug into bytearray_reinit_from_bytes which is called every time self->ob_bytes_object is set to re-init.

bytearray_reinit_from_bytes(PyByteArrayObject *self, Py_ssize_t size,
                            Py_ssize_t alloc)
{
    /* Only the empty bytes may be immortal. */
    assert((alloc == 0) == _Py_IsImmortal(self->ob_bytes_object));

That safety net in free-threaded builds finds an additional case that can happen. _PyBytes_Resize uses _PyObject_IsUniquelyReferenced which under free threading includes that the object is owned by the current thread. If another thread swapped out the underlying bytes the unquiely reference returns false. if it is a downsize that calls PyBytes_FromStringAndSize which passes non-NULL to its first parameter...

Not sure the best way to fix that... I'd lean that _PyBytes_Resize should never return an immutable object for size >= 1 as that can lead to a really subtle only under free-threading bug...

Co-authored-by: Cody Maloney <cmaloney@theoreticalchaos.com>

Thanks for the review @cmaloney, and nice catch! I've applied both of your suggestions.

cc: @vstinner would be good to get more eyes on the _PyBytes_Resize part in particular as this would be nice to get into 3.15

I agree we should backport, I worry this can be quite nasty, and it's not an unrealistic scenario.

Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>

The overall change LGTM, but I would prefer to revert the _PyBytes_Resize() change from this PR.

Please ignore my previous comment: the _PyBytes_Resize() change is needed by this fix. test_bytes fails (in debug mode) without this change.

I suggest to complete the _PyBytes_Resize() C API to check that the result is a fresh object:

diff --git a/Lib/test/test_capi/test_bytes.py b/Lib/test/test_capi/test_bytes.py
index 38cda931e7d..590e118c04e 100644
--- a/Lib/test/test_capi/test_bytes.py
+++ b/Lib/test/test_capi/test_bytes.py
@@ -1,3 +1,4 @@
+import sys
 import unittest
 from test.support import import_helper
 
@@ -231,26 +232,41 @@ def test_decodeescape(self):
 
     def test_resize(self):
         """Test _PyBytes_Resize()"""
-        resize = _testcapi.bytes_resize
+        _resize = _testcapi.bytes_resize
+
+        def resize(obj, size, new):
+            result = _resize(obj, size, new)
+            if 1 <= len(result):
+                if new or size != len(obj):
+                    # gh-156995: Make sure that the result is a fresh object.
+                    # Previously, _PyBytes_Resize(&obj, 1) returned a singleton
+                    # if _PyObject_IsUniquelyReferenced() is false.
+                    self.assertEqual(sys.getrefcount(result), 1)
+                    self.assertFalse(sys._is_immortal(result))
+            else:
+                # check that the result is the empty bytes string singleton
+                self.assertTrue(sys._is_immortal(result))
+            return result
 
         for new in True, False:
-            self.assertEqual(resize(b'abc', 0, new), b'')
-            self.assertEqual(resize(b'abc', 1, new), b'a')
-            self.assertEqual(resize(b'abc', 2, new), b'ab')
-            self.assertEqual(resize(b'abc', 3, new), b'abc')
-            b = resize(b'abc', 4, new)
-            self.assertEqual(len(b), 4)
-            self.assertEqual(b[:3], b'abc')
-
-            self.assertEqual(resize(b'a', 0, new), b'')
-            self.assertEqual(resize(b'a', 1, new), b'a')
-            b = resize(b'a', 2, new)
-            self.assertEqual(len(b), 2)
-            self.assertEqual(b[:1], b'a')
-
-            self.assertEqual(resize(b'', 0, new), b'')
-            self.assertEqual(len(resize(b'', 1, new)), 1)
-            self.assertEqual(len(resize(b'', 2, new)), 2)
+            with self.subTest(new=new):
+                self.assertEqual(resize(b'abc', 0, new), b'')
+                self.assertEqual(resize(b'abc', 1, new), b'a')
+                self.assertEqual(resize(b'abc', 2, new), b'ab')
+                self.assertEqual(resize(b'abc', 3, new), b'abc')
+                b = resize(b'abc', 4, new)
+                self.assertEqual(len(b), 4)
+                self.assertEqual(b[:3], b'abc')
+
+                self.assertEqual(resize(b'a', 0, new), b'')
+                self.assertEqual(resize(b'a', 1, new), b'a')
+                b = resize(b'a', 2, new)
+                self.assertEqual(len(b), 2)
+                self.assertEqual(b[:1], b'a')
+
+                self.assertEqual(resize(b'', 0, new), b'')
+                self.assertEqual(len(resize(b'', 1, new)), 1)
+                self.assertEqual(len(resize(b'', 2, new)), 2)
 
         self.assertRaises(SystemError, resize, b'abc', -1, False)
         self.assertRaises(SystemError, resize, bytearray(b'abc'), 3, False)

Co-authored-by: Victor Stinner <vstinner@python.org>

LGTM. Nice fix. Modified code is well tested by new tests.

Thanks @StanFromIreland for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

GH-157260 is a backport of this pull request to the 3.15 branch.

…le-byte singletons (GH-156996) (#157260) Co-authored-by: Stan Ulbrych <stan@python.org> Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com> Co-authored-by: Victor Stinner <vstinner@python.org>

…e-byte singletons (python#156996) Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com> Co-authored-by: Victor Stinner <vstinner@python.org>

On Free Threading, _PyBytes_Resize(&obj, 1) no longer returns a single byte singleton if the current thread is different than the thread which created the object. Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com> Co-authored-by: Victor Stinner <vstinner@python.org> (cherry picked from commit fd569ea)

GH-157373 is a backport of this pull request to the 3.14 branch.

…157373) On Free Threading, _PyBytes_Resize(&obj, 1) no longer returns a single byte singleton if the current thread is different than the thread which created the object. Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com> Co-authored-by: Victor Stinner <vstinner@python.org> (cherry picked from commit fd569ea) Co-authored-by: Stan Ulbrych <stan@python.org> Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>

…on#156996) (python#157373) _PyBytes_Resize(&obj, 1) no longer returns a single byte singleton if the reference count is greater than 1. (cherry picked from commit fd569ea) Co-authored-by: Stan Ulbrych <stan@python.org> Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com> Co-authored-by: Victor Stinner <vstinner@python.org>

…157373) (#157436) [3.14] gh-156995: Fix _PyBytes_Resize() in Free Threading (#156996) (#157373) _PyBytes_Resize(&obj, 1) no longer returns a single byte singleton if the reference count is greater than 1. (cherry picked from commit fd569ea) Co-authored-by: Stan Ulbrych <stan@python.org> Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>

…e-byte singletons (python#156996) Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com> Co-authored-by: Victor Stinner <vstinner@python.org>

Fix bytearray.take_bytes() corrupting shared single-byte bytes objects

fad1503

StanFromIreland requested review from cmaloney and vstinner September 5, 2026 16:35

StanFromIreland added the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Sep 5, 2026

bedevere-app Bot mentioned this pull request Sep 5, 2026

bytearray.take_bytes() can corrupt shared single-byte bytes objects #156995

Closed

bedevere-app Bot added the awaiting core review label Sep 5, 2026

Fixup comment

a67dbe9

cmaloney reviewed Sep 5, 2026

View reviewed changes

Apply Cody's suggestions

125aa2a

Co-authored-by: Cody Maloney <cmaloney@theoreticalchaos.com>

cmaloney approved these changes Sep 5, 2026

View reviewed changes

cmaloney reviewed Sep 5, 2026

View reviewed changes

Comment thread Objects/bytearrayobject.c Outdated

Update Objects/bytearrayobject.c

b781cab

Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>

vstinner reviewed Sep 9, 2026

View reviewed changes

Comment thread Objects/bytesobject.c

vstinner reviewed Sep 9, 2026

View reviewed changes

Comment thread Objects/bytesobject.c

vstinner mentioned this pull request Sep 10, 2026

gh-157242: Fix PyBytesWriter_Resize() on MemoryError #157243

Merged

Apply @vstinner 's suggested tests

e9f3a4d

Co-authored-by: Victor Stinner <vstinner@python.org>

StanFromIreland requested a review from vstinner September 10, 2026 07:16

vstinner approved these changes Sep 10, 2026

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels Sep 10, 2026

vstinner merged commit fd569ea into python:main Sep 10, 2026
58 checks passed

bedevere-app Bot removed the awaiting merge label Sep 10, 2026

bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Sep 10, 2026

StanFromIreland deleted the bytearray.take_bytes branch September 10, 2026 15:30

gh-156995: Fix `bytearray.take_bytes()` corrupting shared single-byte bytes objects by StanFromIreland · Pull Request #156996 · python/cpython · GitHub

Repository navigation

gh-156995: Fix bytearray.take_bytes() corrupting shared single-byte bytes objects - #156996

gh-156995: Fix bytearray.take_bytes() corrupting shared single-byte bytes objects#156996
vstinner merged 5 commits into
python:mainfrom
StanFromIreland:bytearray.take_bytes

StanFromIreland commented Sep 5, 2026 •
edited by bedevere-app Bot
Loading

Uh oh!

Uh oh!

cmaloney left a comment •
edited
Loading

Uh oh!

Uh oh!

StanFromIreland commented Sep 5, 2026

Uh oh!

cmaloney left a comment

Uh oh!

StanFromIreland commented Sep 5, 2026

Uh oh!

Uh oh!

vstinner left a comment

Uh oh!

Uh oh!

vstinner left a comment

Uh oh!

Uh oh!

vstinner left a comment

Uh oh!

Uh oh!

miss-islington-app Bot commented Sep 10, 2026

Uh oh!

bedevere-app Bot commented Sep 10, 2026

Uh oh!

bedevere-app Bot commented Sep 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gh-156995: Fix `bytearray.take_bytes()` corrupting shared single-byte bytes objects by StanFromIreland · Pull Request #156996 · python/cpython · GitHub

Repository navigation

gh-156995: Fix bytearray.take_bytes() corrupting shared single-byte bytes objects - #156996

gh-156995: Fix bytearray.take_bytes() corrupting shared single-byte bytes objects#156996vstinner merged 5 commits intopython:mainpython/cpython:mainfrom StanFromIreland:bytearray.take_bytesStanFromIreland/cpython:bytearray.take_bytesCopy head branch name to clipboard

Conversation

StanFromIreland commented Sep 5, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmaloney left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

StanFromIreland commented Sep 5, 2026

Uh oh!

cmaloney left a comment

Choose a reason for hiding this comment

Uh oh!

StanFromIreland commented Sep 5, 2026

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-app Bot commented Sep 10, 2026

Uh oh!

bedevere-app Bot commented Sep 10, 2026

Uh oh!

bedevere-app Bot commented Sep 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gh-156995: Fix bytearray.take_bytes() corrupting shared single-byte bytes objects#156996
vstinner merged 5 commits into
python:mainfrom
StanFromIreland:bytearray.take_bytes

StanFromIreland commented Sep 5, 2026 •
edited by bedevere-app Bot
Loading

cmaloney left a comment •
edited
Loading