gh-125996: fix thread safety of ordered dict by kumaraditya303 · Pull Request #133734 · python/cpython · GitHub

Issue: nogil segmentation fault on ordered dict operations #125996

Would you please also provide some timing measurements for basic operations?

Would you please also provide some timing measurements for basic operations?

I added some benchmark for get/set operation to ftscaling and benchmarked it:

index 1a59e25189d..20f15200051 100644
--- a/Tools/ftscalingbench/ftscalingbench.py
+++ b/Tools/ftscalingbench/ftscalingbench.py
@@ -28,6 +28,7 @@
 import threading
 import time
 from operator import methodcaller
+from collections import OrderedDict
 
 # The iterations in individual benchmarks are scaled by this factor.
 WORK_SCALE = 100
@@ -43,6 +44,19 @@ def register_benchmark(func):
     ALL_BENCHMARKS[func.__name__] = func
     return func
 
+@register_benchmark
+def odict_get():
+    tab = OrderedDict((i, i) for i in range(100))
+    for i in range(1000 * WORK_SCALE):
+        tab.get(i % 100)
+
+@register_benchmark
+def odict_set():
+    tab = OrderedDict((i, i) for i in range(100))
+    for i in range(1000 * WORK_SCALE):
+        tab[i % 100] = i
+
+
 @register_benchmark
 def object_cfunction():
     accu = 0

Results:

❯ ./python.exe Tools/ftscalingbench/ftscalingbench.py odict_get odict_set
Running benchmarks with 10 threads
odict_get                  4.7x faster
odict_set                  4.4x faster

LGTM

Co-authored-by: Sam Gross <colesbury@gmail.com>

Thanks @kumaraditya303 for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

Sorry, @kumaraditya303, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 6481539a6d9692ddf13ab01baad1bc9133409413 3.14

GH-140053 is a backport of this pull request to the 3.14 branch.

pythonGH-133734) (cherry picked from commit 6481539) Co-authored-by: Kumar Aditya <kumaraditya@python.org>

…133734) (#140053) (cherry picked from commit 6481539)

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 FreeBSD Refleaks 3.14 (tier-3) has failed when building commit 5110780.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1800/builds/552) and take a look at the build logs.
Check if the failure is related to this commit (5110780) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1800/builds/552

Failed tests:

test.test_concurrent_futures.test_interpreter_pool

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1371, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1333, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1267, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1296, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1270, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1388, in find_spec
  File "<frozen importlib._bootstrap_external>", line 166, in _path_isfile
  File "<frozen importlib._bootstrap_external>", line 158, in _path_is_mode_type
  File "<frozen importlib._bootstrap_external>", line 152, in _path_stat
ValueError: embedded null byte
Exception in initializer:
RuntimeError: Failed to import encodings module


Traceback (most recent call last):
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/concurrent/futures/thread.py", line 99, in _worker
    ctx.initialize()
    ~~~~~~~~~~~~~~^^
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/concurrent/futures/interpreter.py", line 62, in initialize
    self.interp = interpreters.create()
                  ~~~~~~~~~~~~~~~~~~~^^
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/concurrent/interpreters/__init__.py", line 65, in create
    id = _interpreters.create(reqrefs=True)
concurrent.interpreters.InterpreterError: interpreter creation failed
Timeout (0:45:00)!
Thread 0x000000084442aa00 [InterpreterPoolExec] (most recent call first):
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/concurrent/interpreters/__init__.py", line 220 in _call
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/concurrent/interpreters/__init__.py", line 238 in call
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/concurrent/futures/interpreter.py", line 85 in run
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/concurrent/futures/thread.py", line 86 in run
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/concurrent/futures/thread.py", line 119 in _worker
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/threading.py", line 1023 in run
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/threading.py", line 1081 in _bootstrap_inner
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/threading.py", line 1043 in _bootstrap


Traceback (most recent call last):
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/concurrent/futures/thread.py", line 99, in _worker
    ctx.initialize()
    ~~~~~~~~~~~~~~^^
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/concurrent/futures/interpreter.py", line 62, in initialize
    self.interp = interpreters.create()
                  ~~~~~~~~~~~~~~~~~~~^^
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/concurrent/interpreters/__init__.py", line 65, in create
    id = _interpreters.create(reqrefs=True)
concurrent.interpreters.InterpreterError: interpreter creation failed
Timeout (0:45:00)!
Thread 0x00000008415fb800 [InterpreterPoolExec] (most recent call first):
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/concurrent/futures/thread.py", line 116 in _worker
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/threading.py", line 1023 in run
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/threading.py", line 1081 in _bootstrap_inner
  File "/buildbot/buildarea/3.14.ware-freebsd.refleak/build/Lib/threading.py", line 1043 in _bootstrap

bedevere-app Bot mentioned this pull request May 9, 2025

nogil segmentation fault on ordered dict operations #125996

Closed

kumaraditya303 marked this pull request as ready for review May 13, 2025 09:08

kumaraditya303 requested review from markshannon and methane as code owners May 13, 2025 09:08

bedevere-app Bot added the awaiting core review label May 13, 2025

kumaraditya303 requested a review from colesbury May 13, 2025 09:08

picnixz reviewed May 17, 2025

View reviewed changes

Comment thread Include/internal/pycore_dict.h Outdated

Comment thread Objects/odictobject.c Outdated

Comment thread Objects/odictobject.c Outdated

Comment thread Objects/odictobject.c Outdated

Comment thread Objects/odictobject.c Outdated

Comment thread Objects/odictobject.c Outdated

colesbury reviewed May 27, 2025

View reviewed changes

Comment thread Objects/odictobject.c Outdated

Comment thread Objects/odictobject.c Outdated

Comment thread Objects/odictobject.c Outdated

kumaraditya303 added the topic-free-threading label Jun 23, 2025

kumaraditya303 and others added 5 commits July 5, 2025 17:23

fix thread safety of ordered dict

1373660

📜🤖 Added by blurb_it.

72ba12a

fmt

bbd7487

use lock held clear of dict

ef2aed1

remove unused function

473cfa0

kumaraditya303 force-pushed the ordereddict branch from e87cd54 to 473cfa0 Compare July 5, 2025 11:53

kumaraditya303 added 2 commits July 5, 2025 18:22

use critical section on iterator

4bf573e

use PyDict_Clear

9d1ae53

Merge remote-tracking branch 'upstream/main' into ordereddict

587dae8

kumaraditya303 added the needs backport to 3.14 bugs and security fixes label Oct 8, 2025

kumaraditya303 requested a review from colesbury October 8, 2025 14:19

methane approved these changes Oct 9, 2025

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels Oct 9, 2025

colesbury approved these changes Oct 13, 2025

View reviewed changes

Comment thread Objects/odictobject.c Outdated

kumaraditya303 and others added 2 commits October 13, 2025 22:21

Update Objects/odictobject.c

4d8e596

Co-authored-by: Sam Gross <colesbury@gmail.com>

Merge branch 'main' into ordereddict

22b3b72

kumaraditya303 merged commit 6481539 into python:main Oct 13, 2025
45 checks passed

bedevere-app Bot removed the awaiting merge label Oct 13, 2025

kumaraditya303 deleted the ordereddict branch October 13, 2025 17:25

miss-islington-app Bot assigned kumaraditya303 Oct 13, 2025

bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Oct 13, 2025

kumaraditya303 added a commit to kumaraditya303/cpython that referenced this pull request Oct 13, 2025

[3.14] pythongh-125996 : fix thread safety of collections.OrderedDict (

0d55084

pythonGH-133734) (cherry picked from commit 6481539) Co-authored-by: Kumar Aditya <kumaraditya@python.org>

kumaraditya303 added a commit that referenced this pull request Oct 13, 2025

[3.14] gh-125996 : fix thread safety of collections.OrderedDict (GH-…

5110780

…133734) (#140053) (cherry picked from commit 6481539)

colesbury mentioned this pull request Dec 15, 2025

thread-safety issue in pure-Python zoneinfo cache on 3.13t #142763

Closed

robertsdotpm mentioned this pull request Jun 29, 2026

OrderedDict: iterator constructor odictiter_new is unlocked under free-threading -- use-after-free SIGSEGV (GH-133734 follow-up to gh-125996) #152537

Closed

gh-125996: fix thread safety of ordered dict by kumaraditya303 · Pull Request #133734 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-125996: fix thread safety of ordered dict - #133734

gh-125996: fix thread safety of ordered dict#133734kumaraditya303 merged 10 commits intopython:mainpython/cpython:mainfrom kumaraditya303:ordereddictkumaraditya303/cpython:ordereddictCopy head branch name to clipboard

Conversation

kumaraditya303 commented May 9, 2025 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

colesbury left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kumaraditya303 commented Jul 5, 2025

Uh oh!

colesbury left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

miss-islington-app Bot commented Oct 13, 2025

Uh oh!

miss-islington-app Bot commented Oct 13, 2025

Uh oh!

bedevere-app Bot commented Oct 13, 2025

Uh oh!

bedevere-bot commented Oct 13, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gh-125996: fix thread safety of ordered dict#133734
kumaraditya303 merged 10 commits into
python:mainfrom
kumaraditya303:ordereddict

kumaraditya303 commented May 9, 2025 •
edited by bedevere-app Bot
Loading