gh-151722: Defer GC tracking of frozendict to end of construction by corona10 · Pull Request #151740 · python/cpython · GitHub

Issue: Data race on frozendict reads during construction in the free-threading build #151722

TSAN script

import threading, gc

N = 4000
stop = threading.Event()
box = []

def reader():
    while not stop.is_set():
        if box:
            o = box[0]
            try:
                len(o); hash(o); repr(o)
            except Exception:
                pass

class Evil:
    def __init__(self):
        self.n = 0
    def keys(self):
        return [f"k{i}" for i in range(N)]
    def __getitem__(self, k):
        self.n += 1
        if self.n == 10 and not box:        # one scan/construction; k0..k8 present
            for o in gc.get_objects():
                if type(o) is frozendict and "k0" in o and len(o) < N:
                    box.append(o)
                    break
        return 1

t = threading.Thread(target=reader, daemon=True)
t.start()
for _ in range(40):
    box.clear()
    frozendict(Evil())
stop.set()
t.join(timeout=2)
print("observed half-built:", bool(box))

AS-IS

➜  cpython git:(main) ✗ TSAN_OPTIONS="halt_on_error=0" ./python.exe repro_151722.py
==================
WARNING: ThreadSanitizer: data race (pid=56255)
  Atomic write of size 8 at 0x00012a0e63d8 by main thread:
    #0 insert_combined_dict dictobject.c:1906 (python.exe:arm64+0x1000cca38)
    #1 insertdict dictobject.c:2005 (python.exe:arm64+0x1000bcb84)
    #2 setitem_take2_lock_held dictobject.c:2785 (python.exe:arm64+0x1000bbf38)
    #3 dict_merge dictobject.c:4280 (python.exe:arm64+0x1000bfd00)
    #4 frozendict_vectorcall dictobject.c:5331 (python.exe:arm64+0x1000cc248)
    #5 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100061c10)

  Previous read of size 8 at 0x00012a0e63d8 by thread T1:
    #0 _PyDict_Next dictobject.c:3166 (python.exe:arm64+0x1000be214)
    #1 frozendict_hash dictobject.c:8339 (python.exe:arm64+0x1000cbef4)
    #2 builtin_hash bltinmodule.c:1869 (python.exe:arm64+0x1001c792c)
    #3 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10027a4ec)

  Thread T1 (tid=4632788, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x33834)
    #1 do_start_joinable_thread thread_pthread.h:281 (python.exe:arm64+0x100279b6c)
    #2 do_start_new_thread _threadmodule.c:1919 (python.exe:arm64+0x100332da4)
    #3 thread_PyThread_start_joinable_thread _threadmodule.c:2042 (python.exe:arm64+0x100331f0c)
    #4 cfunction_call methodobject.c:564 (python.exe:arm64+0x1000de0b8)

SUMMARY: ThreadSanitizer: data race dictobject.c:1906 in insert_combined_dict
==================
==================
WARNING: ThreadSanitizer: data race (pid=56255)
  Read of size 8 at 0x00012a0e6488 by thread T1:
    #0 _PyDict_Next dictobject.c:3171 (python.exe:arm64+0x1000be260)
    #1 anydict_repr_impl dictobject.c:3621 (python.exe:arm64+0x1000ce668)
    #2 frozendict_repr dictobject.c:8271 (python.exe:arm64+0x1000cbdb0)
    #3 PyObject_Repr object.c:784 (python.exe:arm64+0x1000e46a0)
    #4 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10027a4ec)

  Previous atomic write of size 8 at 0x00012a0e6488 by main thread:
    #0 insert_combined_dict dictobject.c (python.exe:arm64+0x1000cc9e8)
    #1 insertdict dictobject.c:2005 (python.exe:arm64+0x1000bcb84)
    #2 setitem_take2_lock_held dictobject.c:2785 (python.exe:arm64+0x1000bbf38)
    #3 dict_merge dictobject.c:4280 (python.exe:arm64+0x1000bfd00)
    #4 frozendict_vectorcall dictobject.c:5331 (python.exe:arm64+0x1000cc248)
    #5 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100061c10)

  Thread T1 (tid=4632788, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x33834)
    #1 do_start_joinable_thread thread_pthread.h:281 (python.exe:arm64+0x100279b6c)
    #2 do_start_new_thread _threadmodule.c:1919 (python.exe:arm64+0x100332da4)
    #3 thread_PyThread_start_joinable_thread _threadmodule.c:2042 (python.exe:arm64+0x100331f0c)
    #4 cfunction_call methodobject.c:564 (python.exe:arm64+0x1000de0b8)

SUMMARY: ThreadSanitizer: data race dictobject.c:3171 in _PyDict_Next
==================
==================
WARNING: ThreadSanitizer: data race (pid=56255)
  Atomic write of size 8 at 0x00012a74ff80 by main thread:
    #0 dictresize dictobject.c:2279 (python.exe:arm64+0x1000cd9b4)
    #1 insert_combined_dict dictobject.c:1881 (python.exe:arm64+0x1000cc84c)
    #2 insertdict dictobject.c:2005 (python.exe:arm64+0x1000bcb84)
    #3 setitem_take2_lock_held dictobject.c:2785 (python.exe:arm64+0x1000bbf38)
    #4 dict_merge dictobject.c:4280 (python.exe:arm64+0x1000bfd00)
    #5 frozendict_vectorcall dictobject.c:5331 (python.exe:arm64+0x1000cc248)
    #6 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100061c10)

  Previous read of size 8 at 0x00012a74ff80 by thread T1:
    #0 _PyDict_Next dictobject.c:3166 (python.exe:arm64+0x1000be208)
    #1 anydict_repr_impl dictobject.c:3621 (python.exe:arm64+0x1000ce668)
    #2 frozendict_repr dictobject.c:8271 (python.exe:arm64+0x1000cbdb0)
    #3 PyObject_Repr object.c:784 (python.exe:arm64+0x1000e46a0)
    #4 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10027a4ec)

  Thread T1 (tid=4632788, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x33834)
    #1 do_start_joinable_thread thread_pthread.h:281 (python.exe:arm64+0x100279b6c)
    #2 do_start_new_thread _threadmodule.c:1919 (python.exe:arm64+0x100332da4)
    #3 thread_PyThread_start_joinable_thread _threadmodule.c:2042 (python.exe:arm64+0x100331f0c)
    #4 cfunction_call methodobject.c:564 (python.exe:arm64+0x1000de0b8)

SUMMARY: ThreadSanitizer: data race dictobject.c:2279 in dictresize
==================
==================
WARNING: ThreadSanitizer: data race (pid=56255)
  Read of size 8 at 0x00012a0e6480 by thread T1:
    #0 _PyDict_Next dictobject.c:3177 (python.exe:arm64+0x1000be2b8)
    #1 anydict_repr_impl dictobject.c:3621 (python.exe:arm64+0x1000ce668)
    #2 frozendict_repr dictobject.c:8271 (python.exe:arm64+0x1000cbdb0)
    #3 PyObject_Repr object.c:784 (python.exe:arm64+0x1000e46a0)
    #4 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10027a4ec)

  Previous atomic write of size 8 at 0x00012a0e6480 by main thread:
    #0 insert_combined_dict dictobject.c:1895 (python.exe:arm64+0x1000cc9a0)
    #1 insertdict dictobject.c:2005 (python.exe:arm64+0x1000bcb84)
    #2 setitem_take2_lock_held dictobject.c:2785 (python.exe:arm64+0x1000bbf38)
    #3 dict_merge dictobject.c:4280 (python.exe:arm64+0x1000bfd00)
    #4 frozendict_vectorcall dictobject.c:5331 (python.exe:arm64+0x1000cc248)
    #5 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100061c10)

  Thread T1 (tid=4632788, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x33834)
    #1 do_start_joinable_thread thread_pthread.h:281 (python.exe:arm64+0x100279b6c)
    #2 do_start_new_thread _threadmodule.c:1919 (python.exe:arm64+0x100332da4)
    #3 thread_PyThread_start_joinable_thread _threadmodule.c:2042 (python.exe:arm64+0x100331f0c)
    #4 cfunction_call methodobject.c:564 (python.exe:arm64+0x1000de0b8)

SUMMARY: ThreadSanitizer: data race dictobject.c:3177 in _PyDict_Next
==================
==================
WARNING: ThreadSanitizer: data race (pid=56255)
  Read of size 8 at 0x00012a81c018 by thread T1:
    #0 _PyDict_Next dictobject.c:3166 (python.exe:arm64+0x1000be214)
    #1 anydict_repr_impl dictobject.c:3621 (python.exe:arm64+0x1000ce668)
    #2 frozendict_repr dictobject.c:8271 (python.exe:arm64+0x1000cbdb0)
    #3 PyObject_Repr object.c:784 (python.exe:arm64+0x1000e46a0)
    #4 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10027a4ec)

  Previous atomic write of size 8 at 0x00012a81c018 by main thread:
    #0 insert_combined_dict dictobject.c:1906 (python.exe:arm64+0x1000cca38)
    #1 insertdict dictobject.c:2005 (python.exe:arm64+0x1000bcb84)
    #2 setitem_take2_lock_held dictobject.c:2785 (python.exe:arm64+0x1000bbf38)
    #3 dict_merge dictobject.c:4280 (python.exe:arm64+0x1000bfd00)
    #4 frozendict_vectorcall dictobject.c:5331 (python.exe:arm64+0x1000cc248)
    #5 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100061c10)

  Thread T1 (tid=4632788, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x33834)
    #1 do_start_joinable_thread thread_pthread.h:281 (python.exe:arm64+0x100279b6c)
    #2 do_start_new_thread _threadmodule.c:1919 (python.exe:arm64+0x100332da4)
    #3 thread_PyThread_start_joinable_thread _threadmodule.c:2042 (python.exe:arm64+0x100331f0c)
    #4 cfunction_call methodobject.c:564 (python.exe:arm64+0x1000de0b8)

SUMMARY: ThreadSanitizer: data race dictobject.c:3166 in _PyDict_Next
==================
==================
WARNING: ThreadSanitizer: data race (pid=56255)
  Atomic write of size 8 at 0x00012a0e6918 by main thread:
    #0 insert_combined_dict dictobject.c:1906 (python.exe:arm64+0x1000cca38)
    #1 insertdict dictobject.c:2005 (python.exe:arm64+0x1000bcb84)
    #2 setitem_take2_lock_held dictobject.c:2785 (python.exe:arm64+0x1000bbf38)
    #3 dict_merge dictobject.c:4280 (python.exe:arm64+0x1000bfd00)
    #4 frozendict_vectorcall dictobject.c:5331 (python.exe:arm64+0x1000cc248)
    #5 _PyEval_EvalFrameDefault generated_cases.c.h:2418 (python.exe:arm64+0x1001d0048)

  Previous read of size 8 at 0x00012a0e6918 by thread T1:
    #0 _PyDict_Next dictobject.c:3166 (python.exe:arm64+0x1000be214)
    #1 anydict_repr_impl dictobject.c:3621 (python.exe:arm64+0x1000ce668)
    #2 frozendict_repr dictobject.c:8271 (python.exe:arm64+0x1000cbdb0)
    #3 PyObject_Repr object.c:784 (python.exe:arm64+0x1000e46a0)
    #4 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10027a4ec)

  Thread T1 (tid=4632788, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x33834)
    #1 do_start_joinable_thread thread_pthread.h:281 (python.exe:arm64+0x100279b6c)
    #2 do_start_new_thread _threadmodule.c:1919 (python.exe:arm64+0x100332da4)
    #3 thread_PyThread_start_joinable_thread _threadmodule.c:2042 (python.exe:arm64+0x100331f0c)
    #4 cfunction_call methodobject.c:564 (python.exe:arm64+0x1000de0b8)

SUMMARY: ThreadSanitizer: data race dictobject.c:1906 in insert_combined_dict
==================
==================
WARNING: ThreadSanitizer: data race (pid=56255)
  Atomic write of size 8 at 0x00012a74ffc0 by main thread:
    #0 insertdict dictobject.c:2008 (python.exe:arm64+0x1000bcba4)
    #1 setitem_take2_lock_held dictobject.c:2785 (python.exe:arm64+0x1000bbf38)
    #2 dict_merge dictobject.c:4280 (python.exe:arm64+0x1000bfd00)
    #3 frozendict_vectorcall dictobject.c:5331 (python.exe:arm64+0x1000cc248)
    #4 _PyEval_EvalFrameDefault generated_cases.c.h:2418 (python.exe:arm64+0x1001d0048)

  Previous read of size 8 at 0x00012a74ffc0 by thread T1:
    #0 frozendict_length dictobject.c:3701 (python.exe:arm64+0x1000d243c)
    #1 _PyEval_EvalFrameDefault generated_cases.c.h:3880 (python.exe:arm64+0x1001d1d98)
    #2 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10027a4ec)

  Thread T1 (tid=4632788, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x33834)
    #1 do_start_joinable_thread thread_pthread.h:281 (python.exe:arm64+0x100279b6c)
    #2 do_start_new_thread _threadmodule.c:1919 (python.exe:arm64+0x100332da4)
    #3 thread_PyThread_start_joinable_thread _threadmodule.c:2042 (python.exe:arm64+0x100331f0c)
    #4 cfunction_call methodobject.c:564 (python.exe:arm64+0x1000de0b8)

SUMMARY: ThreadSanitizer: data race dictobject.c:2008 in insertdict
==================
==================
WARNING: ThreadSanitizer: data race (pid=56255)
  Read of size 8 at 0x00012a0e69c8 by thread T1:
    #0 _PyDict_Next dictobject.c:3171 (python.exe:arm64+0x1000be260)
    #1 anydict_repr_impl dictobject.c:3621 (python.exe:arm64+0x1000ce668)
    #2 frozendict_repr dictobject.c:8271 (python.exe:arm64+0x1000cbdb0)
    #3 PyObject_Repr object.c:784 (python.exe:arm64+0x1000e46a0)
    #4 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10027a4ec)

  Previous atomic write of size 8 at 0x00012a0e69c8 by main thread:
    #0 insert_combined_dict dictobject.c (python.exe:arm64+0x1000cc9e8)
    #1 insertdict dictobject.c:2005 (python.exe:arm64+0x1000bcb84)
    #2 setitem_take2_lock_held dictobject.c:2785 (python.exe:arm64+0x1000bbf38)
    #3 dict_merge dictobject.c:4280 (python.exe:arm64+0x1000bfd00)
    #4 frozendict_vectorcall dictobject.c:5331 (python.exe:arm64+0x1000cc248)
    #5 _PyEval_EvalFrameDefault generated_cases.c.h:2418 (python.exe:arm64+0x1001d0048)

  Thread T1 (tid=4632788, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x33834)
    #1 do_start_joinable_thread thread_pthread.h:281 (python.exe:arm64+0x100279b6c)
    #2 do_start_new_thread _threadmodule.c:1919 (python.exe:arm64+0x100332da4)
    #3 thread_PyThread_start_joinable_thread _threadmodule.c:2042 (python.exe:arm64+0x100331f0c)
    #4 cfunction_call methodobject.c:564 (python.exe:arm64+0x1000de0b8)

SUMMARY: ThreadSanitizer: data race dictobject.c:3171 in _PyDict_Next
==================
==================
WARNING: ThreadSanitizer: data race (pid=56255)
  Atomic write of size 8 at 0x00012a74ffd0 by main thread:
    #0 dictresize dictobject.c:2279 (python.exe:arm64+0x1000cd9b4)
    #1 insert_combined_dict dictobject.c:1881 (python.exe:arm64+0x1000cc84c)
    #2 insertdict dictobject.c:2005 (python.exe:arm64+0x1000bcb84)
    #3 setitem_take2_lock_held dictobject.c:2785 (python.exe:arm64+0x1000bbf38)
    #4 dict_merge dictobject.c:4280 (python.exe:arm64+0x1000bfd00)
    #5 frozendict_vectorcall dictobject.c:5331 (python.exe:arm64+0x1000cc248)
    #6 _PyEval_EvalFrameDefault generated_cases.c.h:2418 (python.exe:arm64+0x1001d0048)

  Previous read of size 8 at 0x00012a74ffd0 by thread T1:
    #0 _PyDict_Next dictobject.c:3166 (python.exe:arm64+0x1000be208)
    #1 anydict_repr_impl dictobject.c:3621 (python.exe:arm64+0x1000ce668)
    #2 frozendict_repr dictobject.c:8271 (python.exe:arm64+0x1000cbdb0)
    #3 PyObject_Repr object.c:784 (python.exe:arm64+0x1000e46a0)
    #4 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10027a4ec)

  Thread T1 (tid=4632788, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x33834)
    #1 do_start_joinable_thread thread_pthread.h:281 (python.exe:arm64+0x100279b6c)
    #2 do_start_new_thread _threadmodule.c:1919 (python.exe:arm64+0x100332da4)
    #3 thread_PyThread_start_joinable_thread _threadmodule.c:2042 (python.exe:arm64+0x100331f0c)
    #4 cfunction_call methodobject.c:564 (python.exe:arm64+0x1000de0b8)

SUMMARY: ThreadSanitizer: data race dictobject.c:2279 in dictresize
==================
==================
WARNING: ThreadSanitizer: data race (pid=56255)
  Read of size 8 at 0x00012a0e69c0 by thread T1:
    #0 _PyDict_Next dictobject.c:3177 (python.exe:arm64+0x1000be2b8)
    #1 anydict_repr_impl dictobject.c:3621 (python.exe:arm64+0x1000ce668)
    #2 frozendict_repr dictobject.c:8271 (python.exe:arm64+0x1000cbdb0)
    #3 PyObject_Repr object.c:784 (python.exe:arm64+0x1000e46a0)
    #4 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10027a4ec)

  Previous atomic write of size 8 at 0x00012a0e69c0 by main thread:
    #0 insert_combined_dict dictobject.c:1895 (python.exe:arm64+0x1000cc9a0)
    #1 insertdict dictobject.c:2005 (python.exe:arm64+0x1000bcb84)
    #2 setitem_take2_lock_held dictobject.c:2785 (python.exe:arm64+0x1000bbf38)
    #3 dict_merge dictobject.c:4280 (python.exe:arm64+0x1000bfd00)
    #4 frozendict_vectorcall dictobject.c:5331 (python.exe:arm64+0x1000cc248)
    #5 _PyEval_EvalFrameDefault generated_cases.c.h:2418 (python.exe:arm64+0x1001d0048)

  Thread T1 (tid=4632788, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x33834)
    #1 do_start_joinable_thread thread_pthread.h:281 (python.exe:arm64+0x100279b6c)
    #2 do_start_new_thread _threadmodule.c:1919 (python.exe:arm64+0x100332da4)
    #3 thread_PyThread_start_joinable_thread _threadmodule.c:2042 (python.exe:arm64+0x100331f0c)
    #4 cfunction_call methodobject.c:564 (python.exe:arm64+0x1000de0b8)

SUMMARY: ThreadSanitizer: data race dictobject.c:3177 in _PyDict_Next
==================
==================
WARNING: ThreadSanitizer: data race (pid=56255)
  Read of size 8 at 0x00012a0ff198 by thread T1:
    #0 _PyDict_Next dictobject.c:3166 (python.exe:arm64+0x1000be214)
    #1 anydict_repr_impl dictobject.c:3621 (python.exe:arm64+0x1000ce448)
    #2 frozendict_repr dictobject.c:8271 (python.exe:arm64+0x1000cbdb0)
    #3 PyObject_Repr object.c:784 (python.exe:arm64+0x1000e46a0)
    #4 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10027a4ec)

  Previous write of size 8 at 0x00012a0ff198 by main thread:
    #0 new_keys_object dictobject.c:853 (python.exe:arm64+0x1000cc6d4)
    #1 dictresize dictobject.c:2174 (python.exe:arm64+0x1000cccd0)
    #2 insert_combined_dict dictobject.c:1881 (python.exe:arm64+0x1000cc84c)
    #3 insertdict dictobject.c:2005 (python.exe:arm64+0x1000bcb84)
    #4 setitem_take2_lock_held dictobject.c:2785 (python.exe:arm64+0x1000bbf38)
    #5 dict_merge dictobject.c:4280 (python.exe:arm64+0x1000bfd00)
    #6 frozendict_vectorcall dictobject.c:5331 (python.exe:arm64+0x1000cc248)
    #7 _PyEval_EvalFrameDefault generated_cases.c.h:2418 (python.exe:arm64+0x1001d0048)

  Thread T1 (tid=4632788, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x33834)
    #1 do_start_joinable_thread thread_pthread.h:281 (python.exe:arm64+0x100279b6c)
    #2 do_start_new_thread _threadmodule.c:1919 (python.exe:arm64+0x100332da4)
    #3 thread_PyThread_start_joinable_thread _threadmodule.c:2042 (python.exe:arm64+0x100331f0c)
    #4 cfunction_call methodobject.c:564 (python.exe:arm64+0x1000de0b8)

SUMMARY: ThreadSanitizer: data race dictobject.c:3166 in _PyDict_Next
==================
==================
WARNING: ThreadSanitizer: data race (pid=56255)
  Read of size 8 at 0x00012a108118 by thread T1:
    #0 _PyDict_Next dictobject.c:3166 (python.exe:arm64+0x1000be214)
    #1 anydict_repr_impl dictobject.c:3621 (python.exe:arm64+0x1000ce668)
    #2 frozendict_repr dictobject.c:8271 (python.exe:arm64+0x1000cbdb0)
    #3 PyObject_Repr object.c:784 (python.exe:arm64+0x1000e46a0)
    #4 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10027a4ec)

  Previous atomic write of size 8 at 0x00012a108118 by main thread:
    #0 dictresize dictobject.c:2292 (python.exe:arm64+0x1000cdb28)
    #1 insert_combined_dict dictobject.c:1881 (python.exe:arm64+0x1000cc84c)
    #2 insertdict dictobject.c:2005 (python.exe:arm64+0x1000bcb84)
    #3 setitem_take2_lock_held dictobject.c:2785 (python.exe:arm64+0x1000bbf38)
    #4 dict_merge dictobject.c:4280 (python.exe:arm64+0x1000bfd00)
    #5 frozendict_vectorcall dictobject.c:5331 (python.exe:arm64+0x1000cc248)
    #6 _PyEval_EvalFrameDefault generated_cases.c.h:2418 (python.exe:arm64+0x1001d0048)

  Thread T1 (tid=4632788, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x33834)
    #1 do_start_joinable_thread thread_pthread.h:281 (python.exe:arm64+0x100279b6c)
    #2 do_start_new_thread _threadmodule.c:1919 (python.exe:arm64+0x100332da4)
    #3 thread_PyThread_start_joinable_thread _threadmodule.c:2042 (python.exe:arm64+0x100331f0c)
    #4 cfunction_call methodobject.c:564 (python.exe:arm64+0x1000de0b8)

SUMMARY: ThreadSanitizer: data race dictobject.c:3166 in _PyDict_Next
==================
ThreadSanitizer:DEADLYSIGNAL
==56255==ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000029 (pc 0x00010106faec bp 0x000170b8a540 sp 0x000170b8a500 T4632788)
==56255==The signal is caused by a READ memory access.
==56255==Hint: address points to the zero page.
    #0 __tsan_atomic64_load <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x5faec)
    #1 _PyDict_Next dictobject.c (python.exe:arm64+0x1000be2c8)
    #2 anydict_repr_impl dictobject.c:3621 (python.exe:arm64+0x1000ce668)
    #3 frozendict_repr dictobject.c:8271 (python.exe:arm64+0x1000cbdb0)
    #4 PyObject_Repr object.c:784 (python.exe:arm64+0x1000e46a0)
    #5 _PyEval_EvalFrameDefault generated_cases.c.h:2712 (python.exe:arm64+0x1001d07c4)
    #6 _PyEval_Vector ceval.c:2141 (python.exe:arm64+0x1001cc028)
    #7 _PyObject_VectorcallPrepend call.c:855 (python.exe:arm64+0x100063290)
    #8 context_run context.c:728 (python.exe:arm64+0x1001fff94)
    #9 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100061c10)
    #10 _Py_VectorCallInstrumentation_StackRefSteal ceval.c:768 (python.exe:arm64+0x1001cc64c)
    #11 _PyEval_EvalFrameDefault generated_cases.c.h:1906 (python.exe:arm64+0x1001cf74c)
    #12 _PyEval_Vector ceval.c:2141 (python.exe:arm64+0x1001cc028)
    #13 _PyObject_VectorcallPrepend call.c:855 (python.exe:arm64+0x100063290)
    #14 thread_run _threadmodule.c:388 (python.exe:arm64+0x100333580)
    #15 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10027a4ec)
    #16 __tsan_thread_start_func <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x337a0)
    #17 _pthread_start <null> (libsystem_pthread.dylib:arm64e+0x6c54)
    #18 thread_start <null> (libsystem_pthread.dylib:arm64e+0x1c18)

==56255==Register values:
 x[0] = 0x0000000101000000   x[1] = 0x0000100000000050   x[2] = 0x00000000cf7d1ffe   x[3] = 0x0000000000000008
 x[4] = 0x0000000000000003   x[5] = 0x0000000000000000   x[6] = 0x0000000000000000   x[7] = 0x0000000000000000
 x[8] = 0x0000000000000000   x[9] = 0x000000000000001f  x[10] = 0x00000000c0000000  x[11] = 0x0000000000000000
x[12] = 0x000000010b485628  x[13] = 0x0000100000000050  x[14] = 0x0000000000000003  x[15] = 0x0000000000004010
x[16] = 0x0000000000000000  x[17] = 0x00000001010c4a40  x[18] = 0x0000000000000000  x[19] = 0x0000000000000029
x[20] = 0x0000000101000000  x[21] = 0x00000001003c62cc  x[22] = 0x0000000000000000  x[23] = 0x0000000000000001
x[24] = 0x0000000000000000  x[25] = 0x00000000000007ee  x[26] = 0x0000000000010707  x[27] = 0x000000012ace6240
x[28] = 0x000000012c0e0190     fp = 0x0000000170b8a540     lr = 0x000000010106fadc     sp = 0x0000000170b8a500
ThreadSanitizer can not provide additional info.
SUMMARY: ThreadSanitizer: SEGV dictobject.c in _PyDict_Next
==56255==ABORTING
[1]    56255 abort      TSAN_OPTIONS="halt_on_error=0" ./python.exe repro_151722.py

TO-BE

➜  cpython git:(gh-151722) ✗ TSAN_OPTIONS="halt_on_error=0" ./python.exe repro_1
51722.py
observed half-built: False

cc @tonghuaroot

Thanks for the cc. Built the PR (--disable-gil --tsan) — confirms it closes the frozendict(mapping) path (TSan-clean where stock trips + crashes).

Two sibling creation paths the defer-track doesn't reach yet, both pre-existing + TSan-confirmed on this branch:

frozendict.fromkeys(non_dict_iterable, v) — _PyDict_FromKeys creates the result tracked, then fills it in the PyIter_Next loop, so it's observable half-built (clean TSan race: frozendict_length vs _PyDict_FromKeys/setitem_take2_lock_held). Small fix in your frozendict_vectorcall idiom (untrack the fill loop, re-track after) — I've built + verified it (race → 0, fast paths untouched, test_dict/test_free_threading green) plus a regression test. Happy to send as a PR against your branch or a standalone follow-up, whichever you prefer.
frozendict subclass — a subclass gets a tracking tp_alloc, so the trailing _PyObject_GC_TRACK is a no-op; FD(mapping) is observable half-built. Flagging — that's the alloc path, your call.

Also no committed test yet — the fromkeys one above doubles as a test_racing_* for your path if useful.

Is this if statement needed?

I think we don't need it

@tonghuaroot

For frozendict.fromkey, you can submit the separate PR after this PR is merged.
For frozendict subclas, I will take a look at this separately

This function is strange. Why does it ignore its parameters?
Creating a dict with arguments should create a dict from those arguments.

cpython/Objects/dictobject.c

Lines 5245 to 5269 in aa5b164

    
           dict_new(PyTypeObject *type, PyObject *args, PyObject *kwds) 
        
           { 
        
               assert(type != NULL); 
        
               assert(type->tp_alloc != NULL); 
        
               // dict subclasses must implement the GC protocol 
        
               assert(_PyType_IS_GC(type)); 
        
               PyObject *self = type->tp_alloc(type, 0); 
        
               if (self == NULL) { 
        
                   return NULL; 
        
               } 
        
               PyDictObject *d = (PyDictObject *)self; 
        
               d->ma_used = 0; 
        
               d->_ma_watcher_tag = 0; 
        
               // We don't inc ref empty keys because they're immortal 
        
               assert((Py_EMPTY_KEYS)->dk_refcnt == _Py_DICT_IMMORTAL_INITIAL_REFCNT); 
        
               d->ma_keys = Py_EMPTY_KEYS; 
        
               d->ma_values = NULL; 
        
               ASSERT_CONSISTENT(d); 
        
               if (!_PyObject_GC_IS_TRACKED(d)) { 
        
                   _PyObject_GC_TRACK(d); 
        
               } 
        
               return self; 
        
           }

The function already ignore those params, and it was done by dict_init

Indeed:

>>> dict.__new__(dict, (), a=1,b=2)
{}

but frozendict is inconsistent:

>>> frozendict.__new__(frozendict, (), a=1,b=2)
frozendict({'a': 1, 'b': 2})

I think that than we need to modify frozenset also

>>> set.__new__(set, [1,2,3])
set()
>>> frozenset.__new__(frozenset, [1,2,3])
frozenset({1, 2, 3})

Let's handle this at separate issue.

I don't see what's wrong here. For me, it works as expected:

>>> dict((), a=1,b=2)
{'a': 1, 'b': 2}
>>> frozendict((), a=1,b=2)
frozendict({'a': 1, 'b': 2})

The fact that arguments are used by __new__() or __init__() is just an implementation detail. I don't think that it matters for the "constructor" API.

See #151722 (comment) for the proper fix for the issue.

Having said that, deferring tracking of any object until the object is complete is a good idea.
No object can be part of an unreachable cycle during creation.

@markshannon @methane I've updated the PR. PTAL :)

#127027

No more lazy track from here, we can remove this logic safely.

This removal makes me a little bit nervous. You may add the following assertion at the top of dict_dict_merge() (after other assertions).

    if (PyDict_Check(mp)) {
        assert(_PyObject_GC_IS_TRACKED(mp));
    }

LGTM

Suggested change

      
                   object is never reachable from another thread. */
          
                   object is never reachable from another thread (using the gc module). */

You may add a reference to the issue gh-151722.

Suggested change

      
                /* tp_new only allocates; args/kwds are consumed by dict_init (tp_init). */
          
                /* tp_new ignores args/kwds; args/kwds are consumed by dict_init (tp_init). */

This removal makes me a little bit nervous. You may add the following assertion at the top of dict_dict_merge() (after other assertions).

    if (PyDict_Check(mp)) {
        assert(_PyObject_GC_IS_TRACKED(mp));
    }

Thanks @corona10 for the PR 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

GH-151954 is a backport of this pull request to the 3.15 branch.

…ion (gh-151740) (gh-151954) gh-151722: Defer GC tracking of frozendict to end of construction (gh-151740) (cherry picked from commit 6920036) Co-authored-by: Donghee Na <donghee.na@python.org>

…on (pythongh-151740)

corona10 requested review from markshannon and methane as code owners June 19, 2026 15:36

bedevere-app Bot mentioned this pull request Jun 19, 2026

Data race on frozendict reads during construction in the free-threading build #151722

Closed

bedevere-app Bot added the awaiting core review label Jun 19, 2026

corona10 requested review from colesbury, kumaraditya303 and vstinner June 19, 2026 15:36

corona10 mentioned this pull request Jun 19, 2026

gh-151722: Fix data race on frozendict reads during construction in free-threading build #151724

Closed

corona10 added needs backport to 3.15 pre-release feature fixes, bugs and security fixes topic-free-threading labels Jun 19, 2026

corona10 force-pushed the gh-151722 branch from a621be3 to c42a3ec Compare June 20, 2026 03:04

pythongh-151722 : Defer GC tracking of frozendict to end of construction

cd1b1a4

corona10 force-pushed the gh-151722 branch from c42a3ec to cd1b1a4 Compare June 20, 2026 03:05

methane reviewed Jun 20, 2026

View reviewed changes

markshannon reviewed Jun 20, 2026

View reviewed changes

corona10 added 3 commits June 20, 2026 20:06

Address code review

399d810

fix

286eafd

fix

d74be81

corona10 changed the title gh-151722: Defer GC tracking of frozendict to end of construction [WIP] gh-151722: Defer GC tracking of frozendict to end of construction Jun 20, 2026

Update

897bdec

corona10 changed the title [WIP] gh-151722: Defer GC tracking of frozendict to end of construction gh-151722: Defer GC tracking of frozendict to end of construction Jun 20, 2026

corona10 requested a review from markshannon June 20, 2026 12:08

corona10 commented Jun 20, 2026

View reviewed changes

vstinner approved these changes Jun 22, 2026

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels Jun 22, 2026

Address code review

69a6316

corona10 enabled auto-merge (squash) June 22, 2026 23:21

corona10 merged commit 6920036 into python:main Jun 22, 2026
54 checks passed

bedevere-app Bot removed the awaiting merge label Jun 22, 2026

bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Jun 22, 2026

This was referenced Jun 23, 2026

gh-151722: Defer GC tracking of frozendict.fromkeys() result until fully built #151967

Closed

[WIP] gh-151722: Defer GC tracking of frozendict.fromkeys() as possible #152021

Closed

pyctrl pushed a commit to pyctrl/cpython that referenced this pull request Jul 5, 2026

pythongh-151722 : Defer GC tracking of frozendict to end of constructi…

fad3af2

…on (pythongh-151740)

	dict_new(PyTypeObject type, PyObject args, PyObject *kwds)
	{
	assert(type != NULL);
	assert(type->tp_alloc != NULL);
	// dict subclasses must implement the GC protocol
	assert(_PyType_IS_GC(type));

	PyObject *self = type->tp_alloc(type, 0);
	if (self == NULL) {
	return NULL;
	}
	PyDictObject d = (PyDictObject )self;

	d->ma_used = 0;
	d->_ma_watcher_tag = 0;
	// We don't inc ref empty keys because they're immortal
	assert((Py_EMPTY_KEYS)->dk_refcnt == _Py_DICT_IMMORTAL_INITIAL_REFCNT);
	d->ma_keys = Py_EMPTY_KEYS;
	d->ma_values = NULL;
	ASSERT_CONSISTENT(d);
	if (!_PyObject_GC_IS_TRACKED(d)) {
	_PyObject_GC_TRACK(d);
	}
	return self;
	}

                           STORE_USED(mp, other->ma_used);
                           ASSERT_CONSISTENT(mp);
-                          if (_PyObject_GC_IS_TRACKED(other) && !_PyObject_GC_IS_TRACKED(mp)) {

-                  PyObject *self = frozendict_new(_PyType_CAST(type), NULL, NULL);
+                  /* Keep the frozendict untracked until it is fully built, so a half-built
+                     object is never reachable from another thread. */

	object is never reachable from another thread. */
	object is never reachable from another thread (using the gc module). */

+              static PyObject *
+              dict_new(PyTypeObject *type, PyObject *Py_UNUSED(args), PyObject *Py_UNUSED(kwds))
+              {
+                  /* tp_new only allocates; args/kwds are consumed by dict_init (tp_init). */

	/* tp_new only allocates; args/kwds are consumed by dict_init (tp_init). */
	/* tp_new ignores args/kwds; args/kwds are consumed by dict_init (tp_init). */

gh-151722: Defer GC tracking of frozendict to end of construction by corona10 · Pull Request #151740 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-151722: Defer GC tracking of frozendict to end of construction - #151740

gh-151722: Defer GC tracking of frozendict to end of construction#151740corona10 merged 6 commits intopython:mainpython/cpython:mainfrom corona10:gh-151722corona10/cpython:gh-151722Copy head branch name to clipboard

Conversation

corona10 commented Jun 19, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

corona10 commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TSAN script

AS-IS

TO-BE

Uh oh!

corona10 commented Jun 19, 2026

Uh oh!

tonghuaroot commented Jun 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

corona10 commented Jun 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markshannon commented Jun 20, 2026

Uh oh!

corona10 commented Jun 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-app Bot commented Jun 22, 2026

Uh oh!

bedevere-app Bot commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gh-151722: Defer GC tracking of frozendict to end of construction#151740
corona10 merged 6 commits into
python:mainfrom
corona10:gh-151722

corona10 commented Jun 19, 2026 •
edited by bedevere-app Bot
Loading

corona10 commented Jun 19, 2026 •
edited
Loading