SEGV in `faulthandler.dump_traceback_later` · Issue #140815 · python/cpython · GitHub

Crash report

What happened?

import faulthandler
import sys
import unittest

class Test(unittest.TestCase):
    def setUp(self):
        faulthandler.dump_traceback_later(10 * 1e-308, exit=True, file=sys.__stderr__)
    def test_sendall(self):
        os.mkfifo(filename)

if __name__ == '__main__':
    unittest.main()

FAILED (errors=1)
    #0 0x5646ef225cff in dump_traceback ../Python/traceback.c:1111
    #1 0x5646ef229ae2 in _Py_DumpTracebackThreads ../Python/traceback.c:1302
    #2 0x5646ef2709d4 in faulthandler_thread ../Modules/faulthandler.c:706
    #3 0x5646ef2220af in pythread_wrapper ../Python/thread_pthread.h:234
    #4 0x76c294e77a41 in asan_thread_start ../../../../src/libsanitizer/asan/asan_interceptors.cpp:234
    #5 0x76c294bbaaa3  (/lib/x86_64-linux-gnu/libc.so.6+0x9caa3) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)
    #6 0x76c294c47a33 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x129a33) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)

Address 0x76c292fb39da is located in stack of thread T0 at offset 2522 in frame
    #0 0x5646eeb38bcf in _PyEval_EvalFrameDefault ../Python/ceval.c:1032

  This frame has 37 object(s):
    [32, 36) 'method_found'
    [48, 52) 'level'
    [64, 68) 'handler'
    [80, 84) 'lasti'
    [96, 104) 'temp'
    [128, 136) 'res_o'
    [160, 168) 'match_o'
    [192, 200) 'rest_o'
    [224, 232) 'null_or_index'
    [256, 264) 'null_or_index'
    [288, 296) 'executor'
    [320, 328) 'bc_o'
    [352, 360) 'value_o'
    [384, 392) 'v_o'
    [416, 424) 'retval_o'
    [448, 456) 'ann_dict'
    [480, 488) 'kwnames' (line 1906)
    [512, 528) 'stack'
    [544, 560) 'stack'
    [576, 592) 'args'
    [608, 648) 'stack'
    [688, 752) 'stack_array' (line 1909)
    [784, 872) 'values_o_temp'
    [912, 1000) 'pieces_o_temp'
    [1040, 1128) 'args_o_temp'
    [1168, 1256) 'args_o_temp'
    [1296, 1384) 'args_o_temp'
    [1424, 1512) 'args_o_temp'
    [1552, 1640) 'args_o_temp'
    [1680, 1768) 'args_o_temp'
    [1808, 1896) 'args_o_temp'
    [1936, 2024) 'args_o_temp'
    [2064, 2152) 'args_o_temp'
    [2192, 2280) 'args_o_temp'
    [2320, 2408) 'args_o_temp'
    [2448, 2544) 'entry' (line 1051) <== Memory access at offset 2522 is inside this variable
    [2576, 2656) 'buffer' (line 518)
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-use-after-return ../Python/traceback.c:1111 in dump_traceback
Shadow bytes around the buggy address:
  0x76c292fb3700: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x76c292fb3780: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x76c292fb3800: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x76c292fb3880: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x76c292fb3900: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
=>0x76c292fb3980: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5[f5]f5 f5 f5 f5
  0x76c292fb3a00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x76c292fb3a80: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x76c292fb3b00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x76c292fb3b80: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x76c292fb3c00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
Thread T1 created by T0 here:
    #0 0x76c294f0e1f9 in pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:245
    #1 0x5646ef22228b in do_start_joinable_thread ../Python/thread_pthread.h:281
    #2 0x5646ef2229a1 in PyThread_start_new_thread ../Python/thread_pthread.h:336
    #3 0x5646ef274706 in faulthandler_dump_traceback_later_impl ../Modules/faulthandler.c:868
    #4 0x5646ef274706 in faulthandler_dump_traceback_later ../Modules/clinic/faulthandler.c.h:366
    #5 0x5646eec9f677 in _PyObject_VectorcallTstate ../Include/internal/pycore_call.h:169
    #6 0x5646eec9f677 in PyObject_Vectorcall ../Objects/call.c:327
    #7 0x5646eeb42bba in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:2920
    #8 0x5646ef020785 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #9 0x5646ef020785 in _PyEval_Vector ../Python/ceval.c:2005
    #10 0x5646eeca90f0 in _PyObject_VectorcallTstate ../Include/internal/pycore_call.h:169
    #11 0x5646eeca90f0 in method_vectorcall ../Objects/classobject.c:95
    #12 0x5646eeca478e in _PyVectorcall_Call ../Objects/call.c:273
    #13 0x5646eeca478e in _PyObject_Call ../Objects/call.c:348
    #14 0x5646eeca478e in PyObject_Call ../Objects/call.c:373
    #15 0x5646eeb43e9c in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:2616
    #16 0x5646ef020785 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #17 0x5646ef020785 in _PyEval_Vector ../Python/ceval.c:2005
    #18 0x5646eeca2db3 in _PyObject_VectorcallDictTstate ../Objects/call.c:135
    #19 0x5646eeca346c in _PyObject_Call_Prepend ../Objects/call.c:504
    #20 0x5646eee5ff64 in call_method ../Objects/typeobject.c:3077
    #21 0x5646eee5ff64 in slot_tp_call ../Objects/typeobject.c:10606
    #22 0x5646eec9dc5d in _PyObject_MakeTpCall ../Objects/call.c:242
    #23 0x5646eeb417ac in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:4021
    #24 0x5646ef020785 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #25 0x5646ef020785 in _PyEval_Vector ../Python/ceval.c:2005
    #26 0x5646eeca90f0 in _PyObject_VectorcallTstate ../Include/internal/pycore_call.h:169
    #27 0x5646eeca90f0 in method_vectorcall ../Objects/classobject.c:95
    #28 0x5646eeca478e in _PyVectorcall_Call ../Objects/call.c:273
    #29 0x5646eeca478e in _PyObject_Call ../Objects/call.c:348
    #30 0x5646eeca478e in PyObject_Call ../Objects/call.c:373
    #31 0x5646eeb43e9c in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:2616
    #32 0x5646ef020785 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #33 0x5646ef020785 in _PyEval_Vector ../Python/ceval.c:2005
    #34 0x5646eeca2db3 in _PyObject_VectorcallDictTstate ../Objects/call.c:135
    #35 0x5646eeca346c in _PyObject_Call_Prepend ../Objects/call.c:504
    #36 0x5646eee5ff64 in call_method ../Objects/typeobject.c:3077
    #37 0x5646eee5ff64 in slot_tp_call ../Objects/typeobject.c:10606
    #38 0x5646eec9dc5d in _PyObject_MakeTpCall ../Objects/call.c:242
    #39 0x5646eeb40ad2 in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:1620
    #40 0x5646ef020785 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #41 0x5646ef020785 in _PyEval_Vector ../Python/ceval.c:2005
    #42 0x5646eeca90f0 in _PyObject_VectorcallTstate ../Include/internal/pycore_call.h:169
    #43 0x5646eeca90f0 in method_vectorcall ../Objects/classobject.c:95
    #44 0x5646eeca478e in _PyVectorcall_Call ../Objects/call.c:273
    #45 0x5646eeca478e in _PyObject_Call ../Objects/call.c:348
    #46 0x5646eeca478e in PyObject_Call ../Objects/call.c:373
    #47 0x5646eeb43e9c in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:2616
    #48 0x5646ef020785 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #49 0x5646ef020785 in _PyEval_Vector ../Python/ceval.c:2005
    #50 0x5646eeca2db3 in _PyObject_VectorcallDictTstate ../Objects/call.c:135
    #51 0x5646eeca346c in _PyObject_Call_Prepend ../Objects/call.c:504
    #52 0x5646eee5ff64 in call_method ../Objects/typeobject.c:3077
    #53 0x5646eee5ff64 in slot_tp_call ../Objects/typeobject.c:10606
    #54 0x5646eec9dc5d in _PyObject_MakeTpCall ../Objects/call.c:242
    #55 0x5646eeb40ad2 in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:1620
    #56 0x5646ef020785 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #57 0x5646ef020785 in _PyEval_Vector ../Python/ceval.c:2005
    #58 0x5646eeca2db3 in _PyObject_VectorcallDictTstate ../Objects/call.c:135
    #59 0x5646eeca346c in _PyObject_Call_Prepend ../Objects/call.c:504
    #60 0x5646eee4d2e0 in call_method ../Objects/typeobject.c:3077
    #61 0x5646eee4d2e0 in slot_tp_init ../Objects/typeobject.c:10835
    #62 0x5646eee3f457 in type_call ../Objects/typeobject.c:2461
    #63 0x5646eec9dc5d in _PyObject_MakeTpCall ../Objects/call.c:242
    #64 0x5646eeb5c9f8 in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:1620
    #65 0x5646ef01ffb6 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #66 0x5646ef01ffb6 in _PyEval_Vector ../Python/ceval.c:2005
    #67 0x5646ef01ffb6 in PyEval_EvalCode ../Python/ceval.c:888
    #68 0x5646ef1de3fe in run_eval_code_obj ../Python/pythonrun.c:1365
    #69 0x5646ef1de3fe in run_mod ../Python/pythonrun.c:1459
    #70 0x5646ef1e30a7 in pyrun_file ../Python/pythonrun.c:1293
    #71 0x5646ef1e30a7 in _PyRun_SimpleFileObject ../Python/pythonrun.c:521
    #72 0x5646ef1e3bcc in _PyRun_AnyFileObject ../Python/pythonrun.c:81
    #73 0x5646ef2563cc in pymain_run_file_obj ../Modules/main.c:410
    #74 0x5646ef2563cc in pymain_run_file ../Modules/main.c:429
    #75 0x5646ef2563cc in pymain_run_python ../Modules/main.c:691
    #76 0x5646ef257cae in Py_RunMain ../Modules/main.c:772
    #77 0x5646ef257cae in pymain_main ../Modules/main.c:802
    #78 0x5646ef257cae in Py_BytesMain ../Modules/main.c:826
    #79 0x76c294b481c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)
    #80 0x76c294b4828a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)

==2938816==ABORTING

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

Linked PRs

Is

this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork

relevant? cc @vstinner

I have following repro:

import faulthandler
import sys


faulthandler.dump_traceback_later(10 * 1e-308, exit=True, file=sys.__stderr__)
assert False

It fails with assert in the C code:

Running Debug|x64 interpreter...
Traceback (most recent call last):
Timeout (0:00:00.000001)!
Thread 0x00006e38 (most recent call first):
  File "X:\Work\cpython\main\Lib\traceback.py"  File "X:\Work\cpython\issues\b140815.py", line 8, in <module>
    assert False
           ^^^^^
Assertion failed: !PyStackRef_IsNull(f->f_executable), file X:\Work\cpython\main\Include\internal\pycore_interpframe.h, line 21
AssertionError

If no one minds, I'd like to take a look.

This segfault occurs because the faulthandler_thread runs concurrently with interpreter finalization.

This happens because the timeout value of the dump_traceback_later is 10 * 1e-308 which is cropped to 1us. Therefore, the faulthandler_thread runs as soon as possible.

faulthandler_thread doesn't use any synchronization primitives, so it is not a thread-safe, even in the GIL-enabled build.

I found an issue #128400 and a PR #128422 where the call to the StopTheWorld function was added to the faulthandler_dump_traceback_py_impl to prevent multithreading issues in the no-GIL build. Therefore, I believe this approach is valid for this issue as well. For the GIL-enabled build, I added the GIL acquisition.

pythongh-140815: Fix faulthandler for invalid/freed frame

@sergey-miryanov:

This segfault occurs because the faulthandler_thread runs concurrently with interpreter finalization.

_Py_DumpTracebackThreads() is called by the faulthandler.dump_traceback_later() thread while the main thread is freeing its frame.

_Py_DumpTracebackThreads() reads freed memory and does crash randomly. It already calls _PyMem_IsPtrFreed() in a few places to detect freed memory, but it's not enough.

faulthandler_thread doesn't use any synchronization primitives, so it is not a thread-safe, even in the GIL-enabled build.

This is by design. faulthandler.dump_traceback_later(exit=True) dumps the traceback and exits the process to implement a software watchdog. It is used by buildbots and other CIs to exit after a timeout.

faulthandler.dump_traceback_later() should not attempt to acquire the GIL or any other lock since it's used to dump the traceback in case in other deadlocks.

I wrote #140921 to fix the issue: detect if a frame object or a code object is invalid or freed.

With my fix, I can no longer reproduce the crash: I ran the script in a loop for 1 minute.

faulthandler.dump_traceback_later() should not attempt to acquire the GIL or any other lock since it's used to dump the traceback in case in other deadlocks.

Thanks to point me this! It was not clear to me after reading the documentation (it is only one reference to the deadlock on the page and I just missed it).

Dummy example of deadlock:

import faulthandler
import threading

def work():
    lock = threading.Lock()
    lock.acquire()
    lock.acquire()
    lock.release()
    lock.release()

faulthandler.dump_traceback_later(5, exit=True)
work()
faulthandler.cancel_dump_traceback_later()

Output:

Timeout (0:00:05)!
Thread 0x00007fc30261ebc0 [python] (most recent call first):
  File "/home/vstinner/python/main/x.py", line 7 in work
  File "/home/vstinner/python/main/x.py", line 12 in <module>

faulthandler shows where the deadlock occurs.

faulthandler shows where the deadlock occurs.

I can use this at work somehow 🤔 ... Thanks!

gh-140815: Fix faulthandler for invalid/freed frame (#140921)

pythongh-140815: Fix faulthandler for invalid/freed frame (pythonGH-1…

pythongh-140815: Fix faulthandler for invalid/freed frame (python#140921

[3.14] gh-140815: Fix faulthandler for invalid/freed frame (GH-140921) (

[3.13] gh-140815: Fix faulthandler for invalid/freed frame (#140921) (#…

Instead of crashing, the initial reproducer now completes with incomplete traceback like:

$ ./python x.py
Timeout (0:00:00.000001)!
Thread 0x00007f107d86bbc0 [python] (most recent call first):
  File "/home/vstinner/python/main/Lib/tokenize.py", line ??? in find_cookie
  <invalid frame>

Even if, as I explained in other comments, it's not 100% reliable. I consider that the initial issue has been fixed and I close the issue. Thanks for your bug report.

pythongh-140815: Fix faulthandler for invalid/freed frame (python#140921

fix(idlewatch): sentinela em thread Python, sem o watchdog em C do fa…

Backport gh-140815 faulthandler trampoline-frame fix

SEGV in `faulthandler.dump_traceback_later` · Issue #140815 · python/cpython · GitHub

Repository navigation

SEGV in faulthandler.dump_traceback_later #140815

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

SEGV in `faulthandler.dump_traceback_later` · Issue #140815 · python/cpython · GitHub

Repository navigation

SEGV in faulthandler.dump_traceback_later #140815

Description

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Activity

picnixz commented on Oct 31, 2025

sergey-miryanov commented on Oct 31, 2025

sergey-miryanov commented on Nov 1, 2025

vstinner commented on Nov 2, 2025

vstinner commented on Nov 2, 2025

sergey-miryanov commented on Nov 2, 2025

vstinner commented on Nov 2, 2025

sergey-miryanov commented on Nov 2, 2025

vstinner commented on Nov 5, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions