gh-158880: Fix asyncio ps showing tasks from only one interpreter by deadlovelll · Pull Request #158900 · python/cpython · GitHub

Fix asyncio ps showing tasks from only one interpreter

For details see gh-158880

asyncio.sleep feels like a flaky risk. What do you think about using busy_retry instead?

Thanks! I fixed it

Unless I'm missing something, get_async_stack_trace() still has the same problem. Maybe it should be a follow-up, though.

Yeah, it has the problem, i planned to open a separate issue for this to avoid "big-bang" pr

This is just a list:

cpython/Modules/_remote_debugging/module.c

Lines 719 to 720 in 9028df3

    
           uintptr_t current_interpreter = self->interpreter_addr; 
        
           while (current_interpreter != 0) {

cpython/Modules/_remote_debugging/module.c

Lines 884 to 885 in 9028df3

    
           current_interpreter = GET_MEMBER(uintptr_t, interp_state_buffer, 
        
                   self->debug_offsets.interpreter_state.next);

cpython/Modules/_remote_debugging/frames.c

Lines 459 to 462 in 182f323

    
           const size_t MAX_INTERPRETERS = 256; 
        
           size_t interp_count = 0; 
        
           while (current_interp != 0 && interp_count < MAX_INTERPRETERS) {

Is there a reproducible race scenario where we'd see a cycle? (It was quiet easy to reproduce issues like ABA.)

No, I couldn't reproduce a cycle. I thought the interpreter walk could loop
because of the previous PR, that's why I added the set. I agree with your suggestion and will rework it soon

I rewrote the traverse

One way to silence https://github.com/python/cpython/actions/runs/37499870518/job/112393935495#step:6:1023 is by wrapping this in except TRANSIENT_ERRORS:

Perhaps a better approach, similar to #158801

cpython/Lib/test/test_external_inspection.py

Lines 2440 to 2460 in 0e75a05

    
                       from _queue import SimpleQueue 
        
                       from test import support 
        
                       go = threading.Lock() 
        
                       stop = threading.Lock() 
        
                       go.acquire() 
        
                       stop.acquire() 
        
                       ready = SimpleQueue() 
        
                       def leaf(): 
        
                           ready.put(None) 
        
                           stop.acquire() 
        
                       def start_leaf(): 
        
                           ready.put(None) 
        
                           go.acquire() 
        
                           leaf() 
        
                       def park(): 
        
                           ready.put(None) 
        
                           stop.acquire()

cpython/Lib/test/test_external_inspection.py

Lines 2469 to 2482 in 0e75a05

    
                       # SimpleQueue.put() and Lock.acquire() do not push Python frames. 
        
                       # Once notified, the worker's stack stays stable until go is released. 
        
                       threading.Thread(target=leaf, daemon=True).start() 
        
                       ready.get(timeout=support.SHORT_TIMEOUT) 
        
                       for _ in range(16): 
        
                           threading.Thread(target=park, daemon=True).start() 
        
                           ready.get(timeout=support.SHORT_TIMEOUT) 
        
                       threading.Thread(target=start_leaf, daemon=True).start() 
        
                       ready.get(timeout=support.SHORT_TIMEOUT) 
        
                       u = RemoteUnwinder(os.getpid(), all_threads=True, cache_frames=False) 
        
                       assert leaf_count(u) == 1 
        
                       go.release() 
        
                       ready.get(timeout=support.SHORT_TIMEOUT)

I thought about how to stop the test flaking, so I decided to use queues in the test rather than silence exceptions. I ran the tests overnight many times, and there were no failures, ci here passed too

pythongh-158880 : Fix asyncio ps showing tasks from only one interpreter

03b736b

deadlovelll requested a review from pablogsal as a code owner October 6, 2026 09:06

bedevere-app Bot mentioned this pull request Oct 6, 2026

python -m asyncio ps shows tasks from only one interpreter #158880

Open

bedevere-app Bot added the awaiting review label Oct 6, 2026

maurycy reviewed Oct 6, 2026

View reviewed changes

deadlovelll added 2 commits October 6, 2026 17:56

remove potential flake in test

51482c0

use short timeout in test

7c205da

maurycy reviewed Oct 6, 2026

View reviewed changes

deadlovelll added 2 commits October 7, 2026 11:34

rewrite subinterp traversal

2f82834

rewrite test with queues

6430f21

This was referenced Oct 7, 2026

profiling.sampling --async-aware misses main interpreter tasks when subinterpreters exist #158968

Open

gh-158968: Fix profiling.sampling --async-aware showing tasks from only one interpreter #158971

Open

+                              loop.run_in_executor(pool, _asyncio_in_subinterpreter)
+                              task = asyncio.create_task(main_worker(), name="main_worker")
+                              self.addCleanup(task.cancel)
+                              await asyncio.sleep(1)

gh-158880: Fix asyncio ps showing tasks from only one interpreter by deadlovelll · Pull Request #158900 · python/cpython · GitHub

Repository navigation

gh-158880: Fix asyncio ps showing tasks from only one interpreter - #158900

gh-158880: Fix asyncio ps showing tasks from only one interpreter#158900
deadlovelll wants to merge 5 commits into
python:mainfrom
deadlovelll:gh-158880-interp

deadlovelll commented Oct 6, 2026

Uh oh!

maurycy Oct 6, 2026 •
edited
Loading

Uh oh!

Uh oh!

deadlovelll Oct 6, 2026

Uh oh!

maurycy Oct 6, 2026 •
edited
Loading

Uh oh!

Uh oh!

deadlovelll Oct 6, 2026

Uh oh!

maurycy Oct 6, 2026 •
edited
Loading

Uh oh!

Uh oh!

deadlovelll Oct 6, 2026

Uh oh!

deadlovelll Oct 7, 2026

Uh oh!

maurycy Oct 6, 2026 •
edited
Loading

Uh oh!

Uh oh!

deadlovelll Oct 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

                   // Process all threads
-                  if (iterate_threads(self, process_thread_for_async_stack_trace, result) < 0) {
+                  if (iterate_threads(self, self->interpreter_addr,

                       return NULL;
                   }
-                  if (refresh_generation_caches_for_interpreter(self, self->interpreter_addr) < 0) {
+                  PyObject *seen = PySet_New(NULL);

	uintptr_t current_interpreter = self->interpreter_addr;
	while (current_interpreter != 0) {

	current_interpreter = GET_MEMBER(uintptr_t, interp_state_buffer,
	self->debug_offsets.interpreter_state.next);

	const size_t MAX_INTERPRETERS = 256;
	size_t interp_count = 0;

	while (current_interp != 0 && interp_count < MAX_INTERPRETERS) {

		for info in RemoteUnwinder(
		os.getpid()).get_all_awaited_by()

	from _queue import SimpleQueue
	from test import support

	go = threading.Lock()
	stop = threading.Lock()
	go.acquire()
	stop.acquire()
	ready = SimpleQueue()

	def leaf():
	ready.put(None)
	stop.acquire()

	def start_leaf():
	ready.put(None)
	go.acquire()
	leaf()

	def park():
	ready.put(None)
	stop.acquire()

	# SimpleQueue.put() and Lock.acquire() do not push Python frames.
	# Once notified, the worker's stack stays stable until go is released.
	threading.Thread(target=leaf, daemon=True).start()
	ready.get(timeout=support.SHORT_TIMEOUT)
	for _ in range(16):
	threading.Thread(target=park, daemon=True).start()
	ready.get(timeout=support.SHORT_TIMEOUT)
	threading.Thread(target=start_leaf, daemon=True).start()
	ready.get(timeout=support.SHORT_TIMEOUT)

	u = RemoteUnwinder(os.getpid(), all_threads=True, cache_frames=False)
	assert leaf_count(u) == 1
	go.release()
	ready.get(timeout=support.SHORT_TIMEOUT)

gh-158880: Fix asyncio ps showing tasks from only one interpreter by deadlovelll · Pull Request #158900 · python/cpython · GitHub

Repository navigation

gh-158880: Fix asyncio ps showing tasks from only one interpreter - #158900

gh-158880: Fix asyncio ps showing tasks from only one interpreter#158900deadlovelll wants to merge 5 commits intopython:mainpython/cpython:mainfrom deadlovelll:gh-158880-interpdeadlovelll/cpython:gh-158880-interpCopy head branch name to clipboard

Conversation

deadlovelll commented Oct 6, 2026

Uh oh!

maurycy Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deadlovelll Oct 6, 2026

Choose a reason for hiding this comment

Uh oh!

maurycy Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deadlovelll Oct 6, 2026

Choose a reason for hiding this comment

Uh oh!

maurycy Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deadlovelll Oct 6, 2026

Choose a reason for hiding this comment

Uh oh!

deadlovelll Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

maurycy Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deadlovelll Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gh-158880: Fix asyncio ps showing tasks from only one interpreter#158900
deadlovelll wants to merge 5 commits into
python:mainfrom
deadlovelll:gh-158880-interp

maurycy Oct 6, 2026 •
edited
Loading

maurycy Oct 6, 2026 •
edited
Loading

maurycy Oct 6, 2026 •
edited
Loading

maurycy Oct 6, 2026 •
edited
Loading