[3.10] gh-90155: Fix bug in asyncio.Semaphore and strengthen FIFO guarantee (GH-93222) - #97020
Merged
miss-islington merged 1 commit intoSep 22, 2022
Merged
miss-islington merged 1 commit into
miss-islington merged 1 commit into
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
…antee (pythonGH-93222) The main problem was that an unluckily timed task cancellation could cause the semaphore to be stuck. There were also doubts about strict FIFO ordering of tasks allowed to pass. The Semaphore implementation was rewritten to be more similar to Lock. Many tests for edge cases (including cancellation) were added. (cherry picked from commit 24e0379) Co-authored-by: Cyker Way <cykerway@gmail.com>
gvanrossum
approved these changes
Sep 22, 2022
Copy link
Copy Markdown
Contributor
Author
|
Status check is done, and it's a success ✅. |
Sorry, something went wrong.
altendky
added a commit
to Chia-Network/chia-blockchain
that referenced
this pull request
Oct 15, 2022
We access the `._waiters` private attribute of the Python asyncio `Semaphore` class. This was changed in Python 3.10.8 (and other versions) to be initialized to `None` instead of an empty deque. Our existing unconditional length checks failed on the new `None` default. This seems to block syncing. python/cpython#97020 https://github.com/python/cpython/compare/v3.10.7..v3.10.8#diff-0fee1befb15023abc0dad2623effa93a304946796929f6cb445d11a57821e737 Reported traceback: ```python-traceback 2022-10-12T20:03:59.367 full_node full_node_server : INFO Connected with full_node {'host': '65.34.144.6', 'port': 8444} 2022-10-12T20:03:59.370 full_node full_node_server : ERROR Exception: object of type 'NoneType' has no len(), {'host': '65.34.144.6', 'port': 8444}. Traceback (most recent call last): File "/home/summa/chia-blockchain/chia/server/server.py", line 598, in wrapped_coroutine result = await coroutine File "/home/summa/chia-blockchain/chia/full_node/full_node_api.py", line 114, in new_peak waiter_count = len(self.full_node.new_peak_sem._waiters) TypeError: object of type 'NoneType' has no len() 2022-10-12T20:03:59.371 full_node full_node_server : ERROR Exception: object of type 'NoneType' has no len() <class 'TypeError'>, closing connection {'host': '65.34.144.6', 'port': 8444}. Traceback (most recent call last): File "/home/summa/chia-blockchain/chia/server/server.py", line 608, in api_call response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout) File "/usr/lib/python3.10/asyncio/tasks.py", line 408, in wait_for return await fut File "/home/summa/chia-blockchain/chia/server/server.py", line 605, in wrapped_coroutine raise e File "/home/summa/chia-blockchain/chia/server/server.py", line 598, in wrapped_coroutine result = await coroutine File "/home/summa/chia-blockchain/chia/full_node/full_node_api.py", line 114, in new_peak waiter_count = len(self.full_node.new_peak_sem._waiters) TypeError: object of type 'NoneType' has no len() 2022-10-12T20:03:59.487 full_node full_node_server : INFO Connection closed: 65.34.144.6, node id: 506fe4c05ce6b72bb707471842e552307c7a547aa9ba981175db5c08fa3e47e6 ```
wallentx
pushed a commit
to Chia-Network/chia-blockchain
that referenced
this pull request
Oct 17, 2022
We access the `._waiters` private attribute of the Python asyncio `Semaphore` class. This was changed in Python 3.10.8 (and other versions) to be initialized to `None` instead of an empty deque. Our existing unconditional length checks failed on the new `None` default. This seems to block syncing. python/cpython#97020 https://github.com/python/cpython/compare/v3.10.7..v3.10.8#diff-0fee1befb15023abc0dad2623effa93a304946796929f6cb445d11a57821e737 Reported traceback: ```python-traceback 2022-10-12T20:03:59.367 full_node full_node_server : INFO Connected with full_node {'host': '65.34.144.6', 'port': 8444} 2022-10-12T20:03:59.370 full_node full_node_server : ERROR Exception: object of type 'NoneType' has no len(), {'host': '65.34.144.6', 'port': 8444}. Traceback (most recent call last): File "/home/summa/chia-blockchain/chia/server/server.py", line 598, in wrapped_coroutine result = await coroutine File "/home/summa/chia-blockchain/chia/full_node/full_node_api.py", line 114, in new_peak waiter_count = len(self.full_node.new_peak_sem._waiters) TypeError: object of type 'NoneType' has no len() 2022-10-12T20:03:59.371 full_node full_node_server : ERROR Exception: object of type 'NoneType' has no len() <class 'TypeError'>, closing connection {'host': '65.34.144.6', 'port': 8444}. Traceback (most recent call last): File "/home/summa/chia-blockchain/chia/server/server.py", line 608, in api_call response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout) File "/usr/lib/python3.10/asyncio/tasks.py", line 408, in wait_for return await fut File "/home/summa/chia-blockchain/chia/server/server.py", line 605, in wrapped_coroutine raise e File "/home/summa/chia-blockchain/chia/server/server.py", line 598, in wrapped_coroutine result = await coroutine File "/home/summa/chia-blockchain/chia/full_node/full_node_api.py", line 114, in new_peak waiter_count = len(self.full_node.new_peak_sem._waiters) TypeError: object of type 'NoneType' has no len() 2022-10-12T20:03:59.487 full_node full_node_server : INFO Connection closed: 65.34.144.6, node id: 506fe4c05ce6b72bb707471842e552307c7a547aa9ba981175db5c08fa3e47e6 ```
wallentx
pushed a commit
to Chia-Network/chia-blockchain
that referenced
this pull request
Nov 6, 2022
* treat Semaphore._waiters as length zero when it is None We access the `._waiters` private attribute of the Python asyncio `Semaphore` class. This was changed in Python 3.10.8 (and other versions) to be initialized to `None` instead of an empty deque. Our existing unconditional length checks failed on the new `None` default. This seems to block syncing. python/cpython#97020 https://github.com/python/cpython/compare/v3.10.7..v3.10.8#diff-0fee1befb15023abc0dad2623effa93a304946796929f6cb445d11a57821e737 Reported traceback: ```python-traceback 2022-10-12T20:03:59.367 full_node full_node_server : INFO Connected with full_node {'host': '65.34.144.6', 'port': 8444} 2022-10-12T20:03:59.370 full_node full_node_server : ERROR Exception: object of type 'NoneType' has no len(), {'host': '65.34.144.6', 'port': 8444}. Traceback (most recent call last): File "/home/summa/chia-blockchain/chia/server/server.py", line 598, in wrapped_coroutine result = await coroutine File "/home/summa/chia-blockchain/chia/full_node/full_node_api.py", line 114, in new_peak waiter_count = len(self.full_node.new_peak_sem._waiters) TypeError: object of type 'NoneType' has no len() 2022-10-12T20:03:59.371 full_node full_node_server : ERROR Exception: object of type 'NoneType' has no len() <class 'TypeError'>, closing connection {'host': '65.34.144.6', 'port': 8444}. Traceback (most recent call last): File "/home/summa/chia-blockchain/chia/server/server.py", line 608, in api_call response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout) File "/usr/lib/python3.10/asyncio/tasks.py", line 408, in wait_for return await fut File "/home/summa/chia-blockchain/chia/server/server.py", line 605, in wrapped_coroutine raise e File "/home/summa/chia-blockchain/chia/server/server.py", line 598, in wrapped_coroutine result = await coroutine File "/home/summa/chia-blockchain/chia/full_node/full_node_api.py", line 114, in new_peak waiter_count = len(self.full_node.new_peak_sem._waiters) TypeError: object of type 'NoneType' has no len() 2022-10-12T20:03:59.487 full_node full_node_server : INFO Connection closed: 65.34.144.6, node id: 506fe4c05ce6b72bb707471842e552307c7a547aa9ba981175db5c08fa3e47e6 ``` * add LimitedSemaphore
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reviewers
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The main problem was that an unluckily timed task cancellation could cause
the semaphore to be stuck. There were also doubts about strict FIFO ordering
of tasks allowed to pass.
The Semaphore implementation was rewritten to be more similar to Lock.
Many tests for edge cases (including cancellation) were added.
(cherry picked from commit 24e0379)
Co-authored-by: Cyker Way cykerway@gmail.com