gh-125828 : Fix 'get_value' missing for [Bounded]Semaphore on multiprocessing MacOSX - C workaround - #130913
Open
YvesDup wants to merge 36 commits into
Open
gh-125828 : Fix 'get_value' missing for [Bounded]Semaphore on multiprocessing MacOSX - C workaround#130913YvesDup wants to merge 36 commits into
YvesDup wants to merge 36 commits into
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
YvesDup
marked this pull request as draft
April 15, 2025 12:32
Copy link
Copy Markdown
|
This PR is stale because it has been open for 30 days with no activity. |
Sorry, something went wrong.
1. All past blocking points are resolved.
2. Atomicity,
+ Implementation and rebuild Semlock are atomic (protected via the dedicated semaphore),
+ Release, get_value and is_zero methods are atomic too (protected via the mutex of each semaphore),
+ Acquire function is nearly-atomic.
3. Shared memory,
+ The shared memory and its associated semphore have unique names based on
the main process ID.
+ a new counter was added to count pending thread
when blocking acquire operations happen.
+ When a Semlock is destroyed, the shared memory and its dediocted semaphore are destroyed too
if there is no more opened Semclock.
Copy link
Copy Markdown
Documentation build overview80 files changed · ± 78 modified · - 2 deleted ± Modified
- Deleted |
Sorry, something went wrong.
+ refactoring code on SemLock_imp and SemLock__rebuild_impl, + change conditional compilation value PyDEBUG to Py_REF_DEBUG, + add sizeof CounteObject in the header + add 3 unit tests
Copy link
Copy Markdown
Contributor
Author
|
This last version is always based on a shared memory protected with a lock.
Names of these shared memory and lock are unique for the main process and all its child processes. |
Sorry, something went wrong.
YvesDup
marked this pull request as ready for review
June 2, 2026 09:52
- Fix bug: the lost of errno value in SemLock_impl, - Reduce shared memory sise, used kern.posix.max.sem value instaed of SC_SEM_NSEMS_MAX value for array size of shared memeory, - Add a unit test that raises an error when the maximum opened sems is reached. Last code reviews are relaized with Claude code
…/check-c-globals.py`` tool
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reviewers
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Following #125829 discussions and after a long deeping in the _multiprocessing.semaphore.c C file, I suggest a workaround based on a shared memory that contains an array to store all the internal semaphore counters.
At the moment, all the tests are passing, but I'm having 2 side problems, the second of which is really embarrassing:
Regarding this last point, I wonder if only the process that opened the shared memory should call this function? Or should all processes that called shm_open call shm_unlink? Unfortunately, the Posix documentation isn't precise.
I worked on the count of opened/closed processes, with the aim of calling shm_unlink when the last process has closed, without success. I also had a look to the posixshmem module, without success too.
Could the resource_tracker and/or the use of the Finalize class of the multiprocessing.util module help me to fix this problem ? Is it possible to use these 2 Python libs from C API ?
This memory management difficulty make me think that this choice may not be appropriate. I am a bit stuck.
Thank you for your feedback and ideas.