| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| Expand Up | @@ -1436,6 +1436,109 @@ application). | |
| list appear empty for the duration, and raises :exc:`ValueError` if it can | ||
| detect that the list has been mutated during a sort. | ||
|
|
||
| .. admonition:: Thread safety | ||
|
|
||
| Reading a single element from a :class:`list` is | ||
| :term:`atomic <atomic operation>`: | ||
|
|
||
| .. code-block:: | ||
| :class: green | ||
|
|
||
| lst[i] # list.__getitem__ | ||
|
|
||
| The following methods traverse the list and use :term:`atomic <atomic operation>` | ||
| reads of each item to perform their function. That means that they may | ||
| return results affected by concurrent modifications: | ||
|
|
||
| .. code-block:: | ||
| :class: maybe | ||
|
|
||
| item in lst | ||
| lst.index(item) | ||
| lst.count(item) | ||
|
|
||
| All of the above methods/operations are also lock-free. They do not block | ||
| concurrent modifications. Other operations that hold a lock will not block | ||
| these from observing intermediate states. | ||
|
Comment thread
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualityIs "intermediate states" correct, or is it just that readers might observe the state either before or after concurrent modification? The way it's written now a very literal-minded reader might conclude that a reader might observe a torn read or other C data race.
Sorry, something went wrong.
All reactions
Copy link
Copy Markdown
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualityIt's not about data races, it's about intermediate states when the lock-free operations race with operations that touch multiple elements. For example, if list.index and list.reverse race with one another (I know, bad example, because list.reverse does have a data race, but let's assume that gets fixed), list.index might return the reversed index, even though some elements might not have been reversed yet.
Sorry, something went wrong.
All reactions
|
||
|
|
||
| All other operations from here on block using the per-object lock. | ||
|
Comment thread
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualityMaybe a glossary reference is useful here? Something like: per-object lock: In free-threading builds ... Whereas in GIL-enabled builds ...
Sorry, something went wrong.
All reactions
|
||
|
|
||
| Writing a single item via ``lst[i] = x`` is safe to call from multiple | ||
| threads and will not corrupt the list. | ||
|
Comment thread
Comment on lines
+1466
to
+1467
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualityIt is safe in the sense that it won't corrupt the list, but should we also mention that it results in a race condition?
Sorry, something went wrong.
All reactions
|
||
|
|
||
| The following operations return new objects and appear | ||
| :term:`atomic <atomic operation>` to other threads: | ||
|
|
||
| .. code-block:: | ||
| :class: good | ||
|
|
||
| lst1 + lst2 # concatenates two lists into a new list | ||
| x * lst # repeats lst x times into a new list | ||
| lst.copy() # returns a shallow copy of the list | ||
|
|
||
| Methods that only operate on a single elements with no shifting required are | ||
| :term:`atomic <atomic operation>`: | ||
|
|
||
| .. code-block:: | ||
| :class: good | ||
|
|
||
| lst.append(x) # append to the end of the list, no shifting required | ||
| lst.pop() # pop element from the end of the list, no shifting required | ||
|
|
||
| The :meth:`~list.clear` method is also :term:`atomic <atomic operation>`. | ||
| Other threads cannot observe elements being removed. | ||
|
|
||
| The :meth:`~list.sort` method is not :term:`atomic <atomic operation>`. | ||
| Other threads cannot observe intermediate states during sorting, but the | ||
| list appears empty for the duration of the sort. | ||
|
|
||
| The following operations may allow lock-free operations to observe | ||
| intermediate states since they modify multiple elements in place: | ||
|
Comment thread
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualityIt's not clear to me what "intermediate states" means here. See my comment above. Would help maybe to clarify globally (i.e. somewhere not in the list docs) what observing an object in an intermediate state means. Is any possible layout of the list while it's being processed possible?
Sorry, something went wrong.
All reactions
|
||
|
|
||
| .. code-block:: | ||
| :class: maybe | ||
|
|
||
| lst.insert(idx, item) # shifts elements | ||
| lst.pop(idx) # idx not at the end of the list, shifts elements | ||
| lst *= x # copies elements in place | ||
|
|
||
| The :meth:`~list.remove` method may allow concurrent modifications since | ||
| element comparison may execute arbitrary Python code (via | ||
| :meth:`~object.__eq__`). | ||
|
|
||
| :meth:`~list.extend` is safe to call from multiple threads. However, its | ||
| guarantees depend on the iterable passed to it. If it is a :class:`list`, a | ||
| :class:`tuple`, a :class:`set`, a :class:`frozenset`, a :class:`dict` or a | ||
| :ref:`dictionary view object <dict-views>` (but not their subclasses), the | ||
| ``extend`` operation is safe from concurrent modifications to the iterable. | ||
| Otherwise, an iterator is created which can be concurrently modified by | ||
| another thread. The same applies to inplace concatenation of a list with | ||
| other iterables when using ``lst += iterable``. | ||
|
|
||
| Similarly, assigning to a list slice with ``lst[i:j] = iterable`` is safe | ||
| to call from multiple threads, but ``iterable`` is only locked when it is | ||
| also a :class:`list` (but not its subclasses). | ||
|
|
||
| Operations that involve multiple accesses, as well as iteration, are never | ||
| atomic. For example: | ||
|
|
||
| .. code-block:: | ||
| :class: bad | ||
|
|
||
| # NOT atomic: read-modify-write | ||
| lst[i] = lst[i] + 1 | ||
|
|
||
| # NOT atomic: check-then-act | ||
| if lst: | ||
| item = lst.pop() | ||
|
|
||
| # NOT thread-safe: iteration while modifying | ||
| for item in lst: | ||
| process(item) # another thread may modify lst | ||
|
|
||
| Consider external synchronization when sharing :class:`list` instances | ||
| across threads. See :ref:`freethreading-python-howto` for more information. | ||
|
Comment thread
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualityThis is getting long enough that maybe it deserves to be in its own page in the reference docs, along with thread-safety notes for all the builtin types. Then we can link to those reference docs from here. I think our hope when we originally wanted to include these notes directly in the docs for the builtins was that these notes would be pretty short. But it turns out there are a decent number of caveats and that hope might not be realistic.
Sorry, something went wrong.
lysnikolaou reacted with thumbs up emoji
All reactions
Copy link
Copy Markdown
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualityI agree with this. Splitting everything out in a new page in the reference docs and then cross-linking sounds like the better approach, especially since dict docs are going to be even longer. @encukou What do you think?
Sorry, something went wrong.
encukou reacted with thumbs up emoji
All reactions
Copy link
Copy Markdown
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualityHi! Sorry I missed the notification.
Sorry, something went wrong.
All reactions
Copy link
Copy Markdown
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualityOk, let's get this in as-is and once we have more built-in types (I have dict and set PRs ready to go), we'll have a better picture of how the new document should look like.
Sorry, something went wrong.
All reactions
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualitySounds good to me then. Maybe let's wait to backport until we're done with the standard library types? But do whatever makes the most sense to you.
Sorry, something went wrong.
All reactions
|
||
|
|
||
|
|
||
| .. _typesseq-tuple: | ||
|
|
||
| Expand Down | ||
| Back | FazBrowse Home | New Git URL |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualityMaybe this could say something like “Operations/methods that involve iteration are generally not atomic, except when used with specific built-in types”, and iteration itself can be moved here?
Mentioning iteration might help people make sense of this, i.e. it's no longer two arbitrary lists of operations/methods.
Then the bad section below would be left only with examples of “manually” combining multiple operations.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualityI don't know how I feel about this. "Operations/methods that involve iteration are generally not atomic" is probably not a mnemonic we want people to use, because there are methods that are atomic but traverse the list. Granted most of those are ones that also mutate it, but e.g. list.copy doesn't.
But the idea of separating iteration from manually combining multiple operations is good. Maybe we should do just that?
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualitySounds good.
(I guess it's “operations that involve arbitrary iterators and/or comparison functions”, but that's too long; readers whom that would help can figure it out from the list.)
As for iteration, it sounds like the guarantees are the same for single-threaded code: iteration of a list that is being modified may skip elements or yield repeated elements, but will not crash or produce elements that were never part of the list. Is that right?
Is this the place to document list iterators -- i.e. what happens if you use a shared iterator in several threads?
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualityI've completely reworded the docs to account for lock-free operations. Is this better?
I think documenting iterators should be done in the Iterator docs, not really individually for each iterator type.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.