FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

[pull] master from pythonnet:master by pull[bot] · Pull Request #50 · QSLee-Net/pythonnet · GitHub

Repository navigation

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension .cs  (29) .py  (2) .rst  (2) .yml  (1) All 4 file types selected
Only manifest files
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
2 changes: 1 addition & 1 deletion .github/workflows/main.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ jobs:
instance: macos-15
suffix: -macos-aarch64-none

python: ["3.11", "3.12", "3.13", "3.14", "3.15"]
python: ["3.11", "3.12", "3.13", "3.14", "3.14t", "3.15", "3.15t"]

exclude:
# fails to call mono methods
Expand Down
1 change: 1 addition & 0 deletions doc/source/index.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ page. Use the `Python.NET issue tracker`_ to report issues.
python
dotnet
codecs
threading
pyreference
reference

Expand Down
214 changes: 214 additions & 0 deletions doc/source/threading.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -0,0 +1,214 @@
Threading
=========

This page explains how Python.NET interacts with the Python Global Interpreter
Lock (GIL) and with managed threads, and what guarantees the runtime makes
when your code is multi-threaded. It covers both classic CPython builds and
the free-threaded build introduced in CPython 3.13 (``Py_GIL_DISABLED``).

The model in one paragraph
--------------------------

Python.NET embeds CPython, so every interaction with a Python object —
including reading a ``PyObject``'s attributes, calling a Python callable,
constructing a Python value, or letting a ``PyObject`` go out of scope — must
happen while the calling thread is *attached* to the interpreter. On a
classic (GIL-enabled) CPython build "attached" means "holds the GIL"; on a
free-threaded build it means "has an active thread state". In both cases the
attachment API is the same: ``Py.GIL()`` on the C# side and
``threading.Thread`` / ``_thread`` on the Python side. Forgetting to attach
will crash the process or corrupt memory.

Acquiring the GIL from C#
-------------------------

When .NET code calls into Python it must hold the GIL. Use the ``Py.GIL()``
disposable to acquire and release it::

using (Py.GIL())
{
dynamic np = Py.Import("numpy");
var arr = np.array(new[] { 1, 2, 3 });
// ... interact with arr ...
}

``Py.GIL()`` is re-entrant: nesting calls on the same thread is harmless and
cheap. Always pair acquisition with disposal — the ``using`` form does this
automatically, and you must release the GIL on the same thread that acquired
it.

If you need a Python object to outlive the ``using`` block, copy what you
need (e.g. ``.As<int[]>()`` or ``new PyObject(value)``) before releasing the
GIL.

Releasing the GIL for long-running .NET work
--------------------------------------------

If a managed call holds the GIL but then does long-running work that does not
touch Python (heavy CPU, blocking I/O, native interop), release the GIL so
other Python threads can run::

IntPtr threadState = PythonEngine.BeginAllowThreads();
try
{
DoCpuHeavyWork(); // safe: no Python C API calls
}
finally
{
PythonEngine.EndAllowThreads(threadState);
}

Inside the ``BeginAllowThreads``/``EndAllowThreads`` block you must not touch
any Python object. If you need to talk to Python from worker threads spawned
in this region, those threads must acquire the GIL themselves with
``Py.GIL()``.

Calling .NET from Python threads
--------------------------------

Calling a managed method from a Python ``threading.Thread`` works
transparently — Python.NET handles GIL acquisition/release around the
managed call. The managed code sees the GIL held on entry and is free to
release it via ``BeginAllowThreads`` if it does its own blocking work.

Calling Python from CLR threads
-------------------------------

A CLR thread that was *not* spawned by Python (a thread-pool task, a
``Thread`` started in C#, an ``async`` continuation that resumed on a
different thread, etc.) must acquire the GIL before touching any
``PyObject``::

Task.Run(() =>
{
using (Py.GIL())
{
// safe to use PyObjects here
}
});

Forgetting this is the most common pythonnet threading bug. Symptoms range
from immediate segfaults to subtle refcount corruption that crashes much
later.

Reference counting and finalizers
---------------------------------

``PyObject`` follows the .NET ``IDisposable`` pattern. ``Dispose()`` (or the
end of a ``using`` block) drops the underlying Python reference; the GC
finalizer queues the same release for the next time Python.NET is on the GIL.

Two practical consequences:

* **Don't share a single ``PyObject`` instance across threads without
serialising access.** ``PyObject`` is not internally locked. If multiple
threads concurrently dispose the same instance, the underlying refcount can
go negative.

* **Don't rely on the GC finalizer running promptly.** The PyObject is only
freed when a Python.NET API later reacquires the GIL. If your application
shuts down without that happening, finalizable PyObjects can be reported as
leaked.

Free-threaded Python (PEP 703)
------------------------------

Starting with the free-threaded CPython 3.13+ build (``Py_GIL_DISABLED``),
the GIL is no longer the serialisation point for Python C API calls.
Python.NET is tested against the ``3.14t`` (free-threaded) interpreter and
behaves as follows under that build:

* ``Py.GIL()`` still acquires a thread state. It is functionally a no-op
for mutual exclusion but is still required for thread-state attachment.
Existing code that uses ``using (Py.GIL())`` continues to work without
changes.
* ``PythonEngine.BeginAllowThreads`` / ``EndAllowThreads`` similarly
manage the thread state and are still needed if you want the GC and
other Python threads to run while you're in long-running unmanaged code.
* Internal Python.NET caches (the reflection cache, generic-type binding
cache, dynamic-dispatch cache, module attribute cache, the interned-
string table, etc.) are thread-safe. You may read and call CLR types
concurrently from any number of threads without external locking.
* The reference-counting protocol uses CPython's ``Py_REFCNT`` symbol on
3.14+, which returns the merged biased + shared refcount; values you read
from ``PyObject.Refcount`` are correct under free-threading.

Behaviour that is *unchanged* between GIL and free-threaded builds:

* A managed object exposed to Python (e.g. via ``System.Object`` or a
Python subclass of a CLR type) is still owned by a single CLR side: you
must not mutate its plain CLR fields from multiple threads without your
own locking. Python.NET only protects its own bookkeeping, not your
domain data.
* Operations on a single ``PyObject`` instance still require external
serialisation — see "Reference counting" above.

Patterns
--------

Concurrent CLR access from Python
"""""""""""""""""""""""""""""""""

Hammering CLR attributes / generic types from many threads is supported::

from threading import Thread
import System
from System.Collections.Generic import List

def worker():
for _ in range(1000):
_ = System.String.Empty
_ = List[int]()

threads = [Thread(target=worker) for _ in range(8)]
for t in threads: t.start()
for t in threads: t.join()

This works on both GIL and free-threaded builds.

Python callback invoked from a managed thread
"""""""""""""""""""""""""""""""""""""""""""""

If a managed component calls back into a Python delegate from a thread it
spawned, that callback path acquires the GIL internally — you do not need to
add ``Py.GIL()`` around the Python code in the delegate.

Spawning a managed thread from inside ``Py.GIL()``
""""""""""""""""""""""""""""""""""""""""""""""""""

If you start a managed thread while holding the GIL and the thread needs to
call back into Python, release the GIL first so the new thread can acquire
it::

using (Py.GIL())
{
var pyCallback = scope.Get("on_done");
PythonEngine.BeginAllowThreads(); // let workers acquire the GIL
try
{
// spawn workers, wait for them...
}
finally
{
PythonEngine.EndAllowThreads(...);
}
}

Without the ``BeginAllowThreads`` the spawned thread blocks forever waiting
for the GIL the parent thread is still holding.

Common pitfalls
---------------

* Holding ``Py.GIL()`` across ``Task.Run`` / ``await`` boundaries. Async
continuations can resume on a different thread; the GIL handle is
thread-bound and must be released on the same thread that acquired it.
* Passing a ``PyObject`` to a managed worker without taking ownership. If
the producer disposes its handle while the consumer is still using it,
the worker will operate on a freed object. Wrap the producer's
``PyObject`` with ``new PyObject(value)`` before handing it off, or use
``NewReference()``.
* Calling a Python callable that does CPU-bound work without releasing the
GIL. Other Python threads cannot make progress in that case, even on a
free-threaded build where the GIL is otherwise a no-op (the callable
itself may still touch contended Python state).
39 changes: 20 additions & 19 deletions src/embed_tests/TestFinalizer.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ private static void FullGCCollect()
{
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect(); // reclaim objects whose finalizers just ran
}

[Test]
Expand All @@ -51,28 +52,28 @@ public void CollectBasicObject()
Finalizer.Instance.BeforeCollect += handler;

IntPtr pyObj = MakeAGarbage(out var shortWeak, out var longWeak);
FullGCCollect();
// The object has been resurrected
Warn.If(
shortWeak.IsAlive,
"The referenced object is alive although it should have been collected",
shortWeak
);
Assert.That(
longWeak.IsAlive,
Is.True,
$"The reference object is not alive although it should still be"
);

// The real contract: after the wrapper is GC'd, the underlying
// Python pointer must end up in Finalizer's queue. Poll because
// .NET Framework / .NET Core differ in how many GC cycles it takes.
List<IntPtr> garbage = null;
for (int attempt = 0; attempt < 10; attempt++)
{
var garbage = Finalizer.Instance.GetCollectedObjects();
Assert.NotZero(garbage.Count, "There should still be garbage around");
Warn.Unless(
garbage.Contains(pyObj),
$"The {nameof(longWeak)} reference doesn't show up in the garbage list",
garbage
);
FullGCCollect();
garbage = Finalizer.Instance.GetCollectedObjects();
if (garbage.Contains(pyObj)) break;
Thread.Sleep(20);
}

Warn.If(shortWeak.IsAlive,
"shortWeak is alive after FullGCCollect; runtime hasn't reclaimed the wrapper yet",
shortWeak);
// longWeak.IsAlive at this point is .NET-GC-implementation-defined
// (Framework reclaims post-finalize objects more eagerly than Core);
// intentionally not asserted.

Assert.That(garbage, Has.Member(pyObj),
"PyObject did not reach Finalizer.Instance.GetCollectedObjects()");
try
{
Finalizer.Instance.Collect();
Expand Down
12 changes: 11 additions & 1 deletion src/embed_tests/TestInterrupt.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,17 @@ import time
Assert.That(asyncCall.Wait(TimeSpan.FromSeconds(5)), Is.True, "Async thread was not interrupted in time");
PythonEngine.EndAllowThreads(threadState);

Assert.That(asyncCall.Result, Is.EqualTo(0));
// On free-threaded CPython 3.14, PyRun_SimpleString may return -1 even
// when the script catches the async-injected KeyboardInterrupt — the
// C-level error indicator depends on which bytecode boundary the
// async-exc fires at, and isn't always cleared the way GIL builds clear
// it. The interrupt firing and the script terminating cleanly are what
// this test exercises; the return code is a side-effect that's only
// deterministic under the GIL.
if (Python.Runtime.Native.ABI.IsFreeThreaded)
Assert.That(asyncCall.Result, Is.AnyOf(0, -1));
else
Assert.That(asyncCall.Result, Is.EqualTo(0));
}
}
}
5 changes: 3 additions & 2 deletions src/embed_tests/TestNativeTypeOffset.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,10 @@ public class TestNativeTypeOffset
public void LoadNativeTypeOffsetClass()
{
PyObject sys = Py.Import("sys");
// We can safely ignore the "m" abi flag
// "m" is benign; "t" (free-threaded) is handled via ABI.ObjectHeadOffset
// rather than the install-time-generated NativeTypeOffset class.
var abiflags = sys.HasAttr("abiflags") ? sys.GetAttr("abiflags").ToString() : "";
abiflags = abiflags.Replace("m", "");
abiflags = abiflags.Replace("m", "").Replace("t", "");
if (!string.IsNullOrEmpty(abiflags))
{
string typeName = "Python.Runtime.NativeTypeOffset, Python.Runtime";
Expand Down
37 changes: 37 additions & 0 deletions src/embed_tests/TestPyBuffer.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
using System;
using System.Runtime.CompilerServices;
using System.Text;
using System.Threading;
using NUnit.Framework;
using Python.Runtime;
using Python.Runtime.Codecs;
Expand Down Expand Up @@ -129,6 +130,42 @@ public void MultidimensionalNumPyArray()
});
}

[Test]
public void ConcurrentDispose()
{
// Two threads racing on Dispose() must not double-release the view —
// Interlocked.Exchange on disposedValue gates PyBuffer_Release.
// Smoke test: no crash, exception, or buffer-protocol violation.
using var _ = Py.GIL();
using var arr = ByteArrayFromAsciiString("hello world! !$%&/()=?");

const int iterations = 200;
for (int i = 0; i < iterations; i++)
{
PyBuffer buf = arr.GetBuffer();

IntPtr ts = PythonEngine.BeginAllowThreads();
using var barrier = new Barrier(2);
Exception captured = null;
Action race = () =>
{
try
{
barrier.SignalAndWait();
using (Py.GIL()) buf.Dispose();
}
catch (Exception ex) { Interlocked.CompareExchange(ref captured, ex, null); }
};
var t1 = new Thread(() => race());
var t2 = new Thread(() => race());
t1.Start(); t2.Start();
t1.Join(); t2.Join();
PythonEngine.EndAllowThreads(ts);

if (captured != null) throw captured;
}
}

[MethodImpl(MethodImplOptions.NoInlining)]
static void MakeBufAndLeak(PyObject bufProvider)
{
Expand Down
Loading
Loading

Back | FazBrowse Home | New Git URL