CPython 3.14.0/3.14.1 fatally crash Ray async actors running on boost
fiber stacks (python/cpython#141944, fixed upstream in 3.14.2), which
kills the JobSupervisor on every Anyscale job. The py3.14 images are
stuck on 3.14.0 for two reasons: the unpinned 'conda install
python=3.14' resolve is frozen in the wanda layer cache, and the
libffi=3.4.6 pin in base-deps forces conda to pick python 3.14.0 even
on a fresh build (newer 3.14.x conda-forge builds require libffi>=3.5).

Pin python 3.14.6 in base-deps and base-slim, and let 3.14 bring its
own libffi. Verified: the async-actor repro crashes in
rayproject/ray:nightly-py314-cpu (3.14.0) and passes with the same
nightly cp314 wheel on 3.14.6.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

CPython 3.14.0/3.14.1 fatally crash Ray async actors running on boost
fiber stacks (python/cpython#141944, fixed upstream in 3.14.2), which
kills the JobSupervisor on every Anyscale job. The py3.14 base image
resolves python 3.14.0 because the exact libffi=3.4.6 pin, installed as
a separate conda step, re-solves the env and downgrades python to the
only 3.14 build compatible with libffi<3.5.

Replace the two-step install with a single solve using a libffi floor
(>=3.4.6, keeping the intent of the original pin): every python version
then resolves its newest patch release with a compatible libffi —
3.14.6 + libffi 3.5.2 for py3.14. Verified via dry-run for 3.10-3.14
and ctypes smoke tests on 3.10 (libffi 3.7.0) and 3.14 (libffi 3.5.2);
the async-actor repro crashes on 3.14.0 and passes with the same
nightly cp314 wheel on 3.14.6.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…mage resolving CPython 3.14.0 (#64857)

## Description

### Release smoke test
Adds a `hello_world_py314` nightly smoke release test (aws variation
only), mirroring the existing `hello_world_py313` entry:

- `release/ray_release/schema.json` — add `"3.14"` to the `python` enum
(release-test config validation rejects `python: "3.14"` without this).
- `release/ray_release/config.py` — add `"3.14"` to the cpu/cu123 BYOD
python allowlist (the parallel gate the release runner walks).
- `release/release_tests.yaml` — new `hello_world_py314` test, nightly,
`byod: {}`, same `hello_world_compute_config.yaml` as the other
hello_world tests.

The py3.14 `ray-anyscale` release-test images (cpu + cuda) are already
built and published on master via `.buildkite/release/build.rayci.yml`,
so no image plumbing is needed here.

### Base-image fix (what made the smoke test fail)
The first run of this test failed with the JobSupervisor dying at job
startup: `Fatal Python error: _Py_CheckRecursiveCall: Unrecoverable
stack overflow`. Root cause: the py3.14 images ship **CPython 3.14.0**,
which fatally crashes any Ray async actor running on a boost fiber stack
(python/cpython#141944, fixed upstream in **3.14.2**).

Why the images resolve 3.14.0: `docker/base-deps/Dockerfile` installs an
exact `libffi=3.4.6` pin as a *separate* conda step after installing
python. That second solve downgrades python to the only 3.14 build
compatible with `libffi<3.5` — which is 3.14.0. (The stale wanda layer
cache compounds this, but even a fresh rebuild today re-resolves 3.14.0
because of the pin.)

Fix: replace the two-step install with a **single solve using a libffi
floor** (`libffi>=3.4.6`, preserving the intent of the original pin —
the 3.4.2/defaults-channel libffi was buggy). No per-version special
casing; every python version resolves its newest patch release with a
compatible libffi. Resolved versions today: py3.10→3.10.20 (libffi
3.7.0), py3.11→3.11.15, py3.12→3.12.13, py3.13→3.13.14 (libffi 3.5.2),
py3.14→**3.14.6** (libffi 3.5.2). The Dockerfile change also busts the
stale wanda cache.

Note: #64772 (fiber stack-protection re-anchoring) is complementary, not
a fix for this crash — its `PyUnstable_ThreadState_SetStackProtection`
call only exists on 3.14.2+, so it no-ops on the 3.14.0 currently in the
images. Once this lands, #64772 fixes the remaining per-task async-actor
memory leak.

## Verification

- Reproduced the crash: async-actor repro
(`ray.get(A.remote().hi.remote())` with an `async def` method) dies in
`rayproject/ray:nightly-py314-cpu` (CPython 3.14.0) with the exact
failure signature from release-test job
`prodjob_d4dctduzm3h6eu812vrrehiuzl`.
- Verified the fix: the same unpatched nightly cp314 wheel on CPython
3.14.6 (`python:3.14-slim`) runs the repro successfully.
- Verified the combined solve under miniforge 24.11.3-0 (same as the
Dockerfile): dry-runs for python 3.10–3.14 all resolve (versions above),
plus real installs with `ctypes` smoke tests on 3.10 (libffi 3.7.0) and
3.14.6 (libffi 3.5.2).
- `python -m pytest -q release/ray_release/tests/test_config.py` — 23
passed; full collection validates (319 tests) including
`hello_world_py314.aws`.

## Duplicate-work note

#63237 contains an earlier version of the release-test config bundled
with image-build plumbing that has since landed on master through other
PRs. This PR carves out the remaining release-test config plus the
base-image fix; #63237 can be closed or rebased down to the raylet fix.

AI assistance (Claude Code) was used for this PR; all changes reviewed
by the submitter.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…protection to fiber stacks (#64772)

# Description

On Python 3.14 + Linux, every async-actor task permanently leaks ~518
KiB of live malloc (the per-task `asyncio.Task`,
`concurrent.futures.Future`, Cython coroutine + scopes, and two msgpack
`Packer`s with 256 KiB internal buffers).
Closes #63290

### Root cause

**1. CPython 3.14 changed how it avoids stack overflow when freeing
objects.** Freeing one object can recursively free many others (a dict
frees its values, which free their contents, …), and each level is a
nested C call. To keep that from overflowing the C stack, CPython has
long had a safety mechanism (the "trashcan"): when it decides it's too
deep, it doesn't free the object right away. Instead it parks the object
on a per-thread *delete-later* list and drains the list once there's
stack headroom again. Up to 3.13, "too deep" was a simple recursion
counter. In 3.14 it's decided by comparing the actual machine **stack
pointer** against the stack bounds CPython recorded for the thread when
it attached (from pthreads, on Linux).

**2. Ray async actors don't run task code on the thread's normal
stack.** Each task executes on a small 256 KiB boost fiber stack
allocated elsewhere in memory. The problem is that CPython still thinks
the thread runs on its original pthread stack.

So while a task runs on a fiber, every "am I near the stack limit?"
check compares the fiber's stack pointer against the *pthread* stack's
bounds. On Linux, fiber stacks happen to be allocated at lower addresses
than the pthread stack, so CPython concludes the stack is hopelessly
overflowed and parks **every** object freed during the task (including
return-value serialization and end-of-task cleanup) on the delete-later
list.

That list is only ever drained by a later free on the same thread state
at a healthy stack margin, which never happens here as the Ray thread
only runs on fibers and Ray creates a fresh Python thread state per task
and destroys it at task end. This means that CPython destroys a thread
state **without draining its delete-later list** and the parked objects
are orphaned permanently. That's the leak.

Why the confusing symptoms:

- `boost::make_fcontext` in the issue's flamegraphs just marks *where*
the leaked allocations were made (on a fiber stack); the fiber stacks
themselves are freed correctly.
- macOS is unaffected only by luck: fiber stacks there land at *higher*
addresses than the pthread stack, so the check passes.
- 3.13 and earlier are unaffected because their trashcan uses the
counter, not the stack pointer.

### Fix

CPython 3.14.2 added an official API for exactly this situation:
`PyUnstable_ThreadState_SetStackProtection` (python/cpython#141661) lets
an embedder tell CPython "this thread is currently executing on *this*
stack." We call it with the fiber's stack bounds:

- at async-actor task entry in `task_execution_handler`, and
- whenever a fiber resumes after `YieldCurrentFiber` (concurrent fibers
share the thread state, so each must re-register its own stack).

With the bounds correct, the near-limit check returns to normal
behavior: objects are freed immediately, and the rare genuinely-deep
free is parked and then properly drained.

Implementation notes: the symbol is looked up via `dlsym`, so `_raylet`
still imports on 3.14.0/3.14.1 (fix skipped there; those releases have a
more severe, since-fixed stack-check bug anyway, python/cpython#141944).
No-op below 3.14 (preprocessor-gated) and on Windows. Stack bounds are
derived from the current stack pointer minus a conservative allowance
for stack already used, so the protection errs toward triggering
slightly early rather than missing an overflow. Side benefit: fibers
gain real C-stack overflow protection (RecursionError) on 3.14, which
they currently lack entirely (`boost::fibers::fixedsize_stack` has no
guard pages). Also makes `FiberState::kStackSize` public so the
anchoring uses the real fiber stack size.

## Related issue number

Closes #63290. Supersedes #63284 (same diagnosis direction, but
hand-rolled `_PyThreadStateImpl` offsets, a deliberate
`gilstate_counter` leak that freezes non-main threads, and a crash
premise that CPython 3.14.2 already fixed upstream).

## Checks

- Verified with a locally built cp314 Linux (aarch64, python:3.14.6
docker) wheel:
- refcount probe: **+4.00 refs/task → 0.00/task** (100 tasks)
- `__del__` deferral probe: dealloc during return serialization on the
fiber **deferred → immediate**
- live-malloc probe (`mallinfo2`, 300 tasks/shape): **~518 KiB/task → ~3
KiB/task** across async call → dict/bytes, async generator, sync
generator on async actor
- reporter-shaped streaming workload (400 tasks, 10 concurrent
sessions): live-malloc delta **0.2 MB total**, fiber-sized mapped
regions 0 → 0
- async-actor smoke: correctness (echo, state, async generators,
recursion), concurrency (20 overlapping 0.5 s sleeps in 0.51 s)
- throughput A/B (500 sequential echo tasks, 3 runs fixed / 2 runs
baseline, same container image): fixed 5624–6076 tasks/s vs unpatched
4551–4825 tasks/s meaning no regression (the unpatched build is slower
while leaking)
- baseline (unpatched) wheel from the same tree reproduces the bug:
+4.00 refs/task, fiber dealloc deferred=True

note: fable did a majority of the heavy lifting in this investigation
with prompting on what to check next and validate the solution

---------

Signed-off-by: Mark Towers <mark@anyscale.com>
Signed-off-by: myan <myan@anyscale.com>
Co-authored-by: Mark Towers <mark@anyscale.com>
Co-authored-by: myan <myan@anyscale.com>
Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com>

#65177)

…protection to fiber stacks (#64772)

# Description

On Python 3.14 + Linux, every async-actor task permanently leaks ~518
KiB of live malloc (the per-task `asyncio.Task`,
`concurrent.futures.Future`, Cython coroutine + scopes, and two msgpack
`Packer`s with 256 KiB internal buffers).
Closes #63290

### Root cause

**1. CPython 3.14 changed how it avoids stack overflow when freeing
objects.** Freeing one object can recursively free many others (a dict
frees its values, which free their contents, …), and each level is a
nested C call. To keep that from overflowing the C stack, CPython has
long had a safety mechanism (the "trashcan"): when it decides it's too
deep, it doesn't free the object right away. Instead it parks the object
on a per-thread *delete-later* list and drains the list once there's
stack headroom again. Up to 3.13, "too deep" was a simple recursion
counter. In 3.14 it's decided by comparing the actual machine **stack
pointer** against the stack bounds CPython recorded for the thread when
it attached (from pthreads, on Linux).

**2. Ray async actors don't run task code on the thread's normal
stack.** Each task executes on a small 256 KiB boost fiber stack
allocated elsewhere in memory. The problem is that CPython still thinks
the thread runs on its original pthread stack.

So while a task runs on a fiber, every "am I near the stack limit?"
check compares the fiber's stack pointer against the *pthread* stack's
bounds. On Linux, fiber stacks happen to be allocated at lower addresses
than the pthread stack, so CPython concludes the stack is hopelessly
overflowed and parks **every** object freed during the task (including
return-value serialization and end-of-task cleanup) on the delete-later
list.

That list is only ever drained by a later free on the same thread state
at a healthy stack margin, which never happens here as the Ray thread
only runs on fibers and Ray creates a fresh Python thread state per task
and destroys it at task end. This means that CPython destroys a thread
state **without draining its delete-later list** and the parked objects
are orphaned permanently. That's the leak.

Why the confusing symptoms:

- `boost::make_fcontext` in the issue's flamegraphs just marks *where*
the leaked allocations were made (on a fiber stack); the fiber stacks
themselves are freed correctly.
- macOS is unaffected only by luck: fiber stacks there land at *higher*
addresses than the pthread stack, so the check passes.
- 3.13 and earlier are unaffected because their trashcan uses the
counter, not the stack pointer.

### Fix

CPython 3.14.2 added an official API for exactly this situation:
`PyUnstable_ThreadState_SetStackProtection` (python/cpython#141661) lets
an embedder tell CPython "this thread is currently executing on *this*
stack." We call it with the fiber's stack bounds:

- at async-actor task entry in `task_execution_handler`, and
- whenever a fiber resumes after `YieldCurrentFiber` (concurrent fibers
share the thread state, so each must re-register its own stack).

With the bounds correct, the near-limit check returns to normal
behavior: objects are freed immediately, and the rare genuinely-deep
free is parked and then properly drained.

Implementation notes: the symbol is looked up via `dlsym`, so `_raylet`
still imports on 3.14.0/3.14.1 (fix skipped there; those releases have a
more severe, since-fixed stack-check bug anyway, python/cpython#141944).
No-op below 3.14 (preprocessor-gated) and on Windows. Stack bounds are
derived from the current stack pointer minus a conservative allowance
for stack already used, so the protection errs toward triggering
slightly early rather than missing an overflow. Side benefit: fibers
gain real C-stack overflow protection (RecursionError) on 3.14, which
they currently lack entirely (`boost::fibers::fixedsize_stack` has no
guard pages). Also makes `FiberState::kStackSize` public so the
anchoring uses the real fiber stack size.

## Related issue number

Closes #63290. Supersedes #63284 (same diagnosis direction, but
hand-rolled `_PyThreadStateImpl` offsets, a deliberate
`gilstate_counter` leak that freezes non-main threads, and a crash
premise that CPython 3.14.2 already fixed upstream).

## Checks

- Verified with a locally built cp314 Linux (aarch64, python:3.14.6
docker) wheel:
- refcount probe: **+4.00 refs/task → 0.00/task** (100 tasks)
- `__del__` deferral probe: dealloc during return serialization on the
fiber **deferred → immediate**
- live-malloc probe (`mallinfo2`, 300 tasks/shape): **~518 KiB/task → ~3
KiB/task** across async call → dict/bytes, async generator, sync
generator on async actor
- reporter-shaped streaming workload (400 tasks, 10 concurrent
sessions): live-malloc delta **0.2 MB total**, fiber-sized mapped
regions 0 → 0
- async-actor smoke: correctness (echo, state, async generators,
recursion), concurrency (20 overlapping 0.5 s sleeps in 0.51 s)
- throughput A/B (500 sequential echo tasks, 3 runs fixed / 2 runs
baseline, same container image): fixed 5624–6076 tasks/s vs unpatched
4551–4825 tasks/s meaning no regression (the unpatched build is slower
while leaking)
- baseline (unpatched) wheel from the same tree reproduces the bug:
+4.00 refs/task, fiber dealloc deferred=True

note: fable did a majority of the heavy lifting in this investigation
with prompting on what to check next and validate the solution

---------






(cherry picked from commit 35591ba)

Signed-off-by: Mark Towers <mark@anyscale.com>
Signed-off-by: myan <myan@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Co-authored-by: Mark Towers <mark.m.towers@gmail.com>
Co-authored-by: Mark Towers <mark@anyscale.com>
Co-authored-by: myan <myan@anyscale.com>
Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com>

…protection to fiber stacks (ray-project#64772)

# Description

On Python 3.14 + Linux, every async-actor task permanently leaks ~518
KiB of live malloc (the per-task `asyncio.Task`,
`concurrent.futures.Future`, Cython coroutine + scopes, and two msgpack
`Packer`s with 256 KiB internal buffers).
Closes ray-project#63290

### Root cause

**1. CPython 3.14 changed how it avoids stack overflow when freeing
objects.** Freeing one object can recursively free many others (a dict
frees its values, which free their contents, …), and each level is a
nested C call. To keep that from overflowing the C stack, CPython has
long had a safety mechanism (the "trashcan"): when it decides it's too
deep, it doesn't free the object right away. Instead it parks the object
on a per-thread *delete-later* list and drains the list once there's
stack headroom again. Up to 3.13, "too deep" was a simple recursion
counter. In 3.14 it's decided by comparing the actual machine **stack
pointer** against the stack bounds CPython recorded for the thread when
it attached (from pthreads, on Linux).

**2. Ray async actors don't run task code on the thread's normal
stack.** Each task executes on a small 256 KiB boost fiber stack
allocated elsewhere in memory. The problem is that CPython still thinks
the thread runs on its original pthread stack.

So while a task runs on a fiber, every "am I near the stack limit?"
check compares the fiber's stack pointer against the *pthread* stack's
bounds. On Linux, fiber stacks happen to be allocated at lower addresses
than the pthread stack, so CPython concludes the stack is hopelessly
overflowed and parks **every** object freed during the task (including
return-value serialization and end-of-task cleanup) on the delete-later
list.

That list is only ever drained by a later free on the same thread state
at a healthy stack margin, which never happens here as the Ray thread
only runs on fibers and Ray creates a fresh Python thread state per task
and destroys it at task end. This means that CPython destroys a thread
state **without draining its delete-later list** and the parked objects
are orphaned permanently. That's the leak.

Why the confusing symptoms:

- `boost::make_fcontext` in the issue's flamegraphs just marks *where*
the leaked allocations were made (on a fiber stack); the fiber stacks
themselves are freed correctly.
- macOS is unaffected only by luck: fiber stacks there land at *higher*
addresses than the pthread stack, so the check passes.
- 3.13 and earlier are unaffected because their trashcan uses the
counter, not the stack pointer.

### Fix

CPython 3.14.2 added an official API for exactly this situation:
`PyUnstable_ThreadState_SetStackProtection` (python/cpython#141661) lets
an embedder tell CPython "this thread is currently executing on *this*
stack." We call it with the fiber's stack bounds:

- at async-actor task entry in `task_execution_handler`, and
- whenever a fiber resumes after `YieldCurrentFiber` (concurrent fibers
share the thread state, so each must re-register its own stack).

With the bounds correct, the near-limit check returns to normal
behavior: objects are freed immediately, and the rare genuinely-deep
free is parked and then properly drained.

Implementation notes: the symbol is looked up via `dlsym`, so `_raylet`
still imports on 3.14.0/3.14.1 (fix skipped there; those releases have a
more severe, since-fixed stack-check bug anyway, python/cpython#141944).
No-op below 3.14 (preprocessor-gated) and on Windows. Stack bounds are
derived from the current stack pointer minus a conservative allowance
for stack already used, so the protection errs toward triggering
slightly early rather than missing an overflow. Side benefit: fibers
gain real C-stack overflow protection (RecursionError) on 3.14, which
they currently lack entirely (`boost::fibers::fixedsize_stack` has no
guard pages). Also makes `FiberState::kStackSize` public so the
anchoring uses the real fiber stack size.

## Related issue number

Closes ray-project#63290. Supersedes ray-project#63284 (same diagnosis direction, but
hand-rolled `_PyThreadStateImpl` offsets, a deliberate
`gilstate_counter` leak that freezes non-main threads, and a crash
premise that CPython 3.14.2 already fixed upstream).

## Checks

- Verified with a locally built cp314 Linux (aarch64, python:3.14.6
docker) wheel:
- refcount probe: **+4.00 refs/task → 0.00/task** (100 tasks)
- `__del__` deferral probe: dealloc during return serialization on the
fiber **deferred → immediate**
- live-malloc probe (`mallinfo2`, 300 tasks/shape): **~518 KiB/task → ~3
KiB/task** across async call → dict/bytes, async generator, sync
generator on async actor
- reporter-shaped streaming workload (400 tasks, 10 concurrent
sessions): live-malloc delta **0.2 MB total**, fiber-sized mapped
regions 0 → 0
- async-actor smoke: correctness (echo, state, async generators,
recursion), concurrency (20 overlapping 0.5 s sleeps in 0.51 s)
- throughput A/B (500 sequential echo tasks, 3 runs fixed / 2 runs
baseline, same container image): fixed 5624–6076 tasks/s vs unpatched
4551–4825 tasks/s meaning no regression (the unpatched build is slower
while leaking)
- baseline (unpatched) wheel from the same tree reproduces the bug:
+4.00 refs/task, fiber dealloc deferred=True

note: fable did a majority of the heavy lifting in this investigation
with prompting on what to check next and validate the solution

---------

Signed-off-by: Mark Towers <mark@anyscale.com>
Signed-off-by: myan <myan@anyscale.com>
Co-authored-by: Mark Towers <mark@anyscale.com>
Co-authored-by: myan <myan@anyscale.com>
Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com>

…mage resolving CPython 3.14.0 (ray-project#64857)

## Description

### Release smoke test
Adds a `hello_world_py314` nightly smoke release test (aws variation
only), mirroring the existing `hello_world_py313` entry:

- `release/ray_release/schema.json` — add `"3.14"` to the `python` enum
(release-test config validation rejects `python: "3.14"` without this).
- `release/ray_release/config.py` — add `"3.14"` to the cpu/cu123 BYOD
python allowlist (the parallel gate the release runner walks).
- `release/release_tests.yaml` — new `hello_world_py314` test, nightly,
`byod: {}`, same `hello_world_compute_config.yaml` as the other
hello_world tests.

The py3.14 `ray-anyscale` release-test images (cpu + cuda) are already
built and published on master via `.buildkite/release/build.rayci.yml`,
so no image plumbing is needed here.

### Base-image fix (what made the smoke test fail)
The first run of this test failed with the JobSupervisor dying at job
startup: `Fatal Python error: _Py_CheckRecursiveCall: Unrecoverable
stack overflow`. Root cause: the py3.14 images ship **CPython 3.14.0**,
which fatally crashes any Ray async actor running on a boost fiber stack
(python/cpython#141944, fixed upstream in **3.14.2**).

Why the images resolve 3.14.0: `docker/base-deps/Dockerfile` installs an
exact `libffi=3.4.6` pin as a *separate* conda step after installing
python. That second solve downgrades python to the only 3.14 build
compatible with `libffi<3.5` — which is 3.14.0. (The stale wanda layer
cache compounds this, but even a fresh rebuild today re-resolves 3.14.0
because of the pin.)

Fix: replace the two-step install with a **single solve using a libffi
floor** (`libffi>=3.4.6`, preserving the intent of the original pin —
the 3.4.2/defaults-channel libffi was buggy). No per-version special
casing; every python version resolves its newest patch release with a
compatible libffi. Resolved versions today: py3.10→3.10.20 (libffi
3.7.0), py3.11→3.11.15, py3.12→3.12.13, py3.13→3.13.14 (libffi 3.5.2),
py3.14→**3.14.6** (libffi 3.5.2). The Dockerfile change also busts the
stale wanda cache.

Note: ray-project#64772 (fiber stack-protection re-anchoring) is complementary, not
a fix for this crash — its `PyUnstable_ThreadState_SetStackProtection`
call only exists on 3.14.2+, so it no-ops on the 3.14.0 currently in the
images. Once this lands, ray-project#64772 fixes the remaining per-task async-actor
memory leak.

## Verification

- Reproduced the crash: async-actor repro
(`ray.get(A.remote().hi.remote())` with an `async def` method) dies in
`rayproject/ray:nightly-py314-cpu` (CPython 3.14.0) with the exact
failure signature from release-test job
`prodjob_d4dctduzm3h6eu812vrrehiuzl`.
- Verified the fix: the same unpatched nightly cp314 wheel on CPython
3.14.6 (`python:3.14-slim`) runs the repro successfully.
- Verified the combined solve under miniforge 24.11.3-0 (same as the
Dockerfile): dry-runs for python 3.10–3.14 all resolve (versions above),
plus real installs with `ctypes` smoke tests on 3.10 (libffi 3.7.0) and
3.14.6 (libffi 3.5.2).
- `python -m pytest -q release/ray_release/tests/test_config.py` — 23
passed; full collection validates (319 tests) including
`hello_world_py314.aws`.

## Duplicate-work note

ray-project#63237 contains an earlier version of the release-test config bundled
with image-build plumbing that has since landed on master through other
PRs. This PR carves out the remaining release-test config plus the
base-image fix; ray-project#63237 can be closed or rebased down to the raylet fix.

AI assistance (Claude Code) was used for this PR; all changes reviewed
by the submitter.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: 400Ping <jiekaichang@apache.org>

…protection to fiber stacks (ray-project#64772)

# Description

On Python 3.14 + Linux, every async-actor task permanently leaks ~518
KiB of live malloc (the per-task `asyncio.Task`,
`concurrent.futures.Future`, Cython coroutine + scopes, and two msgpack
`Packer`s with 256 KiB internal buffers).
Closes ray-project#63290

### Root cause

**1. CPython 3.14 changed how it avoids stack overflow when freeing
objects.** Freeing one object can recursively free many others (a dict
frees its values, which free their contents, …), and each level is a
nested C call. To keep that from overflowing the C stack, CPython has
long had a safety mechanism (the "trashcan"): when it decides it's too
deep, it doesn't free the object right away. Instead it parks the object
on a per-thread *delete-later* list and drains the list once there's
stack headroom again. Up to 3.13, "too deep" was a simple recursion
counter. In 3.14 it's decided by comparing the actual machine **stack
pointer** against the stack bounds CPython recorded for the thread when
it attached (from pthreads, on Linux).

**2. Ray async actors don't run task code on the thread's normal
stack.** Each task executes on a small 256 KiB boost fiber stack
allocated elsewhere in memory. The problem is that CPython still thinks
the thread runs on its original pthread stack.

So while a task runs on a fiber, every "am I near the stack limit?"
check compares the fiber's stack pointer against the *pthread* stack's
bounds. On Linux, fiber stacks happen to be allocated at lower addresses
than the pthread stack, so CPython concludes the stack is hopelessly
overflowed and parks **every** object freed during the task (including
return-value serialization and end-of-task cleanup) on the delete-later
list.

That list is only ever drained by a later free on the same thread state
at a healthy stack margin, which never happens here as the Ray thread
only runs on fibers and Ray creates a fresh Python thread state per task
and destroys it at task end. This means that CPython destroys a thread
state **without draining its delete-later list** and the parked objects
are orphaned permanently. That's the leak.

Why the confusing symptoms:

- `boost::make_fcontext` in the issue's flamegraphs just marks *where*
the leaked allocations were made (on a fiber stack); the fiber stacks
themselves are freed correctly.
- macOS is unaffected only by luck: fiber stacks there land at *higher*
addresses than the pthread stack, so the check passes.
- 3.13 and earlier are unaffected because their trashcan uses the
counter, not the stack pointer.

### Fix

CPython 3.14.2 added an official API for exactly this situation:
`PyUnstable_ThreadState_SetStackProtection` (python/cpython#141661) lets
an embedder tell CPython "this thread is currently executing on *this*
stack." We call it with the fiber's stack bounds:

- at async-actor task entry in `task_execution_handler`, and
- whenever a fiber resumes after `YieldCurrentFiber` (concurrent fibers
share the thread state, so each must re-register its own stack).

With the bounds correct, the near-limit check returns to normal
behavior: objects are freed immediately, and the rare genuinely-deep
free is parked and then properly drained.

Implementation notes: the symbol is looked up via `dlsym`, so `_raylet`
still imports on 3.14.0/3.14.1 (fix skipped there; those releases have a
more severe, since-fixed stack-check bug anyway, python/cpython#141944).
No-op below 3.14 (preprocessor-gated) and on Windows. Stack bounds are
derived from the current stack pointer minus a conservative allowance
for stack already used, so the protection errs toward triggering
slightly early rather than missing an overflow. Side benefit: fibers
gain real C-stack overflow protection (RecursionError) on 3.14, which
they currently lack entirely (`boost::fibers::fixedsize_stack` has no
guard pages). Also makes `FiberState::kStackSize` public so the
anchoring uses the real fiber stack size.

## Related issue number

Closes ray-project#63290. Supersedes ray-project#63284 (same diagnosis direction, but
hand-rolled `_PyThreadStateImpl` offsets, a deliberate
`gilstate_counter` leak that freezes non-main threads, and a crash
premise that CPython 3.14.2 already fixed upstream).

## Checks

- Verified with a locally built cp314 Linux (aarch64, python:3.14.6
docker) wheel:
- refcount probe: **+4.00 refs/task → 0.00/task** (100 tasks)
- `__del__` deferral probe: dealloc during return serialization on the
fiber **deferred → immediate**
- live-malloc probe (`mallinfo2`, 300 tasks/shape): **~518 KiB/task → ~3
KiB/task** across async call → dict/bytes, async generator, sync
generator on async actor
- reporter-shaped streaming workload (400 tasks, 10 concurrent
sessions): live-malloc delta **0.2 MB total**, fiber-sized mapped
regions 0 → 0
- async-actor smoke: correctness (echo, state, async generators,
recursion), concurrency (20 overlapping 0.5 s sleeps in 0.51 s)
- throughput A/B (500 sequential echo tasks, 3 runs fixed / 2 runs
baseline, same container image): fixed 5624–6076 tasks/s vs unpatched
4551–4825 tasks/s meaning no regression (the unpatched build is slower
while leaking)
- baseline (unpatched) wheel from the same tree reproduces the bug:
+4.00 refs/task, fiber dealloc deferred=True

note: fable did a majority of the heavy lifting in this investigation
with prompting on what to check next and validate the solution

---------

Signed-off-by: Mark Towers <mark@anyscale.com>
Signed-off-by: myan <myan@anyscale.com>
Co-authored-by: Mark Towers <mark@anyscale.com>
Co-authored-by: myan <myan@anyscale.com>
Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com>
Signed-off-by: 400Ping <jiekaichang@apache.org>