Honor adam_w_mode in the CPU multi_tensor_adam binding (#8307)

## Summary

The CPU `fused_adam` extension created its `Adam_Optimizer` once with
default arguments and ignored the `mode` parameter entirely
(`csrc/cpu/adam/fused_adam.cpp`), so `FusedAdam(adam_w_mode=False)` on
the CPU backend always applied decoupled (AdamW) weight decay instead of
L2. Everything else (`lr`, betas, `eps`, `weight_decay`, bias
correction) is already passed per call via `ds_adam_step`; only the
AdamW-vs-L2 flag is fixed at construction. The fix keeps one optimizer
instance per mode (`mode 1 == AdamW`, matching the CUDA kernel's
`ADAM_MODE_1`).

Also trims `test_fused_adam_matches_torch` to fp32: its bf16 cases
compared against `torch.optim` running bf16 math, while the fused
kernels compute in fp32 — never a valid reference. Low-precision dtypes
get an explicit fp32-math reference test in the FusedAdam rework
(#8300).

## How this surfaced

Split out of #8303 at @delock's request: after a master merge,
cpu-torch-latest failed on `test_fused_adam_matches_torch[fp32-adam]`
(98.7% of elements mismatched — systematic, not tolerance noise), and
the investigation traced it to this binding. The fix was verified green
on cpu-torch-latest in #8303's CI (run 32695...) before being extracted
here.

## Validation

- `test_fused_adam_matches_torch[fp32-adam]` / `[fp32-adamw]` now
genuinely exercise both decay modes against `torch.optim.Adam` / `AdamW`
on the active accelerator.
- cpu-torch-latest passed with this exact change as part of #8303's
branch; this PR carries it alone.

---------

Signed-off-by: PKUWZP <zhipeng.rainbowserie@gmail.com>
Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>

Unmanaged gradient accumulation: ZeRO offload support (#8225)

## Summary
* Extends unmanaged gradient accumulation
(`managed_gradient_accumulation=false`) to **ZeRO optimizer-state and
parameter offload** (CPU/NVMe). Follow-up to #8217 (ZeRO stage 3, now
merged).
* Stage 2/3: grads still reduce/partition every `backward()`; `step()`
finalizes deferred offload boundary work (grad norms + FP32/NVMe copy)
via `finalize_gradient_accumulation_boundary()`.
* Stage 1: continues to reduce at `step()` via `allreduce_gradients()`,
which already performs offload boundary finalization when the boundary
flag is true.
* Pipeline parallelism, DeepCompile, Apex AMP, and stage-0/1
`overlap_comm` remain unsupported.

## Test plan
Validated on a 2-GPU node:
* [x] Full `-k Unmanaged` suite (**31 passed**), including:
  * `test_unmanaged_matches_managed_optimizer_offload[1|2|3]`
  * `test_unmanaged_matches_managed_param_offload` (stage 3)
* existing non-offload unmanaged equivalence / varying-GAS / rejection
tests
* [x] Docs updated (`config-json.md`, `training.rst`); previewable on
`rtd-staging`

Made with [Cursor](https://cursor.com)

---------

Signed-off-by: Olatunji Ruwase <tunji.ruwase@snowflake.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>

Update version.txt pre-0.19.4 release (#8218)

Signed-off-by: Logan Adams <loadams@microsoft.com>

Adopt PTF Code of Conduct (#8165)

Signed-off-by: Olatunji Ruwase <tunji.ruwase@snowflake.com>

Mixed-precision: per-policy param/buffer dtype cast (preserve fp32 bu…

…ffers) (#8066)

## Summary
- Add `data_types.param_dtype` and `data_types.buffer_dtype` (both
default `None`), mirroring FSDP `MixedPrecisionPolicy`.
- Replace the blanket `module.half()` / `module.bfloat16()` in
`_configure_distributed_model` with a targeted cast: parameters go to
`param_dtype`; floating buffers keep their loaded dtype unless
`buffer_dtype` is explicitly set.

## Motivation
The blanket cast downcasts every floating buffer, including the rotary
`inv_freq` buffer that HF/FSDP2 keep in fp32. On long contexts the bf16
`inv_freq` loses precision, RoPE angles drift, and logits/grads diverge
from the FSDP2 reference. Preserving fp32 buffers by default fixes this;
`buffer_dtype` is the escape hatch to reproduce the legacy behavior.

## Behavior
- `param_dtype` unset -> derived from the fp16/bf16 enabled flag (legacy
param behavior).
- `buffer_dtype` unset -> buffers keep their loaded dtype (e.g. fp32
`inv_freq`).
- `buffer_dtype` set -> buffers force-cast (legacy blanket-cast parity).

## Test plan
- [ ] `param_dtype=bf16`, `buffer_dtype` unset -> params bf16,
`inv_freq` stays fp32.
- [ ] `buffer_dtype=bf16` -> buffers downcast (legacy parity).
- [ ] bf16/fp16 run with neither key set behaves as before except fp32
buffers preserved.
- [ ] 8B / 32B ZeRO-3 long-context run -> grad_norm tracks the FSDP2
reference.

Made with [Cursor](https://cursor.com)

---------

Signed-off-by: Olatunji Ruwase <tunji.ruwase@snowflake.com>
Signed-off-by: Stas Bekman <stas@stason.org>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Olatunji Ruwase <tjruwase@gmail.com>
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Co-authored-by: Stas Bekman <stas@stason.org>

Fix DeepCompile AOT kwargs patching for PyTorch >= v2.11 (#8024)

DeepCompiles breaks for PyTorch >= v2.11 because these versions can
construct the AOT Autograd backend without a bw_compiler kwarg, while
DeepSpeed's Inductor patch assumes that key is always present.

This PR fixes DeepCompile's AOT Autograd patch so unrelated AOT backend
registrations can pass through unchanged. `TestDeepCompile` passes with
this fix.

Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>

Update version.txt before 0.19.0 release (#7995)

Update version.txt for latest incoming release 0.18.9 (#7935)

Update version (#7903)

Fix hook count performance regression from v0.18.5 (#7886)

Fixes performance regressions reported in #7882 and #7885.

PR #7780 added dynamic hook count computation for reentrant
checkpointing correctness, but placed the call inside every gradient
hook closure. For a model with n parameter tensors, this creates
significant overhead per backward pass.

Summary:

1. Added `should_refresh_expected_hook_count()` predicate that returns
true only at backward phase boundaries (first hook, or new reentrant
phase), so `count_used_parameters_in_backward()` is called once per
phase instead of once per hook.
2. Applied this predicate in ZeRO-1/2 (stage_1_and_2.py) and both ZeRO-3
hook sites (stage3.py), reusing the `cached_max_expected_hooks_seen`
value when refresh isn't needed.
3. Changed enter_backward() to reset hook counters on first real
backward entry, preventing pollution from pre-user-backward autograd
calls (e.g., TiledFusedLogitsLoss).

With 24-layer transformer, ~267M params (147 parameter tensors), ZeRO-2,
8×H100 80GB, bf16, batch size 8, 20 warmup + 20 measured iterations:
  - Before fix: 0.1265s/iter
  - After fix: 0.0505s/iter

---------

Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Co-authored-by: Ramya Ramineni <rraminen@users.noreply.github.com>