fix(sweep): 🐛 split a batch that runs out of GPU memory (#60)
* docs(spec): 📝 design for an SRC sweep over the GPUs of a node
Phase 2 of the multi-GPU plan: one process per GPU, NCCL collectives,
the sketch index split among the GPUs, with a communicator passed in
Resources. Targets strong scaling at the Phase 1 reference size.
Assisted-by: Pi:claude-opus-5-5
* docs(spec): 📝 fold the refinements found while planning into the spec
An abort method on the communicator, the truncation rank agreed through
a callback, output sinks, a logged warning for large in-memory inputs,
and MPICH in CI and in the dev shell.
Assisted-by: Pi:claude-opus-5-5
* docs(plan): 📝 implementation plan for the multi-GPU SRC sweep
Eleven tasks, each test-first, from the communicator to the scaling
runs. Written under the no-full-code rule of the design phase:
interfaces, pseudocode and test specifications.
Assisted-by: Pi:claude-opus-5-5
* docs(plan): 📝 align the multi-GPU spec and plan with main
Phase 1 is merged, logging is the standard library, the docs moved to
the Fumadocs site, and the CUDA 13 CuPy needs the cu13 NCCL wheel.
Assisted-by: Pi:claude-opus-5.5
* feat(utils): ✨ device selection and peer copies between GPUs
Assisted-by: Pi:claude-opus-5.5
* feat(group): ✨ one thread per device and the collectives between them
Allgather, gather and scatter over peer copies, ordered by events, in a
single process. A failing rank aborts the group so that no rank waits
forever; on the CPU the devices are simulated for testing.
Assisted-by: Pi:claude-opus-5.5
* feat(sweep): ✨ split the sweep among the GPUs of a node
Resources(devices=...) gives each device a contiguous block of the
sketch columns. The left-to-right pass needs no communication; per site
of the right-to-left pass the sketch is gathered on the first device for
the QR, the output core scattered by rows, and the projected environment
gathered on every device. Cores are read once and, with peer access,
uploaded in slices and gathered over the peer links. One device keeps
the previous sweep, bit for bit.
Assisted-by: Pi:claude-opus-5.5
* perf(bench): 📈 multi-GPU scaling command for the large benchmark
Assisted-by: Pi:claude-opus-5.5
* docs: 📝 running one sweep on several GPUs
Assisted-by: Pi:claude-opus-5.5
* perf(bench): 📈 multi-GPU tests and scaling on Leonardo (4xA100-64GB)
- Add `leonardo/run_multi.sh`: any command with 4 GPUs in one process, printing
the topology and peer-access matrix; `WARM_DIR` reads the stack beforehand.
- `scaling` also reports a QR distance over `a - b`, which resolves differences
below the sqrt(eps) floor of the inner-product distance.
- GPU tests pass (19/19); D_M = 1000 scaling with QR distances below 1e-12.
The reference size is queued; its results follow.
- Keep `benches/*/leonardo/logs/*.out` in git.
Assisted-by: GitHub Copilot:Claude Opus 5.5
* docs(bench): 📝 correct what the site stalls measure
The stall timer ends before the upload to the device, so it does not include the
split upload or the NVLink gather; it also counts the loader waiting for the GPU.
Assisted-by: GitHub Copilot:Claude Opus 5.5
* fix(sweep): 🐛 split a batch that runs out of GPU memory
At D_M = 4000, chi_out = 2000 on an A100-64GB, CuPy's pool fragments beyond the
margin the plan leaves for it: live memory peaked at 52 GB against the planned
56.7 GB, but the pool held up to 66.7 GB and a 12.6-14.7 GB allocation failed,
at a different point in each run. On out of memory, a kernel now releases the
pool's cached blocks and computes the batch in halves, joined along the batch
axis, keeping the split for the rest of the sweep. The plan and the batches the
store reads and writes are unchanged, so the devices need not agree on it.
Assisted-by: GitHub Copilot:Claude Opus 5.5
* perf(bench): 📈 reference size on 1 and 4 GPUs of Leonardo
At D_M = 4000, chi_out = 2000: 3925.3 s on one A100-64GB, 1099.9 s on four, a
speed-up of 3.57 and an efficiency of 89%. Reading environments back from host
memory and disk takes 82-84% of the right-to-left pass. The docs describe the
out-of-memory split. The 2-GPU point and the distance at this size are queued.
Assisted-by: GitHub Copilot:Claude Opus 5.5
* chore: 🔥 drop the superpowers planning docs
Phase 1 of the plan (#44's own description) called these dead weight.
Assisted-by: GitHub Copilot:Claude Opus 5.5
* fix(sites): 🐛 stage cores held on another GPU
A single-GPU sweep reused any CuPy core in place, so with
Resources(devices=[d]) a core on another GPU met environments and
sketches on GPU d. Reuse a core only when it sits on the sweep's GPU;
stage the others through the host buffer as a group does.
Addresses the Copilot review on #44.
Assisted-by: GitHub Copilot:Claude Opus 5.5
* perf(bench): 📈 2-GPU point at the reference size
Add the 2-GPU run and a second 4-GPU run (scaling --devices 4 2) at
D_M = 4000, chi_out = 2000: 2348.8 s on two GPUs (84% against one GPU),
1290.4 s on four, QR distance 7.5e-13 between them. Correct what the
environment stalls measure: they cover both passes and include waiting
for the GPU.
Assisted-by: GitHub Copilot:Claude Opus 5.5
* docs(bench): 📝 keep the best 4-GPU point at the reference size
Assisted-by: GitHub Copilot:Claude Opus 5.5
* docs(hpc): 📝 HPC clusters section; drop the Leonardo logs and scripts
Remove the benches/*/leonardo folders: the Slurm scripts and the run
logs, which stay in the git history. bench_mpo_mpo.py moves up to
benches/primitives/. The bench READMEs keep the result summaries,
without job IDs.
Add an "HPC clusters" section to the docs with a Leonardo page: the
CUDA 13 forward-compat setup, an example Slurm job script for one or
four GPUs, queues and performance notes.
Assisted-by: GitHub Copilot:Claude Opus 5.5
---------
Co-authored-by: Roberto Di Remigio Eikås <roberto@totaltrash.xyz>