The maintainer identified one LiveMathematicianBench item that belonged in the test split and moved 202512:46 on current main, so the checked-in item files now contain 35 train, 17 validation, and 125 test records. The accompanying split_manifest.json and released-splits table still advertise 35/18/124, leaving the repository's canonical metadata inconsistent with the corrected paper split. Because the hard scorer averages binary per-example results, using the corrected 125-item test file resolves the reported LiveMath denominator mismatch, but consumers following the stale metadata can still materialize or report the wrong split sizes. There are no competing or closed-unmerged cross-referenced PRs.

Summary

Update the LiveMathematicianBench manifest counts and the corresponding data/README.md released-splits row to 35/17/125, preserving the item move already present on main and avoiding unsupported claims about the other benchmarks' run aggregation. Add a focused tests/test_data_manifests.py consistency test that discovers each checked-in data/*/split_manifest.json, follows the canonical train/val/test layout, and compares every declared count with the length of its items.json array. This uses the manifest as the published contract and the checked-in item files as the implementation source of truth, covering the data-to-documentation wiring without changing the runtime scorer or benchmark loader.

Validation

• Happy path: every released split manifest, including LiveMathematicianBench at 35/17/125, declares counts equal to the lengths of its train, val, and test items.json arrays.
• Edge cases: manifests with different benchmark-specific item fields and large item arrays are validated solely by canonical split names and array length, without assuming a particular item schema.
• Error paths: a declared count that differs from its checked-in item array fails with the manifest path, split name, declared count, and actual count, reproducing the stale 18/124 LiveMath metadata failure.

Fixes #158