EvalPlus version
0.3.1
Output of running ls ~/.cache/evalplus
92743def42b30b354a30898e4fa33fb0.pkl HumanEvalPlus-v0.1.10.jsonl MbppPlus-v0.2.0.jsonl (Windows cache dir, %LOCALAPPDATA%\evalplus\evalplus\Cache)
Task ID of the programming task
Mbpp/287
Test input
# plus_input index 3 of Mbpp/287 (MBPP+ v0.2.0); 16 more tests of the same shape
(1000000,)
Description
The task documentation states an integer-valued function:
Write a python function takes in an integer n and returns the sum of squares of first n even natural numbers.
assert square_Sum(2) == 20
The canonical solution computes 2 * n * (n + 1) * (2 * n + 1) / 3 with true division, so every stored expected value is a float. For 17 of the 106 plus tests (the n ≈ 10^6 region), that float is not equal to the mathematically exact sum — e.g. for n = 1000000 the stored expected value is 1.333335333334e+18 while the exact sum of squares is 1333335333334000000 (the stored float equals 1333335333334000128, i.e. 128 greater than the exact integer). Verified against MbppPlus.jsonl.gz v0.2.0 by re-executing the canonical solution: 17 of 106 plus tests have int(exact) != stored_float; largest |deviation| observed is 128.
Here is a correct solution but it is incorrectly falsified by the test because it returns the mathematically exact integer:
def square_Sum(n):
return 2 * n * (n + 1) * (2 * n + 1) // 3On the 17 affected tests the comparison path in evalplus/eval/__init__.py first fails out == exp (the exact int differs numerically from the imprecise float), then enters the float fallback where assert type(out) == type(exp) rejects int against float before np.allclose can apply. So an implementation matching the documented integer semantics fails these tests, while implementations reproducing the canonical's float rounding pass them. The stored oracle and the documented statement of the task disagree on those inputs; we are reporting the mismatch rather than proposing a specific remedy — though we note tools/mbpp/fix_v020.py shows precedent for repairing canonical solutions in place.
Other context:
We hit this while auditing false positives of base-vs-plus scoring for a paper (https://arxiv.org/abs/2607.11022). In the same audit we saw structurally similar tensions — canonical-derived expected values that penalize alternative correct implementations — on Mbpp/267 (same formula family, odd numbers) and Mbpp/806; happy to provide details on request, but this report is deliberately scoped to Mbpp/287 only.
Other context
No response
EvalPlus version
0.3.1
Output of running ls ~/.cache/evalplus
92743def42b30b354a30898e4fa33fb0.pkl HumanEvalPlus-v0.1.10.jsonl MbppPlus-v0.2.0.jsonl (Windows cache dir, %LOCALAPPDATA%\evalplus\evalplus\Cache)
Task ID of the programming task
Mbpp/287
Test input
Description
The task documentation states an integer-valued function:
The canonical solution computes 2 * n * (n + 1) * (2 * n + 1) / 3 with true division, so every stored expected value is a float. For 17 of the 106 plus tests (the n ≈ 10^6 region), that float is not equal to the mathematically exact sum — e.g. for n = 1000000 the stored expected value is 1.333335333334e+18 while the exact sum of squares is 1333335333334000000 (the stored float equals 1333335333334000128, i.e. 128 greater than the exact integer). Verified against MbppPlus.jsonl.gz v0.2.0 by re-executing the canonical solution: 17 of 106 plus tests have int(exact) != stored_float; largest |deviation| observed is 128.
Here is a correct solution but it is incorrectly falsified by the test because it returns the mathematically exact integer:
On the 17 affected tests the comparison path in evalplus/eval/__init__.py first fails out == exp (the exact int differs numerically from the imprecise float), then enters the float fallback where assert type(out) == type(exp) rejects int against float before np.allclose can apply. So an implementation matching the documented integer semantics fails these tests, while implementations reproducing the canonical's float rounding pass them. The stored oracle and the documented statement of the task disagree on those inputs; we are reporting the mismatch rather than proposing a specific remedy — though we note tools/mbpp/fix_v020.py shows precedent for repairing canonical solutions in place.
Other context:
We hit this while auditing false positives of base-vs-plus scoring for a paper (https://arxiv.org/abs/2607.11022). In the same audit we saw structurally similar tensions — canonical-derived expected values that penalize alternative correct implementations — on Mbpp/267 (same formula family, odd numbers) and Mbpp/806; happy to provide details on request, but this report is deliberately scoped to Mbpp/287 only.
Other context
No response