This is a coverage question, not a bug report. While auditing reward-scoring false positives we ran into a discriminability property of Mbpp/284 ("checks whether all items in the list are equal to the given element", check_element) that we would like the maintainers' view on.

Observed phenomenon

An identity-based implementation that violates the documented "equal" semantics:

def check_element(list1, element):
    return all(v is element for v in list1)

behaves differently depending on how the test inputs are constructed (observed on CPython 3.13, evalplus 0.3.1, MBPP+ v0.2.0):

• Through evalplus's own path (inputs deserialized from MbppPlus.jsonl), the implementation is caught — but by exactly one test out of 106: base_input[2] = (['green', 'green', 'green', 'green'], 'green'), where the JSON-loaded equal strings are distinct objects, so is returns False against expected True. None of the 103 plus tests discriminate: every non-empty all-equal plus case uses single-character strings (['a', 'a', ...], 'a'), which CPython interns even when JSON-loaded, so is passes them; the remaining expected-True plus cases have empty lists, where all() is vacuously True for both implementations.
• Through a source-literal path (executing the documentation's assert as code, as the original-format MBPP tests and many RL reward harnesses do), CPython constant-folding makes the four 'green' literals one object, so the identity implementation passes everything — we first noticed this when the same program passed our doc-assert-style runner and then failed when the identical test-3 values were fed via json.loads.

Minimal repro:

import json
def check_element(list1, element):
    return all(v is element for v in list1)  # identity, violates "equal"
lst, el = json.loads('[["green", "green", "green", "green"], "green"]')
print(check_element(lst, el))                                   # False (json path)
print(check_element(["green", "green", "green", "green"], "green"))  # True (literal path)

The question

Whether Mbpp/284 distinguishes == from is currently rests on a single base test and on CPython string-interning behavior rather than on the task's semantics. Would you consider adding one or two plus cases that discriminate the two robustly regardless of construction path — e.g. repeated equal mutable elements such as ([[1], [1]], [1]), which are distinct-but-equal objects on both the JSON path and the source-literal path? (Immutable literals — multi-character strings, large ints, tuples — are merged into one object by CPython's compile-time constant folding on the literal path, so only mutable elements discriminate on both.) Or is this considered out of scope for the plus suite (in which case, is the reliance on base_input[2] as the sole discriminator intentional)?

Context: we hit this during a test-suite false-positive audit (https://arxiv.org/abs/2607.11022); happy to share the enumeration script.

Distinct from #306: that PR defends against a program returning an object whose __eq__ is overridden to always pass (an adversarial return value). The case here is the opposite direction — a plausible is-based implementation that is falsified on some inputs and passes on others purely because of CPython interning/constant-folding of the test inputs, so the discrimination depends on how the inputs are constructed rather than on the implementation being wrong. A type() guard on the output does not address it (output is a bool either way); it is a question about the plus inputs' discriminating power, not about output-type deception.