TL;DR. In the released evalplus 0.3.1 (current PyPI latest), running untrusted_check on Windows with the default configuration returns ('timeout', []) for every program — correct and incorrect alike. The status is not pass, but the empty details array means an empty failing-index set, so any downstream pipeline that inspects per-test failures sees zero failures for all code: a silent false all-clear on return values. We hit this while validating a test-selection artifact and initially got zero findings on Windows for data that produces thousands of findings on Linux.

Mechanism (evalplus 0.3.1, from the published wheel)

1. eval/__init__.py:102-108 — query_maximum_memory_bytes() defaults to 4 GB (non-None unless EVALPLUS_MAX_MEMORY_BYTES=-1), so the memory-limit branch is always taken by default.
2. eval/utils.py:115-116 — reliability_guard does import resource whenever a memory limit is set. The resource module is Unix-only, so the child process raises ModuleNotFoundError on Windows (visible on the child's stderr).
3. eval/__init__.py:143 — the reliability_guard(...) call sits outside unsafe_execute's try: (which opens at :145), so the child dies with stat left at _UNKNOWN and progress at 0.
4. eval/__init__.py:271-275 — _mapping turns _UNKNOWN into None, details is truncated to progress.value (= 0), and if not stat: stat = TIMEOUT — hence ('timeout', []) regardless of the code under test.

Reproduction / control experiment

Self-contained repro (Windows, CPU only; python -m pip install "evalplus==0.3.1"):

# repro_evalplus_windows.py
import os
from evalplus.eval import untrusted_check

ENTRY_POINT = "add_numbers"
CORRECT = "def add_numbers(a, b):\n    return a + b\n"
WRONG   = "def add_numbers(a, b):\n    return a - b\n"   # deliberately wrong
INPUTS   = [[1, 2], [3, 4], [10, 20]]
EXPECTED = [3, 7, 30]
REF_TIME = [0.01, 0.01, 0.01]

def check(code):
    stat, details = untrusted_check(
        "mbpp", code, INPUTS, ENTRY_POINT, EXPECTED,
        atol=0, ref_time=REF_TIME, fast_check=False,
        min_time_limit=1.0, gt_time_limit_factor=4.0,
    )
    return stat, list(details)

if __name__ == "__main__":  # required: Windows spawn re-imports this module in every child
    os.environ.pop("EVALPLUS_MAX_MEMORY_BYTES", None)   # Case A: default config (memory limit on)
    print("[default config]               correct ->", check(CORRECT))
    print("[default config]               wrong   ->", check(WRONG))
    os.environ["EVALPLUS_MAX_MEMORY_BYTES"] = "-1"      # Case B: skip the import-resource path
    print("[EVALPLUS_MAX_MEMORY_BYTES=-1] correct ->", check(CORRECT))
    print("[EVALPLUS_MAX_MEMORY_BYTES=-1] wrong   ->", check(WRONG))

Observed output (Windows, Python 3.13, evalplus 0.3.1):

[default config]               correct -> ('timeout', [])
[default config]               wrong   -> ('timeout', [])
[EVALPLUS_MAX_MEMORY_BYTES=-1] correct -> ('fail', [0, 0, 0])
[EVALPLUS_MAX_MEMORY_BYTES=-1] wrong   -> ('fail', [0, 0, 0])

In the default runs each child process also prints ModuleNotFoundError: No module named 'resource' on stderr — expected; that is the crash at eval/__init__.py:143 → eval/utils.py:116, outside unsafe_execute's try. The discriminating signal in this control is the details array, not pass-vs-fail: with the memory limit on, details is empty (progress never advances past the crash); with EVALPLUS_MAX_MEMORY_BYTES=-1, details is fully populated. That the correct program still reports fail in case B is the separate unconditional signal.setitimer issue noted under "Relation to current master" — it does not affect the point demonstrated here (empty details are caused by the import resource crash, not by actual timeouts).

Relation to current master

Master already guards the import (if underlying_platform != "Windows": import resource, added in #262, May 2025), but that change has not been released to PyPI. Also, from code reading of master (not executed on Windows, please double-check): time_limit in eval/utils.py still calls signal.setitimer / signal.SIGALRM unconditionally, which do not exist on Windows — so each test would raise AttributeError inside the per-test except BaseException handler and be recorded as failed, i.e. the failure mode may have flipped from all-clear to all-fail rather than becoming correct.

Suggested resolutions (any of these would help)

• Cut a PyPI release including feat: allow unsafe Windows execution #262 so the released package stops silently all-clearing on Windows; and/or
• fail fast with an explicit RuntimeError("Windows is not supported for execution") instead of returning ('timeout', []); and/or
• document that result-producing runs require Linux/macOS.