PaintBench evaluates generative image models on MS-Paint-style edits: recolor a region, draw a border, move a shape, complete a grid pattern. Every problem is a deterministic (input_image, instruction, answer_image) triplet generated from a seed, so the answer is pixel-exact and the answer distribution is known by construction — no human raters, no LLM judge, no ambiguity about what "correct" means.

Two benchmarks share the same file layout and evaluation pipeline.

PaintBench — 20 tasks across 4 capability categories, evaluated under 8 visual conditions × 12 problems each (1,920 scored problems), plus a 96-problem preservation/ diagnostic split that probes input-fidelity floor.

TinyGrafixBench — chart-edit analog of the primitive ops: 5 matplotlib chart families × 4 tasks × 30 seeds = 600 problems.

Per-task references with modes, visual conditions, and example instructions live in docs/paintbench-tasks.md and docs/tinygrafixbench-tasks.md.

make setup        # install core + dev deps (numpy, pillow, matplotlib, tqdm, ruff)
make generate-all # generate both benchmarks → benchmarks/
make help         # show every make target

On sandboxed shells (e.g. Cursor's integrated terminal on macOS) where POSIX semaphore creation is blocked, pass JOBS=1 to disable multiprocessing: make generate-all JOBS=1.

Requires Python ≥ 3.12 and uv. Install uv with:

curl -LsSf https://astral.sh/uv/install.sh | sh

Then install one of the dependency sets:

make setup            # core — generation, eval, stats, report, viz
make setup-inference  # + torch/diffusers/transformers (local GPU models)
make setup-api        # + google-genai / openai (Gemini, Nano Banana, GPT-Image)
make setup-data       # + datasets/huggingface-hub (read benchmarks from HF Hub)

For API-backed models, copy .env.example to .env and fill in your API keys (GEMINI_API_KEY, OPENAI_API_KEY, …) before running inference.

make generate-all                                    # 1. generate benchmarks → benchmarks/
make inference MODEL=flux2-dev BENCHMARK=PaintBench  # 2. run a model on a benchmark
#   ...repeat for each (model, benchmark) pair, or: make inference-slurm-h100
make eval     # 3. per-problem CIE76 stats → eval_outputs/problem_stats.jsonl
make stats    # 4. aggregate rollups       → eval_outputs/aggregate_stats.jsonl
make report   # 5. HTML report             → report.html
make viz      # 6. (optional) interactive web visualizer at :8765

On a SLURM cluster, make inference-slurm-h100 dispatches every (model × benchmark) pair in one shot. The driver (scripts/submit_inference_h100.sh) is vendor-neutral, env-var-driven, and runs uv run directly on the compute node. It supports typed --gres=gpu:<type>:N on modern clusters (h100 / h200 / a100 / v100 / l40s, auto-detected from SLURM_PARTITION when it matches a known GPU family). Requires SLURM_PARTITION + SLURM_ACCOUNT; optional knobs (SLURM_QOS, MODELS, BENCHMARKS, …) are documented in the script header.

docs/run_cycle.md has the raw uv run python commands behind each make target, for use inside a Slurm --wrap.

CLAUDE.md                     # orientation for AI coding agents
CONTRIBUTING.md               # dev setup, PR conventions, release process
CHANGELOG.md                  # release notes
pyproject.toml                # project metadata, deps, ruff config
Makefile                      # common operations (make help)
scripts/
  submit_inference_h100.sh    # Slurm submission (modern clusters: uv + typed --gres)
  gen_website_examples.py     # generate one (input, answer) pair per task for docs/web
docs/
  run_cycle.md                # raw `uv run python` commands behind each make target
  metric.md                   # CIE76 / IoU / edit-accuracy explainer + aggregation hierarchy
  extending.md                # recipes for adding models, tasks, chart families
  paintbench-tasks.md         # PaintBench task reference (4 categories x 5 tasks, modes, visual conditions)
  tinygrafixbench-tasks.md    # TinyGrafixBench task reference (5 charts x 4 tasks)
tests/                        # `make test` — determinism + registry + eval + smoke
src/
  generate_benchmark.py       # CLI: PaintBench / TinyGrafixBench
  inference.py                # Run models, save outputs, log hardware metrics
  eval.py                     # Pointwise CIE76 → per-problem stats
  stats.py                    # Aggregate problem_stats.jsonl → aggregate rollups (with 95% bootstrap CIs)
  report.py                   # HTML report from aggregate stats
  visualize.py                # Interactive web visualizer (generate / benchmark / eval tabs)
  benchmark_source.py         # LocalBenchmarkSource + HfBenchmarkSource adapters
  core/                       # Shared primitives
    background.py             #   Background generation (solid, striped)
    canvas.py                 #   Scene composition
    colors.py                 #   Color palette and sampling
    shapes.py                 #   Shape geometry and rendering
  tasks/                      # One module per PaintBench task + base.py
    base.py                   #   Shared utilities (Problem dataclass, etc.)
    translation.py            #   ... one file per task (20 tasks + base)
  tinygrafixbench/            # Chart-edit mini-benchmark
    bar_chart.py              #   4 tasks per chart type
    heatmap.py                #
    line_chart.py             #
    network.py                #
    scatter_plot.py           #
    utils.py                  #   Shared helpers (colors, fonts, themes)
    fonts/DejaVuSans.ttf      #   Bundled for reproducible rendering
benchmarks/                   # (gitignored) generated benchmark outputs
model_outputs/                # (gitignored) raw model outputs from inference
eval_outputs/                 # (gitignored) derived eval stats + diff image cache
data/                         # (gitignored) scratch space
logs/                         # (gitignored) Slurm logs

20 tasks organised into 4 capability categories:

Each task is evaluated across 8 visual conditions × 12 problems (1,920 scored problems total), plus a 96-problem preservation/ diagnostic split that probes input-fidelity floor.

See docs/paintbench-tasks.md for the full per-task reference (modes, visual conditions, parameter grids) and docs/tinygrafixbench-tasks.md for the TinyGrafixBench equivalent.

See docs/extending.md for full recipes covering:

• Adding a new image-edit model to the registry
• Adding a new PaintBench task
• Adding a new TinyGrafixBench chart family
• Building a sibling benchmark on top of the same eval pipeline

For PaintBench tasks specifically, the surface is small: every task in src/tasks/<name>.py exports NAME, PARAMETERS, and generate(...) -> Problem, then gets registered in the TASKS list at the top of src/generate_benchmark.py. All randomness must go through a seeded random.Random (no global random) — the determinism contract is enforced by tests/test_determinism.py. See CLAUDE.md for the architectural conventions.

The full TinyGrafixBench task reference lives in docs/tinygrafixbench-tasks.md.

@article{paintbench2026,
  title   = {{PaintBench}: Deterministic Evaluation of Precise Visual Editing},
  author  = {Xu, Kai and Brown, Ellis and Madhu, Shrikar and
             Fergus, Rob and He, He and Xie, Saining},
  journal = {arXiv preprint arXiv:<ARXIV-ID>},
  year    = {2026}
}