xulux evals for whether a coding agent can build a working assistant-ui app.

This README is the runbook. There is no separate doc. Use the scripts here; do not hand-write job YAML.

Pinned runner: Harbor 0.20 (required). Default sandbox is Blaxel (recommended). Docker works via --env docker. After a run, open report/index.html for the comparison grid.

Each case is a product prompt (build this assistant-ui app). Graders check source, build, runtime, browser behavior, and — for coding agents — whether they used docs / the create CLI.

Reward weights: AR-* (app run) and BR-* (browser) count 2; code/workflow checks count 1. Cascade skips count as fails.

check-env-codeverifiers is not an agent score. Gold overlays call npx assistant-ui@latest create. If that template has moved since a case was locked, this command can score below 1.0 even when the runner is fine — that is a gold refresh, not a script bug.

Linux, macOS, or WSL. Scripts are bash.

1. Node + npm — setup packs the published MCP docs server.

2. Harbor 0.20 — do not float to latest:

   uv tool install 'harbor[blaxel]==0.20'
   # or: pip install 'harbor[blaxel]==0.20'
   harbor --version   # must print 0.20.x

3. Blaxel (default sandbox) — bl CLI and a logged-in workspace, or BL_WORKSPACE + BL_API_KEY in .env.

4. Docker — only if you will pass --env docker.

5. Agent API keys in .env (see below).

git clone https://github.com/assistant-ui/coding-agent-evals.git
cd coding-agent-evals
cp .env.example .env
# edit .env — never commit it
./scripts/setup.sh

setup checks Harbor 0.20, copies .env.example if .env is missing, packs @assistant-ui/mcp-docs-server from npm into each task image, and probes Blaxel + Docker.

The MCP tarball is part of the image. Re-run setup when the published MCP package changes. A new tarball forces a new image build (slow the first time).

Always from the repo root, after setup.

# All cases. Recommended after clone.
./scripts/run.sh check-env-codeverifiers

# One case (faster smoke).
./scripts/run.sh check-env-codeverifiers --cases g1

Expect 1.0 on a healthy lock. If a case is below 1.0, stop and inspect that trial under jobs/ before running agents.

./scripts/run.sh full-eval

That is 11 cases × 3 agents × 3 surfaces = 99 trials, up to 10 at a time, on Blaxel. Budget hours and API spend. None / MCP / skills are three sequential jobs (sandbox env is per job).

custom requires --cases and --agents.

# One harness, MCP surface, two cases (low / cheap models).
./scripts/run.sh custom --cases g1,g2 --agents cursor-cli --surface mcp --mode low

# High-tier models, 30 sandboxes in flight.
./scripts/run.sh custom --cases g1 --agents cursor-cli,claude-code,codex --surface mcp --mode high --concurrent 30

# Three harnesses, one case, skills only (MCP off + Harbor Agent Skills).
./scripts/run.sh custom --cases g1 --agents cursor-cli,claude-code,codex --surface skills

# Local Docker instead of Blaxel.
./scripts/run.sh custom --cases g1 --agents cursor-cli --surface mcp --env docker

--env is not .env. Secrets stay in .env; --env picks the sandbox (Blaxel, Docker, or any other type the pinned runner accepts). Other backends (E2B, Modal, …) are pass-through: we do not first-class test them. Extra Harbor extras must already be installed.

Surfaces: none = web docs only (PB1_MCP=off); mcp = assistant-ui docs MCP; skills = MCP off plus Harbor Agent Skills from assistant-ui/skills on main (PB1_SURFACE=skills on both the sandbox and the grader). PB1_MODE is also set on both so ingest can keep low and high cells apart.

1. Open report/index.html in a browser (file:// is fine). Cells are score / agent time / API cost. Em dash means not run. Gold oracles are a verifier badge on the case, not a harness row. Failed checks show a plain-language title first; the id (BR-06, WF-S-01) is a tag. Prefixes: CQ source, AR app run, BR browser, WF-D docs/MCP/skills, WF-S create scaffold, WF-E errors during the agent run, WF-T agent started and tested the app.
2. Raw trials land in jobs/ (gitignored). Keep that folder; do not commit dumps.
3. Ingest merges the new job into report/ so the next clone still sees the last committed grid plus what you just ran.

The first image build on Blaxel can take a long time (Playwright + Chromium). Later runs reuse the image unless setup wrote a new MCP tarball. If sandbox create hits HTTP 504, wait several minutes before retrying (a timed-out create can leave a sandbox with the same name).

tasks/     G1–G11 cases (do not rewrite these to “fix” a run)
scripts/   setup, runner, ingest
report/    committed HTML + latest grid (the page the team opens)
jobs/      empty in git; local trial output
.env       your keys (gitignored)