generative-ui-bench

How reliably do today's models generate working UI?

One catalog of 70 components, 46 screen briefs, four attempts each, and three generative-UI formats judged by their own SDKs: OpenUI Lang, Google A2UI (v0.9), and Vercel json-render (0.19).

Benchmark · OpenUI · Docs

Every raw model output and every scored verdict is committed here. Anyone can rescore the data offline and diff against the published results, or add a new model with one command.

• 46 screen briefs in 5 size bands (2 to 18 numbered requirements). None names a component or layout.
• A 70-component catalog derived from OpenUI's public component library. catalog/public-catalog.json is the reference surface; node tools/check-catalogs.ts verifies the three protocol catalogs stay equivalent to it.
• One uniform condition: 4 generations per brief, temperature 0.7, reasoning minimal/none, 16,384-token output ceiling.
• Each format's prompt comes from its own SDK's generator, all carrying the same two worked examples. Each format's validation is its own SDK's shipped code plus one shared completeness layer, identical for all three: a run is complete when it parses, renders a root, every reference resolves, every component is reachable from root, and required and enum-typed props check out. A coverage floor requires at least as many components as the brief has numbered requirements.

The headline board (six models, one seat per company, all three formats) is on the benchmark page. OpenUI-only runs beyond it, same condition:

The four qwen27b* labels are one model at four reasoning efforts. All committed results are scored under lang-core 0.2.16.

Node >= 22.18; the harness runs TypeScript directly. The openui scorer is @openuidev/lang-core pinned to exactly 0.2.16; the pin is part of the published condition.

npm install

# A2UI's scorer needs the official python SDK at the pinned revision:
python3 -m venv .venv
.venv/bin/pip install antlr4-tools
.venv/bin/pip install "a2ui-agent-sdk @ git+https://github.com/a2ui-project/a2ui@29b715fa89fc5bb8351d2ea0116f03d4f2e212f2#subdirectory=agent_sdks/python/a2ui_agent"

A2UI_PYTHON=.venv/bin/python node score.ts           # all models
A2UI_PYTHON=.venv/bin/python node score.ts gemini    # one model

score.ts rewrites results/results-<model>.json from the raws alone, so a diff against the committed results is the integrity check. Without A2UI_PYTHON it scores openui and json-render and leaves results files untouched. raw/<label>/truncated.json records generations that hit the output ceiling. Token and cost tables: node tools/count-tokens.ts and node tools/cost-estimate.ts.

BENCH_MODEL=google/gemini-3.6-flash BENCH_LABEL=gemini \
OPENROUTER_API_KEY=... node run.ts openui jsonrender a2ui

BENCH_PROVIDER selects openrouter (default), openai, anthropic, google, or local. Raws are idempotent, so an interrupted run resumes by re-running the same command. See the header of run.ts for every knob. Then node score.ts <label>. New briefs follow briefs/DESIGN.md; a new protocol is one folder under protocols/ exposing a system prompt and an evaluate(text, {reqs}) verdict.

• Empty responses score as blanks. Failed API calls are retried at generation time and never scored as model failures.
• The method builds on Mobile Reality's MDMA benchmark. SDK versions are pinned by the committed package-lock.json and the python install command above. The files under raw/ are verbatim model outputs, published as the benchmark's data record.