Measures whether composing from the Software Periodic Table changes what an agent produces, relative to unconstrained generation.

Read docs/EVAL_RESULTS.md first. It explains why the previously published token-savings figure was withdrawn, why the replacement metric was wrong too, and which metrics are worth trusting.

npm run eval                    # self-check, no API key needed
OPENAI_API_KEY=sk-... npm run eval   # real baseline vs composition

With no key, the harness replays each scenario's own ground-truth plan. That is a check of the metrics and the ontology, not a measurement of model behaviour: every plan is within the table by construction, so recall is 1.0 and acceptance passes by definition. The output says so on every run.

Six scenarios in scenarios.ts, each with a feature request, a ground-truth atom set, and acceptance criteria that name the atoms satisfying them.

test/metrics.test.ts enforces that the criteria and the ground truth stay consistent: every criterion's symbols must exist, every criterion must be satisfiable by the ground truth, and every ground-truth atom must be required by some criterion.

Both arms get the same model, the same feature request, and are asked for the same two artefacts: a component breakdown and the code that realises it. The only difference is what they are told.

The baseline's names are then resolved to symbols within their own family, so a baseline component that happens to coincide with an ontology atom earns the credit. Without the family qualifier the lookup is ambiguous — Email is both an object and a property — and components get silently reassigned to whichever family was declared last.

An earlier version reported totalTokens alongside a name-normalized figure. Both were character counts, and a real tokenizer put the first at −21% and the second at +9% on this repository's own plans — opposite-signed errors, so no constant factor corrects the ratio. The notation advantage it implied (1.36×) measures as 0.99×, which is to say nothing.

test/metrics.test.ts asserts that no export whose name suggests counting tokens or characters can come back. If a cost comparison is ever wanted it needs a real tokenizer on both arms, and a decision about whether notation length is the right question at all.

The baseline was never shown the table, so it cannot stay inside it. Reporting that as a baseline failure measures compliance with an instruction only one arm received.

npx tsx eval/agent-eval.ts baseline      # print the baseline agent prompt
npx tsx eval/agent-eval.ts composition   # print the composition agent prompt
npx tsx eval/agent-eval.ts compare       # score both saved result files

Each agent writes baseline-results.json or composition-results.json at the repository root; override with SPT_BASELINE_OUT / SPT_COMPOSITION_OUT. Both are gitignored. Useful for comparing different agents, or for running an experiment that should not cost API calls.