| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
…pus, validated rule pack One unit: design docs, the consistency rule pack validated on the real engine, a 1,628-scenario benchmark corpus with per-scenario verification, the two-regime evaluation harness, CI enforcement, and the docs. - docs/internals/verified-completions/: RFC, review guide, extraction prompts, coverage audit, dataset critique, PLAN.md (issues #81-#95) - rules/consistency-core.iql: engine-validated; one clause per detection relation (works around #91); eq symmetrized so .why terminates; ontology seeds hardened by review (shareable identifiers removed, attr_domain seeded) - poc/corpus.py|corpus.json: corpus v2.1 - 16 families, 48 sub-variants all n>=33, multi-category labels, clean twins, extractor-truth facts; hostile-review hardened (DATASET-CRITIQUE.md); byte-reproducible - poc/verify_each.py + results/verification_ledger.json: per-scenario verification - structural axes 1,628/1,628, engine pass 1,526/1,526 with exact finding-kind match - poc/full_bench.py: two regimes, double grading with Opus arbiter, tiers, --resume, subscription-OAuth-first auth; completed v1 study (n=30/family, Wilson CIs) archived; v2.1 LLM re-baseline -> #95 - examples/iql/42_consistency_pack/: pack enforced by the e2e suite - scripts/run_snapshot_tests.sh: generated test config, rate limits off, credentials-file auth (#92) - docs/blog/consistency-ontology.md: beginner-facing ontology journey Verified on this tree: cargo fmt, clippy -D warnings, full cargo test suite, e2e snapshots 1,142 passed / 0 failed, IQL lint, corpus regeneration byte-identical, per-scenario ledger, secret scan.
| Back | FazBrowse Home | New Git URL |
The foundation of Verified Completions (tracking issue #89), shipped as one verified unit: design, engine-validated rule pack, executable benchmark with per-scenario verification, tests at every level, and the docs that explain it.
The proven core loop
Conversation text becomes atomic claims with quoted evidence, claims become facts in a per-conversation knowledge graph, and the consistency rule pack derives findings with proof trees. poc/poc_verify.py runs that loop end to end (extract -> validate -> load -> judge); the scorer and ingestion validator are negative-tested (fabricated quotes get dropped, impossible expectations fail rows).
The rule pack, validated on the real engine (#81)
The benchmark (corpus v2.1)
poc/corpus.py deterministically generates 1,628 scenarios across 16 families (~100 each, all 48 sub-variants at n>=33) covering value, negation, temporal, spatial, causal, structural, numeric, counting, identity, classification, and instruction corruption, plus 102 controls; 449 scenarios plant the conflict 12+ turns deep. Every sample carries multi-valued category labels, exact conflicting spans, a clean twin, a natural task, and extractor-truth facts. The corpus is documented for consumers in poc/DATASET.md (design principles, scenario anatomy, per-scenario validation, how to run and extend it) and regenerates byte for byte.
Per-scenario verification (poc/verify_each.py, ledger committed): spans/clean-twins/labels 1,628/1,628; engine pass 1,526/1,526 with EXACT finding-kind match - the engine fires precisely the expected kinds and nothing else, on every corrupted scenario, deterministically.
Completed behavioral study (corpus v1, 12 families at n=30, Wilson CIs, double-graded): corruption drops sound outputs from 95% to 87% [83-90] - 38 silent commits to arbitrary values, 4 contradictions written into deliverables; attaching the InputLayer finding restores 98% [96-99]. Corrupted system prompts are the standout: 17% sound alone -> 93% with the finding. Raw data archived in poc/results/archive_v1_full_bench_390.json. The v2.1 LLM re-baseline continues in #95 (resume-until-verified tooling is in this PR; 100/1,628 done).
Also in this unit
Verified on this exact tree
cargo fmt, clippy (-D warnings), full cargo test suite, e2e snapshots (1,142 passed / 0 failed), IQL lint on pack + fixtures, corpus regeneration byte-identical, per-scenario ledger, secret scan clean.
Engine bugs found and filed along the way: #91 (silent empty multi-clause unions), #92 (config/env-var conflicts), #93 (inserts ack while storing nothing after data-dir loss).
Closes #81.