| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
Empty expectedActions is scored as zero actions. Stop minting contrastive adjacent commands and refuse-then-alternate forms as empty-gold negatives; format_checker now hard-rejects them as BAD_NEGATIVE. Allowed negatives: pure refusal, non-action status/howto questions, and missing-info clarifications. Align synthesizer and quality-verifier prompts with the zero-action contract; add unit coverage including adversarial holes.
- Drop ACTION_VP / refusal regex classifier (unmaintainable verb lists). - Require semantic_checker negativeAssessments (kind + fairEmptyGold). - Code hard-fails unfair assessments as BAD_NEGATIVE; format stays structural. - Tests cover assessment parse/enforce and mock-LLM semantic gate.
- Single zod schema; derive OpenAI JSON schema via z.toJSONSchema - Match assessments by order (equal count); drop path-index maps - Simplify force-reject merge; strip explanatory comments
- Drop dead exports; parse via shared parseWithZod - Derive OpenAI schema from zod without void/\$schema clutter - Add test that assessments pair to negatives by order
- Join assessments to negatives by exact genCase path (bijective set); reject unknown/duplicate/missing paths as BAD_NEGATIVE instead of silent index pairing. - Parse reviewer decision before assessments so structured reject issues survive missing/invalid negativeAssessments. - Share TRANSLATION_BENCH_NEGATIVE_FAIRNESS_RULE in synthesizer context; tighten howto/soft-solicit unfair rules and untrusted payload framing. - Extend reviewerDecision mocks with path-keyed fair assessments; replace order-only path tests with multi-negative path-join coverage.
- Fixed short PATH_MSG/FIX only; use LLM reason as issue message - Drop rewrite-hint and per-case string assembly - Keep path 1:1 cover + fairEmptyGold/kind gate
- Honor concurrency with a worker pool and serialized checkpoint commits
- Keep parameters:{} after stripEmpty for required empty-object schemas
- Inject constant string-union fields (e.g. settings id) during gold validate
- Isolate per-slot generation failures so other workers keep committing
- Fair empty gold is pure_refusal/leave-alone only - Reject definition/status/Q&A and missing_info as empty gold - Synthesizer + quality-verifier prompts match any-tool abstain test - Extend unit coverage for definition Q and soft-solicit rejects
Join assessments to genCases via a path→case map built from the same keys the checker emits, then read dimensions.negativeKind directly.
…params - Allow incomplete case sets when requireCompleteCoverage is false - Prompt: nested objects must carry required schema fields (e.g. timeRange)
…ial gen - Partial-gen branch previously emitted planned caseCount/coverage, so validateTranslationBenchBenchmark always threw and the branch was unreachable dead code - Recompute scheduledActionCount, complete, and caseCount from the cases actually emitted; happy-path (complete) output is unchanged - Makes requireCompleteCoverage=false produce a valid draft directly
Derive FAIR_KINDS from TRANSLATION_BENCH_FAIR_EMPTY_GOLD_KINDS so the allowlist cannot drift from the exported constant.
There was a problem hiding this comment.
Adds a fairness contract ensuring empty-gold translation-bench negatives represent only pure refusals.
Changes:
Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file| File | Description |
|---|---|
| README.AUTOGEN.md | Refreshes generated documentation metadata. |
| actionValidation.ts | Normalizes gold actions for schema validation. |
| benchmark.ts | Uses normalized gold-action validation. |
| dataQualityVerifier.ts | Enforces semantic negative-fairness assessments. |
| datasetGenerator.ts | Adds concurrency and partial-generation handling. |
| generationCandidate.ts | Preserves required parameter objects during cleanup. |
| index.ts | Exports the fairness module. |
| negativeFairness.ts | Implements fairness parsing and enforcement. |
| quality-verifier.prompt.yaml | Defines full-catalog fairness review requirements. |
| synthesizer.prompt.yaml | Restricts empty-gold negatives to pure refusals. |
| utteranceDisambiguation.ts | Documents separate negative handling. |
| translationBench.datasetGenerator.spec.ts | Updates generation fixtures for fairness assessments. |
| translationBench.negativeFairness.spec.ts | Tests fairness validation and semantic enforcement. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Sorry, something went wrong.
…s, ambiguous-route guard Row-by-row 3-model review of the 1k eval showed most all-models "failures" were dataset/scoring fairness issues, not real misses. Address the generator-side ones: - Wire parameterScore specs into every generated case at finalize (`parameterScoreSpecsForExpectedActions`). Free-text echo params such as `originalRequest` and setting free-text now score `nonempty` instead of exact, matching the grader's own classification. Canonical payload hash excludes parameterScore, so dataset identity is preserved. - Add `HARDCODED_NON_EVAL_ACTION_IDS` (`chat.generateResponse`, `utility.claudeTask`) as a single source of truth, unioned into the packaged exclusion set so they are never targeted. - Drop cross-schema duplicate action names from targeting: when the same bare action name is owned by more than one schema (e.g. `deleteWebFlow`), the single gold route is ambiguous, so exclude every sibling. Tests: parameterScore wiring + canonical-hash stability, non-eval exclusion set, and ambiguous cross-schema guard. Full benchmarks suite green (138).
…eckpoint, param-spec cleanup Code-review + Copilot follow-ups on TB generation fairness: - negativeFairness: require dimensions.negativeKind to exactly equal the accepted empty-gold assessment kind. Previously a fair pure_refusal assessment was accepted even when negativeKind was missing/numeric/boolean, letting unlabeled empty-gold rows through. Now the label must match. - datasetGenerator: persist the checkpoint row BEFORE mutating casesBySlot / usedUtterances in commitAccepted, so an I/O failure can no longer leave an uncheckpointed case that the partial-coverage path would return. - Unify the duplicated param-field-mode union: grader now imports TranslationBenchParamFieldMode / TranslationBenchParameterScoreSpec from benchmark instead of re-declaring RunnerParamFieldMode; collapse the spec derivation to Object.fromEntries. - benchmark: enforce parameterScore aligns 1:1 with expectedActions in the probe payload schema (validateProbePayload). - Extract ambiguousCrossSchemaActionIds helper; drop the size===0 ternary. Tests: new integration coverage for generateTranslationBenchBenchmark (concurrent full run + partial run past a failed slot, asserting checkpoint contents and coverage); negativeKind-missing rejection; fixtures now label negatives pure_refusal. Full suite green (141).
- Seed 24 empirically-mined cross-schema collision pairs into KNOWN_CONFUSABLE_PAIRS (browser tab routes, code/utility file ops, visualStudio/code-debug, desktop/display settings, player queues, etc.) - Add generic cross-schema near-duplicate detector in findTranslationBenchConfusableSiblings: flags equivalent actions in different schemas when BOTH action-name token overlap >=0.5 AND description overlap >=0.34, so shared generic verbs alone do not over-flag unrelated actions - Add significantTokensFromText helper for description-token overlap - Extend unit tests with cross-schema newTextFile<->writeFile case plus a readFile negative control Root cause: synthesizer scheduled every catalog action as a gold target but only compared same-schema siblings for disambiguation, so cross-schema near-synonyms produced no disambiguation constraint and yielded ambiguous-gold cases the models unanimously routed elsewhere.
- Move catalog/grader generation into translationBench/policy - Add action-eligibility hard bans + LLM quality picker (model required) - Ship eligible-gold-actions.generated.json; fail-closed load/integrity - Schedule lattice + eval pin allowlist hash; copyAssets requires assets - Tests for picker, policy, nested llmAsAJudge, schedule allowlist-on
…ypeAgentQuestion from eligible actions
…into domnguyen/tb-negative-fairness
…-fairness # Conflicts: # ts/packages/benchmarks/package.json # ts/packages/benchmarks/src/translationBench/policy/policyGenerator.ts # ts/packages/benchmarks/src/translationBench/synthesizer/ambiguityProbe.ts # ts/packages/benchmarks/src/translationBench/synthesizer/benchmark.ts # ts/packages/benchmarks/src/translationBench/synthesizer/dataQualityVerifier.ts # ts/packages/benchmarks/src/translationBench/synthesizer/datasetGenerator.ts # ts/packages/benchmarks/src/translationBench/synthesizer/eligibleActions.ts # ts/packages/benchmarks/src/translationBench/synthesizer/quality-verifier.prompt.yaml # ts/packages/benchmarks/test/translationBench.checkpointPrimitives.spec.ts # ts/packages/benchmarks/test/translationBench.datasetGenerator.spec.ts # ts/packages/benchmarks/test/translationBench.policyGenerator.spec.ts
Adds a TypeScript command that verifies and analyzes the pinned DroidCall dataset from microsoft#2831. ```console $ cd ts && pnpm --filter @typeagent/benchmarks test Test Suites: 26 passed, 26 total Tests: 204 passed, 204 total Result: 204 tests passed, 0 failed ``` --------- Co-authored-by: typeagent-bot <typeagent-bot[bot]@users.noreply.github.com>
…-fairness # Conflicts: # ts/packages/benchmarks/src/translationBench/public_datasets/DroidCall/analyze.ts # ts/packages/benchmarks/src/translationBench/public_datasets/DroidCall/docs/droid-call-grader-improvement.md # ts/packages/benchmarks/src/translationBench/public_datasets/DroidCall/eval/officialDroidCallGrader.ts # ts/pnpm-lock.yaml
| String(value ?? "") | ||
| .replace(/\\/g, "\\textbackslash{}") | ||
| .replace(/([&%$#_{}])/g, "\\$1") |
# Conflicts: # ts/pnpm-lock.yaml
| Back | FazBrowse Home | New Git URL |
Summary
Adds a production translation benchmark pipeline:
Verification