Summary

Runner-level eval batching is now hard-deprecated instead of silently falling back to per-case dispatch. Provider config rejects batch_requests, provider_batching, and providerBatching with guidance to keep queueing or batching inside provider adapters behind normal per-request invoke calls.

The eval orchestrator no longer has a runBatchEvaluation branch, and the core Provider interface no longer exposes runner-selected supportsBatch / invokeBatch capability hooks. Existing concrete provider invokeBatch helper methods remain only as internal/direct helper behavior for surfaces such as script-grader provider clients; they are not reachable from eval-runner dispatch.

Related: av-di3a

Validation

• bun test packages/core/test/evaluation/providers/targets.test.ts packages/core/test/evaluation/validation/targets-validator.test.ts packages/core/test/evaluation/orchestrator.test.ts packages/core/test/evaluation/providers/pi-coding-agent.test.ts
• bun test apps/cli/test/commands/eval/task-bundle.test.ts
• bun --filter @agentv/core build
• bun --filter @agentv/core typecheck
• bun --filter @agentv/sdk build
• bun --filter agentv typecheck
• bun --filter agentv build
• bunx biome check <touched files>
• git diff --check
• bun test packages/core/test/evaluation/loaders/config-loader.test.ts packages/core/test/evaluation/config.test.ts

agentv CLI build exited 0. It printed existing bundler warnings from Zod declaration imports and skipped Dashboard static copy because apps/dashboard/dist was absent.

Code review: skipped because the available subagent tool policy forbids spawning unless the user explicitly asked for delegation; performed a manual diff scan instead.

Evidence

Live dogfood passed with a real PI coding-agent candidate and Azure LLM grader:

bun apps/cli/src/cli.ts eval run \
  .agentv/results/av-di3a-dogfood/inputs/per-case.eval.yaml \
  --providers .agentv/results/av-di3a-dogfood/inputs/providers.yaml \
  --provider pi-sdk-azure \
  --grader-provider azure-grader \
  --workers 1 \
  --output .agentv/results/av-di3a-dogfood/run-azure

The run used evaluate_options.repeat: 2 and produced the expected aggregate repeated case with sample-1 and sample-2 artifacts, both passing.

Private evidence branch: EntityProcess/agentv-private:evidence/av-di3a-hard-deprecate-batch at commit 32995f3.

Post-Deploy Monitoring & Validation

• Watch CI for packages/core/test/evaluation/orchestrator.test.ts, provider validation tests, and CLI eval task-bundle tests.
• Run or inspect a config-validation smoke with each removed field spelling; healthy signal is one hard error containing Runner-level batching was removed.
• Run a small live repeat eval after merge when credentials are available; healthy signal is per-sample artifacts under sample-N/ and no batch-mode fallback warnings.
• Failure signal: any eval run invoking provider batch dispatch from orchestrator, accepting removed batching fields, or emitting legacy batch fallback warnings. Roll back this PR or restore per-case dispatch by reverting the merge commit.
• Validation window: first CI run and first post-merge live dogfood. Owner: AgentV maintainers.