English | 中文

A test and benchmark matrix runner for TensorSharp inference. It sweeps the cartesian product of (model × backend × feature × env-var) by invoking TensorSharp.Cli as a subprocess, parses per-run metrics from its structured log output, persists one JSON file per cell, and emits an aggregate markdown report.

The companion document docs/env_var_feature_matrix.md is the curated reference for which env vars get swept and what features they affect — keep them in sync when you add a new flag.

Building requires the .NET 10 SDK, not only the runtime. The linked guide covers installation on Windows, macOS, and Linux and SDK verification on a new machine.

dotnet build TensorSharp.TestMatrix/TensorSharp.TestMatrix.csproj -c Release
dotnet build TensorSharp.Cli/TensorSharp.Cli.csproj -c Release   # required: subprocess driver target

The runner needs a built TensorSharp.Cli binary; it auto-discovers it at ../TensorSharp.Cli/bin/TensorSharp.Cli relative to its own bindir. Override with --cli-executable <path> or cli_executable in the config.

Default run (everything the host supports, all curated env-var sweeps):

dotnet run --project TensorSharp.TestMatrix -c Release

Each applicable cell also gets a baseline run with no forced env var. Sweep cases pass exactly one env var/value pair to the subprocess after inherited TS_*/related variables are scrubbed.

Curated subset — useful for an interactive dev cycle:

dotnet run --project TensorSharp.TestMatrix -c Release -- \
  --backends ggml_metal,mlx \
  --features short_text,long_text,image \
  --env-vars KV_CACHE_DTYPE,TS_QWEN35_BATCHED \
  --model-dir /Users/ZhongkaiFu/work/model \
  --results-dir results/dev \
  --report report-dev.md

Dry-run (print every case id it would execute):

dotnet run --project TensorSharp.TestMatrix -c Release -- --dry-run

Disable env-var sweeps and run only baseline cells:

dotnet run --project TensorSharp.TestMatrix -c Release -- --env-vars none

Resume an interrupted run without re-executing cells already on disk:

dotnet run --project TensorSharp.TestMatrix -c Release -- --skip-existing

A JSON config file controls model discovery, defaults, and per-model overrides. The default lives at Defaults/matrix-config.json and is copied next to the built assembly. Override fields on the command line — those win over the file.

Per-model overrides look like this:

{
    "models": [
        {
            "id": "gemma4-e4b-q8",
            "family": "gemma4",
            "display_name": "Gemma 4 E4B Q8_0",
            "gguf": "gemma-4-E4B-it-Q8_0.gguf",
            "mmproj": "mmproj-gemma-4-E4B-it-Q8_0.gguf",
            "supports_image": true,
            "supports_audio": true,
            "supports_video": true,
            "supports_tools": true,
            "supports_thinking": true,
            "enabled": true
        }
    ]
}

Relative paths are resolved against model_dir. Auto-discovered models that share an id with a config entry are replaced by the config entry.

DiffusionGemma GGUFs should be added through an explicit models[] entry until ModelDiscovery grows architecture-aware detection for diffusion-gemma / diffusion_gemma. The current feature catalog is still append/decode oriented, so add a dedicated diffusion feature before using TestMatrix output as a formal diffusion regression signal.

The Inputs tree ships only the prompts, not the media — the fixtures are megabytes of binary that would bloat every clone. Generate them instead:

eng/make-test-media.sh ~/work/models/testmedia
dotnet run --project TensorSharp.TestMatrix -c Release -- --media-dir ~/work/models/testmedia

…or set media_dir in the config to point at that directory. The script emits exactly the three files the catalog expects, and their content is what the default ExpectedContains keywords assert:

Video generation needs ffmpeg on PATH (the OpenCV build vendored here decodes video but cannot encode it); audio needs macOS say+afconvert or espeak+ffmpeg. Anything missing is reported and skipped by the script, and the corresponding matrix cells then fail with a clear "media file not found" — they do not silently pass.

• results/<case-id>.json — one file per cell with full metrics + stdout tail + correctness verdict
• report.md — aggregated markdown report (baseline tables, env-var sweep tables, sensitivity summary, skipped combinations, correctness failures, runtime failures)

The report is the artifact uploaded in CI; the JSON files are the source of truth for delta analysis against previous runs.

A cell is marked ok = true only when all of these hold:

1. TensorSharp.Cli exited with code 0 (no crash, no timeout).
2. At least one of prefill_tps, decode_tps, or model_load_ms was parsed from the structured log output (proves the model loaded and ran).
3. Correctness check — if the feature configures ExpectedContains, every required substring must appear in the assistant's output (case-insensitive). Failures are reported in a dedicated "Correctness failures" section separate from runtime failures.

Per-feature expected content lives in Matrix/FeatureCatalog.cs:

The check is intentionally weak (a single keyword is a necessary, not sufficient, condition for a correct answer). It catches catastrophic regressions — model loaded but generated gibberish, multi-turn KV reuse silently broken, tool name not emitted — without trying to be a model-quality benchmark. For richer verification, override ExpectedContains on a feature or add a regression-mode comparison against recorded golden outputs.

When you change the default media file (image, audio, video), update the corresponding feature's ExpectedContains to match.

The matrix is only as useful as the ability to spot regressions. The runner loads a per-host baseline file from Baselines/baseline-<host>.json and compares the current run against it:

• New runtime failure — was passing, now crashes / times out / no metrics
• New correctness failure — was passing, now misses an ExpectedContains substring
• Throughput regression — decode TPS dropped more than --regression-threshold-pct (default 10%)
• Improvement — was failing, now passing (informational)
• Untracked — no baseline entry yet (new cell)

Runtime / correctness / throughput regressions are blocking: with --fail-on-regression set, the runner exits non-zero and the PR fails. Improvements and untracked cells are informational.

# Compare against the committed baseline, fail PR on blocking regressions
dotnet TensorSharp.TestMatrix/bin/TensorSharp.TestMatrix.dll --fail-on-regression

# Re-bless the baseline after a legitimate change (commit the diff yourself)
dotnet TensorSharp.TestMatrix/bin/TensorSharp.TestMatrix.dll --update-baseline

# Compare against a specific baseline file (A/B against an older snapshot)
dotnet TensorSharp.TestMatrix/bin/TensorSharp.TestMatrix.dll \
  --baseline ./oldbaseline.json --fail-on-regression

See Baselines/README.md for the file format, host-class layout, and update workflow.

The repo's GitHub Actions workflow test-matrix.yml uses self-hosted runners labelled tensorsharp-cuda and tensorsharp-mlx.

To keep CI latency low, automatic triggers (every PR and every push to main) run a fast "smoke" profile, not the full matrix:

The smoke profile still catches the things that matter most on a per-commit basis — a throughput regression (pp512/tg128) or a model that loads but answers gibberish (short_text) — while dropping the env-var sweeps, which are the dominant cost.

Running the full matrix: it is no longer run automatically. Trigger it on demand from the Actions tab via workflow_dispatch (leave the backends / features / env_vars inputs blank for the complete matrix, or set them for a custom subset), or run any subset locally — see Running above. The PR comment still includes a link to the uploaded report.md artifact. See the workflow file for runner setup expectations (model directory, NVIDIA drivers, etc.).