English | 中文

Native .NET LLM inference engine for GGUF models — autoregressive LLMs and DiffusionGemma-style text-diffusion, plus Qwen-Image-Edit image editing and MiniMax-H3 video with native 32 kHz stereo audio (and Wan 2.1/2.2 for video alone). Ships a console app, a browser chat UI, and Ollama/OpenAI-compatible HTTP APIs. A pure-.NET engine that trades wins with the hand-tuned C++ llama.cpp on identical GGUF files and the same GPU.

From Tensors to Tokens: Building a Multimodal LLM Inference Engine from Scratch with TensorSharp and Gemma 4 E4B by Zhongkai Fu turns this repository into a guided, end-to-end learning journey. It uses Gemma 4 E4B to connect tensor fundamentals, model execution, multimodal inputs, and the application surfaces of a working LLM inference engine.

Explore the book and its repository reading path · Buy the paperback on Amazon

• ⚡ Trades wins with llama.cpp — from pure .NET. On identical GGUF files and the same GPU, TensorSharp matches or beats llama.cpp on the workloads that matter: Gemma 4 E4B and 2-bit Qwen 3.6 35B-A3B MoE prefill 1.28× faster on CUDA with first tokens 1.27× sooner (multi-turn up to 1.49×); Gemma 4 12B decodes 1.21× faster on Vulkan (up to 1.32× on long context). → Benchmarks
• 🚀 Continuous batching & paged KV cache. vLLM-style paged KV pool with block-hash prefix sharing and an iteration-level scheduler, on by default in the server. → deep dive
• 🧬 DeepSeek V4 Flash (284B MoE) with three whole-model executors. The compressed-sparse-attention 1M-context architecture runs on a direct-CUDA engine (--backend cuda), the native ggml executor (--backend ggml_cuda / ggml_vulkan), and a 100% pure-C# CPU executor (--backend cpu, no native dependencies). Weights layer-split automatically across every visible GPU, so a model far larger than one card still runs; the server hosts it with per-sequence slots and continuous batching. → DeepSeek V4 card
• 🧠 GLM-5.2 (744B-A40B MoE) with tensor parallelism and CPU MoE offload. Multi-head Latent Attention plus a DeepSeek Sparse Attention "lightning indexer" that picks which 2048 cached tokens each query may attend to. --tp N runs every layer on every GPU (heads column/row-parallel, every expert split row-wise) and --cpu-moe keeps the routed experts — 92% of the checkpoint — in system RAM. The default layer split and --cpu-moe reproduce llama.cpp token-for-token on the same backend; --tp sums per-rank partials, so on a 2-bit MoE its last-bit difference reaches the top-8 router and the near-tied tokens can differ. Head-to-head on 3x RTX PRO 6000: pp2048 918.9 vs llama.cpp's 763.1 tok/s, tg64 43.7 vs 42.2. The advertised 1M context (~93 GiB of KV) is a ceiling rather than a promise — once the weights land the loader sizes the context to the VRAM actually free and logs its pick (342,272 tokens on the layer split, 646,400 with --n-cpu-moe 30); MAX_CONTEXT makes a specific length a hard requirement instead. → GLM card
• 🧠 GLM-5.3-Flash (320B MoE) decodes 2.0× llama.cpp. The hybrid successor — 288 routed experts, KDA linear attention on 34 of 45 trunk layers, NoPE MLA + DSA on the other 11 with a pooled indexer (top-k 2048 over 4-cell pools, then expanded to their members), Sinkhorn hyper-connections ×4 — loads through the same native executor and the same GlmDsaModel as GLM-5.2. On 2× RTX PRO 6000 Blackwell (96 GB), GLM-5.3-Flash-UD-Q2_K_XL (101 GiB), layer split, both engines at n_ubatch 2048 and back to back: tg64 73.5 vs llama.cpp's 36.6 tok/s, with prefill within a few percent either way (pp2048 2014 vs 2070, pp16384 1692 vs 1690, pp32768 1446 vs 1483). Vision through mmproj-BF16.gguf (the GLM-OCR ViT): --image, multi-image and multi-turn image sessions. The layer split across every visible GPU, --cpu-moe / --n-cpu-moe and per-sequence native slots all work; --tp tensor parallelism is cleanly refused (use the layer split) and NextN/MTP speculation is not implemented yet. → GLM card
• ⚡ Qwen 3.8 Flash Next — the whole token as one graph. A hybrid MoE: GatedDeltaNet recurrent layers on 36 of 48 layers interleaved with full-attention layers (some behind Qwen Sparse Attention's indexer), a PLE n-gram embedding block, ×4 hyper-connection streams and 512 experts with 10 used. Embedding, in-graph PLE, all 48 layers, the final mixer and the LM head run as (almost) one captured graph per token, from a shape-keyed cache; vision rides the Qwen3.5-VL tower with (T,H,W) IMRoPE, including multi-image and multi-turn image sessions with KV reuse across turns (extend-only — the GDN recurrence cannot rewind). --tp N runs a layer split here, not tensor parallelism: on 2× A100-80GB, Qwen3.8-Flash-Next-UD-Q2_K_XL (73.4 GiB) lands 24.2 GB + 26.2 GB across the two cards with prefill ~1520–1550 t/s and decode ~56 tok/s either way, and greedy output byte-identical to the 1-GPU run — capacity, not speed. → Qwen 3.8 Flash Next card
• 🔮 Speculative decoding — four algorithms over one draft-verify runtime. Multi-token-prediction draft heads accelerate solo decode on Qwen 3.6 (NextN block embedded in the trunk — use an MTP-retaining GGUF such as unsloth/Qwen3.6-35B-A3B-MTP-GGUF; the base repo ships the same file names with the block stripped), GLM 5.2 (its NextN block ships in the stock checkpoint — ~1.3× decode, 94% draft acceptance, on 2× RTX PRO 6000 with --n-cpu-moe 20) and Gemma 4 (separate gemma4-assistant draft GGUF, --spec-draft-model); DeepSeek V4 adds DSpark block drafting (--draft-model), which proposes a whole block of tokens per step for 1.3–1.4× decode (up to 2.0× on multi-turn chat). A fourth algorithm needs no trained weights at all: --spec-type ngram matches the sequence's own suffix against the tokens it has already seen, works on every checkpoint, and measured 45.2 tok/s against 31.4 plain (1.44×) on Qwen3.5-9B (Q8_0, ggml_metal, M5 Pro) — a model that ships no draft head — with byte-identical output. In every case the draft proposes, the trunk verifies in one batched forward, and the output matches standard decode. Off by default; opt in with --spec on either host (the historical --mtp-* spellings are accepted as aliases). → Speculative decoding
• 🔗 Tensor parallelism & distributed clustering. Split a model across multiple GPUs with --tp N — on the direct cuda backend and on GGML CUDA / Vulkan — and extend across machines with peer-to-peer TCP clustering (--tp-node-id / --tp-peers). Megatron-LM column/row-parallel pattern with hierarchical AllReduce; MoE expert parallelism and per-rank GatedDeltaNet kernels on GGML. Fused per-rank execution makes --tp 2 decode 1.39× a single GPU on Gemma 4 E4B and 1.57× on Muse-Glimmer 30B (which also gains 1.34× prefill — the one model that beats a single GPU on both phases), and runs models that do not fit one card at all (Qwen 3.5-35B-A3B; Muse-Glimmer 30B Q8_0 at 28.2 GB on 24 GB cards). Architectures that shard no weights take the same --tp N as a layer split instead — each GPU holds a contiguous run of whole layers (Qwen 3.8 Flash Next; DeepSeek V4 and GLM 5.x layer-split by default) — which buys capacity rather than speed, and an architecture that supports neither now says so on stderr and runs on one GPU instead of silently leaving the others idle. Optional Redis-backed KV cache and Responses API store. → Tensor Parallelism
• 🎨 Qwen-Image-Edit image editing. Prompt + input image → edited image, driving a 60-block MMDiT with a Qwen-Image VAE and Qwen2.5-VL-7B text encoder. CUDA-graph-captured DiT, FlowMatch-Euler true-CFG denoise, live Web UI previews, and a Lightning distillation LoRA fast path (--qwen-image-lora, applied as a runtime side-path over the untouched quantized weights) that takes the default 30 steps × CFG — 60 DiT forwards — down to 4. Beat stable-diffusion.cpp 1.19× on a warm 4-step edit. → Qwen-Image-Edit card
• 🎬🔊 MiniMax-H3 joint audio-video generation. Prompt → video with a native 32 kHz stereo soundtrack, generated together rather than dubbed on: one 19.3B diffusion transformer denoises a packed video+audio latent in a single token sequence, up to 15 s at 24 fps. Text-to-video, image-to-video (the photo becomes the first frame and the prompt drives the motion), first/last-frame morphing, and reference-to-video on the separate Ref2VA checkpoint — up to nine references in any mix of stills (--ref-image), clips (--ref-video, with --ref-video-audio for a clip's own soundtrack) and standalone audio (--ref-audio), each taking its own stretch of the shared timeline before the generated clip, so the person or product carries over while camera, background and composition come entirely from the prompt. All of it CFG-free at 4–8 steps against a 20-step default. Seven native ggml graphs — a 50-layer Qwen3-VL-32B text encoder with its 27-block vision tower and DeepStack taps, the packed-latent DiT with its learned AdaLN curve table and 3-axis float RoPE, a pure-transformer video VAE (36 blocks, no deconvolutions), and an alias-free BigVGAN audio VAE. Frame counts snap to a 17k+5 grid (5, 22, 39, 56, 73, 90 …) and any grid length decodes correctly — the video VAE runs 5 latent frames at a time with a 2-frame look-ahead and cross-fades the seams, while h3_attend pre-scales V by a power of two derived from the key count so that a long clip's unmasked bidirectional attention (8646 packed tokens at 107 frames, against 2364 at 22) stays finite in ggml's FP16 flash-attention accumulator; before that fix a 107-frame clip came back with every pixel black and the audio clamped. Runs 2.4× faster end-to-end than stable-diffusion.cpp at 256×256 and 1.7× at 640×384 on an M5 Pro (ggml_metal); on a 16 GB RTX 3080 Laptop (ggml_cuda) stable-diffusion.cpp takes the end-to-end win instead — 1.15× at 256×256, 1.07× at 640×384 — while TensorSharp stays ahead per denoise step (3.325 s vs 3.338 s), the gap being fixed setup cost of which ~3 s is H.264 encoding and .NET startup rather than inference. Every network verified against the reference: text encoder cos 0.999999, DiT cos 0.998, both VAEs cos 1.000000/0.99999. CLI (--image, --end-image, --ref-image, --video-mode, --no-audio; the soundtrack is written as a sidecar .wav next to the MP4), /api/video-generate, /v1/videos/generations, and two auto-downloading configs — config/minimax-h3-fl2va.json and config/minimax-h3-ref2va.json — that fetch all four networks (~33.5 GB; only the denoiser differs between the two) and load them one at a time, so peak VRAM is the largest of them rather than their sum. → MiniMax-H3 card
• 🎬 Wan 2.1 / 2.2 video generation — video only (text → video and image → video). The video-only alternative to MiniMax-H3, and the home of the repository's single biggest speed lever. Prompt → H.264 MP4; on the Wan 2.2 models (TI2V-5B, I2V-A14B) an uploaded image becomes the video's first frame while the prompt drives motion, camera and scene changes. One resident-weight ggml graph per denoise step (CUDA-graph-captured, flash attention, per-token-timestep modulation for TI2V i2v), causal 3D video VAE encode+decode each as a single graph, A14B's two 14B experts hot-swapped at the timestep boundary, stagewise VRAM handoff — TI2V-5B generates 81-frame 480p image-to-video on a 16 GB GPU in under 8 min, and Wan 2.1 runs 6.0× faster end-to-end than stable-diffusion.cpp on the identical workload. Step-distilled checkpoints are auto-detected from the DiT file name (Turbo / distill / Lightning / lightx2v / FastWan / …-4steps-…) and switch to that step count with guidance off — 4 DiT passes instead of the official recipe's 100, which took the same 1088×832×121-frame image-to-video from 3 h 30 m to 17 m 30 s on an M5 Pro. It is the single biggest speed lever in the repository and needs no flag, only a different --model file. Numerics verified against diffusers (DiT cos > 0.995, VAE encoders cos > 0.999, decode 59.9 dB / >35 dB PSNR). CLI (--image), /v1/videos/generations, and Web UI chat with image upload. → Wan card
• 🌫️ DiffusionGemma text diffusion. Block-wise EntropyBound denoising over a Gemma-4-derived MoE backbone, with CLI flags and a Web UI denoising preview stream. → DiffusionGemma card
• 🖼️ Multimodal. Image / video / audio (Gemma 4); image input for Gemma 3, Qwen 3.5-family, Mistral 3, Nemotron-H Omni, and Muse-Glimmer; PDF documents via CLI and Web UI. → Multimodal
• 🛠️ Tool calling & thinking mode. Multi-turn tool calls and structured chain-of-thought across Qwen 3, Qwen 3.5/3.6-family, Gemma 4, GPT OSS, Nemotron-H, Muse-Glimmer (ATEM markup), and DeepSeek V4 (DSML markup). → Features
• 🔌 Ollama- & OpenAI-compatible APIs plus a browser chat UI — drop-in for existing tooling. → HTTP APIs
• 📄 Config files with auto-download. Put CLI/Server options in a reusable JSON file with ${variables} and { "path", "urls" } entries that fetch the model on first run. → config/README.md
• 🧮 Native quantized compute. Q4_K_M / Q8_0 / MXFP4 / IQ2_XXS / IQ2_S / IQ3_S / IQ3_XXS / IQ4_XS and more run in matmul without dequantizing to FP32. Runs on GGML Metal / CUDA / Vulkan, a direct CUDA/cuBLAS backend, MLX (Apple Silicon), and a pure-C# CPU path — all with CPU fallbacks. On MLX the IQ decode kernels amortize the codebook and scale loads across a whole sub-block instead of re-reading them per weight, which took a mixed-IQ 30B from 3.6 to 14.3 tok/s on an M5 Pro (67% of the fused ggml-metal graph, output byte-identical). → Backends

TensorSharp targets .NET 10. On a new machine, install the full .NET 10 SDK—the .NET Runtime alone cannot build TensorSharp:

Open a new terminal and verify that a 10.0.x SDK is listed:

dotnet --list-sdks

See the cross-platform .NET install overview or Development → Prerequisites for more detail.

Then get running in ~30 seconds on the verified native GGML fast path — Gemma 4 E4B. The other prerequisites are git, curl, CMake 3.20+ (the native GGML library is configured and built with it — on Windows, Visual Studio's "C++ CMake tools for Windows" component ships one and the build will find it), and the toolchain for your GPU backend (see Development → Prerequisites). The recommended public file is gemma-4-E4B-it-Q8_0.gguf (7.48 GiB); text-only inference needs no projector.

Windows + NVIDIA (PowerShell)

git clone https://github.com/zhongkaifu/TensorSharp.git; Set-Location TensorSharp
New-Item -ItemType Directory -Force models | Out-Null
curl.exe -L --fail "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models\gemma-4-E4B-it-Q8_0.gguf
'Answer in one short sentence: what is TensorSharp?' | Set-Content prompt.txt
$env:TENSORSHARP_GGML_NATIVE_ENABLE_CUDA = 'ON'
dotnet run --project TensorSharp.Cli -c Release -p:TensorSharpSkipMlxNative=true -- --model models\gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --max-tokens 128 --backend ggml_cuda

macOS (Apple Silicon) — drop the CUDA env var and use --backend ggml_metal.

Linux + NVIDIA — prefix the dotnet run with TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON and use --backend ggml_cuda.

AMD / Intel / NVIDIA Vulkan — set TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON and use --backend ggml_vulkan.

Linux (Ubuntu) + multiple NVIDIA GPUs — tensor parallelism

Tensor parallelism splits one model across N GPUs. It runs on the direct cuda backend and on the GGML CUDA / Vulkan backends (--backend ggml_cuda, ggml_vulkan). On architectures that shard no weights — Qwen 3.8 Flash Next, DeepSeek V4, GLM 5.x — the same flag runs a layer split instead: one contiguous run of whole layers per GPU. Install the CUDA toolkit first, then:

# On RunPod's Ubuntu 24.04 images, point the loader at the CUDA compat libraries first:
export LD_LIBRARY_PATH=/usr/local/cuda-12.6/compat:$LD_LIBRARY_PATH
# On older Ubuntu releases the .NET 10 SDK comes from the backports PPA:
add-apt-repository ppa:dotnet/backports

apt update && apt install dotnet-sdk-10.0
git clone https://github.com/zhongkaifu/TensorSharp.git
cd TensorSharp
mkdir models
wget "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -O models/gemma-4-E4B-it-Q8_0.gguf
bash TensorSharp.GGML.Native/build-linux.sh
dotnet build -c Release

# 2 GPUs in one process
TensorSharp.Cli/bin/TensorSharp.Cli --model models/gemma-4-E4B-it-Q8_0.gguf \
    --backend cuda --interactive --max-tokens 20000 --tp 2

# Same thing on the GGML CUDA backend (add TENSORSHARP_TP_DEVICES=0,2 to pick GPUs)
TensorSharp.Cli/bin/TensorSharp.Cli --model models/gemma-4-E4B-it-Q8_0.gguf \
    --backend ggml_cuda --interactive --max-tokens 20000 --tp 2

Scale the same model across machines by adding a node ID and the shared peer list — 2 nodes × 2 GPUs gives a global TP degree of 4:

# Node 0
TensorSharp.Cli/bin/TensorSharp.Cli --model models/gemma-4-E4B-it-Q8_0.gguf --backend cuda --tp 2 \
    --tp-node-id 0 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"
# Node 1 (same peer list, different node ID)
TensorSharp.Cli/bin/TensorSharp.Cli --model models/gemma-4-E4B-it-Q8_0.gguf --backend cuda --tp 2 \
    --tp-node-id 1 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"

TensorSharp.Server takes the same --tp, --tp-node-id, and --tp-peers flags (or the TENSORSHARP_TP_* environment variables); in a multi-node cluster the server is node 0 — the driver that serves HTTP — and every other node runs a TensorSharp.Cli worker. Full reference: Tensor Parallelism & Distributed Inference.

Host the same model as a server (browser UI at http://localhost:5000, plus Ollama/OpenAI APIs):

dotnet run --project TensorSharp.Server -c Release -p:TensorSharpSkipMlxNative=true -- --model models/gemma-4-E4B-it-Q8_0.gguf --backend ggml_cuda --max-tokens 512

The server binds 0.0.0.0:5000 by default (change it with --port / --host, or the PORT / HOST environment variables; on macOS port 5000 is taken by the AirPlay Receiver) with no built-in auth or TLS — keep it behind a firewall or an authenticated HTTPS reverse proxy. For image/video/audio add the companion mmproj-gemma-4-E4B-it-Q8_0.gguf with --mmproj.

Both executables print their full option reference — description, default, range, and an example per flag — when started with no arguments or with --help:

dotnet run --project TensorSharp.Cli -c Release -- --help
dotnet run --project TensorSharp.Server -c Release -- --help

Full command reference: CLI · Server · more models to download: Model Downloads · prefer a config file? config/.

Every backend falls back to CPU for any op it does not implement, so output stays correct on all of them.

Full per-backend description: Usage → Compute Backends.

Implemented and exercised by the test/benchmark matrix. Pick a quantization that fits your hardware (Q4_K_M for low memory, Q8_0 for higher quality). More sizes and projector files: Model Downloads.

Several families have a fast lane — a different artifact to download, or one flag — that changes the cost of a run by an order of magnitude. Reach for these before tuning anything else.

Per-family detail, including the numbers behind every row: MiniMax-H3 · Wan · Qwen-Image-Edit · DeepSeek V4 · Muse-Glimmer · Features.

End-to-end per-model documentation (origin, forward graph, components, parameters, prefill/decode optimizations): architecture cards.

A pure-.NET engine going toe-to-toe with the hand-tuned C++ llama.cpp on identical GGUF files, the same NVIDIA RTX 3080 Laptop GPU (16 GB), and one uniform OpenAI /v1/chat/completions surface — with both engines measured on their GGML CUDA and Vulkan builds. Numbers are the geomean speedup of TensorSharp over llama.cpp on the same backend (single-stream, greedy, MTP off); > 1.0× means TensorSharp is faster / lower-latency. Full per-scenario tables: docs/engine_comparison_report.md.

TensorSharp pulls clearly ahead on CUDA prefill / first-token latency (multi-turn prefill wins on every model, up to 1.49×), holds decode parity-or-better on CUDA, and wins Vulkan decode on the dense 12B (up to 1.32× on long context) — even at 2-bit IQ2_XXS quantization. The remaining sub-1.0× cells are active optimization targets. The harness also covers tool-calling, structured-output, image-edit (vs stable-diffusion.cpp), MTP on/off, and parallel-request scenarios you can run yourself via benchmarks/engine_comparison. Every cell is in the full report.

Models too large for that 16 GB rig carry their own head-to-head in their card, measured the same way (both engines, same GGUF, same machine, back to back): GLM-5.2 744B-A40B on 3x RTX PRO 6000 — TensorSharp leads prefill from ~1k prompt tokens up (pp2048 1.20×, pp4096 1.21×) and decode by 1.04×, with llama.cpp a few percent ahead on short prefills.

New here? The sections above are all you need to get running. Everything else is detailed reference:

Zhongkai Fu

See LICENSE for details.