Local LLM server for Windows with NVIDIA GPUs.

Aimed at small businesses (roughly 5–50 people) that want an Ollama-like experience, OpenAI-compatible APIs, and a full admin UI—without Docker or Linux-only stacks.

Inspired by:

• Ollama — simple UX and model workflow
• vLLM — multi-user serving ideas (scheduler / paging)
• ExLlamaV3 — fast EXL3 inference on NVIDIA
• Open WebUI — browser-based administration

Current release: 1.1.1 (stable). Beta: 1.2.1-beta — zero-gaps (A/B, LoRA worker, embeddings ONNX, quantize/import jobs, tools, multi-GPU settings, continuous batching). GitHub pre-release; not latest. Setup.exe bundles the ExLlamaV3 CUDA .pyd, worker deps, Python installer and VC++. PyTorch CUDA is downloaded during install. Admin → Models shows a VRAM fit badge (Fits / Tight / Too large).

Default after install: http://127.0.0.1:14563

ExLlamaSharp-Setup-win-x64.exe — latest GitHub Release.

One-liner (downloads Setup and launches UAC):

irm https://raw.githubusercontent.com/vitorcastro78/ExLlamaSharp/main/packaging/install-web.ps1 | iex

Double-click the Setup.exe → allow UAC → PyTorch CUDA downloads into the venv, ExLlamaV3 extension installs from the package → open http://127.0.0.1:14563.

ExLlamaSharp runs EXL3 models only (the ExLlamaV3 quantized format) on an NVIDIA GPU.

A model is a folder that looks like a Hugging Face snapshot:

How to get one:

1. Admin UI → Models → Library (default search exl3) → Download
2. Or import a local folder that already has those files
3. My Models → Load

Hugging Face repos often keep the actual weights on a bitrate branch such as 4.00bpw / 4.0bpw, not on main. The server resolves that revision automatically and will not treat a README-only clone as a successful download.

Examples that work:

• turboderp/MiniCPM5-1B-exl3
• turboderp/Llama-3.2-1B-Instruct-exl3 (revision 4.0bpw) — small demo / first-run smoke test

Search Hugging Face for exl3 (many IDs end in -exl3). Admin → Models shows a Fits / Tight / Too large badge against this machine’s GPU VRAM (estimate from weight size; it does not auto-select a model).

Not supported (will not load for real inference):

LoRA adapters can be registered in the UI/API and applied at generation time via the EXL3 worker (X-Adapter-Id or worker load_adapter).

Chat uses the tokenizer’s Hugging Face chat template when present (Llama 3 / ChatML fallbacks otherwise).

This section describes every product surface in the current build: installer, tray, Windows service, Admin UI pages, and APIs. Where something is still a stub, that is called out.

The Setup.exe (Inno Setup) installs a self-contained .NET host. End users do not need the .NET SDK, CUDA Toolkit, or LibTorch.

Firewall rule and Start Menu icon are created by the installer. The service listens on 14563 so it does not collide with common local ports (8080, 8787).

ExLlamaSharp.Tray.exe lives in the notification area (single instance). It:

• Polls service + http://127.0.0.1:14563/health every few seconds
• Shows green (healthy), yellow (service up but health failed), or grey (stopped)
• Open Admin UI (double-click or menu)
• Open data folder (%ProgramData%\ExLlamaSharp)
• Start / Stop / Restart the Windows service
• Registers itself in HKCU Run so it starts with the user session

Python resolution order: EXLLAMASHARP_PYTHON, exl3-runtime.json, the app venv, or a repo .venv-exl3. Set EXL3_BC_DSA=0 for current ExLlamaV3 workers.

Chat templates: the worker prefers tokenizer.apply_chat_template. If that is missing, Llama 3 special tokens are used when present; otherwise ChatML (<|im_start|> / <|im_end|>). Special tokens are stripped from streamed replies.

Use any OpenAI SDK. Point base_url at http://127.0.0.1:14563/v1 and send Authorization: Bearer <key>.

Auth: API keys with scopes (chat, completions, embeddings, admin). Per-key RPM/TPM limits return 429.

Ops (no API key): GET /health, GET /ready, GET /metrics (Prometheus).

Design system: Kortexio theme (DM Sans / Fraunces, teal accent). Login cookie + the built-in Admin API key are used so Chat and library calls work from the browser.

Dashboard (/)
Server overview: process status, requests today, loaded model name, GPU utilization / name. Optional advanced cards: tokens/sec, jobs waiting/running, VRAM. Toggle “Show advanced metrics”. First-run onboarding state is stored in %ProgramData%\ExLlamaSharp\ui-state.json.

Chat (/chat)
Playground that streams POST /v1/chat/completions. Shows whether a real model is loaded or the mock engine is answering. Enter sends; Shift+Enter is not required (single-line input). If nothing is loaded, the page tells you to open Models → My Models → Load.

Models (/models)
Three tabs:

• Library — live Hugging Face search (default query exl3). Shows name, repo id, parameter label, size, and Download. Gated repos need a token in Settings → Hugging Face. For EXL3 repos that keep weights on branches such as 4.00bpw (not main), the server resolves the revision automatically and refuses a “success” that only downloaded a README.
• My Models — folders under the models path that contain config.json (scanned from disk). Load puts that model on the GPU.
• Import — register an existing local folder + alias (does not copy files).

Jobs (/jobs)
Queue for pull / quantize / import. Cards for active, waiting, and recent (completed / failed / cancelled). Polls every 2 seconds. Pull jobs show downloaded/total bytes and parameter label. Cancel is available while a job is pending or running. Use Refresh if the Blazor poll looks stuck.

API Keys (/keys)
Create named keys with scopes. The plaintext secret is shown once. List and revoke existing keys. Use these from apps, curl, or the OpenAI SDK.

Usage (/usage)
Business view of the audit trail: requests in the last 7 days, prompt/completion tokens, estimated cost, and a recent activity table (endpoint, tokens, status, latency).

Team (/team)
Lists users who can manage the server (username, role, tenant, last active). Create the first admin in Setup if the list is empty.

Adapters (/adapters)
Table of registered LoRA adapters (name, path, rank, alpha). Apply at inference with header X-Adapter-Id on chat completions (worker loads PEFT LoRA globally).

Metrics (/dashboard/metrics)
Live tokens/sec and job counts. Chart placeholders for TPS / latency. A/B tab is UI scaffolding only.

Logs (/logs)
Live in-memory tail (start/stop), min level, text filter, plus refresh of persisted audit rows.

Diagnostics (/diagnostics)
Runs /health and /ready. Component cards (database, engine, inference, disk, …) and a short list of common fixes (no model loaded, missing nvidia-smi, port in use, API 401).

Tenants (/admin/tenants)
Create/list tenants in SQLite (id, name, subdomain). HTTP /api/v1/tenants is wired to the same DB; isolation at the request layer is not complete.

Settings (/settings)
Persisted server settings:

Setup (/setup)
Five-step wizard: welcome + GPU detect → create admin → models path → network (localhost vs LAN) → finish / optional starter model. Re-runnable from the sidebar.

API Guide (/api)
Copy-paste curl examples against the live base URL (/v1/models, chat completions, …).

About (/about)
Version, build date, .NET/OS, engine (mock/loaded/path/TPS), GPU name and VRAM.

Login (/login)
Username/password for the Admin UI. Default seed on a fresh database: admin / changeme.

• Windows Service + tray autostart
• API key auth and per-key rate limits
• Async audit trail (SQLite)
• Scheduled / on-demand backup
• Live log tail (UI + SSE)
• SignalR dashboard hub (subscribe / ping)
• PWA manifest / service worker on the Admin UI
• Self-contained publish (no .NET SDK on the target PC)

None for the core EXL3 text product after the zero-gaps initiative. By design (not stubs):

• OpenAI images / audio — 501; see the separate Media version plan
• Native DLL generate — worker-only for production text; DLL remains CI / scheduler ABI

1. Install from the latest Release (or the one-liner above).
2. Open http://127.0.0.1:14563 and sign in (admin / changeme).
3. Models → Library — search exl3 (for example turboderp/MiniCPM5-1B-exl3) and Download. Watch Jobs until completed.
4. Models → My Models → Load.
5. Open Chat and send a message.

If inference fails after a custom Python repair, run Setup-Exl3Python.bat in the install folder. The Setup.exe already installs the official CUDA wheel — do not pip install exllamav3 from PyPI.

dotnet restore ExLlamaSharp.slnx
dotnet build ExLlamaSharp.slnx -c Release
dotnet run --project src/ExLlamaSharp.Server/ExLlamaSharp.Server.csproj

Open http://127.0.0.1:14563.

powershell -ExecutionPolicy Bypass -File packaging\Build-Installer.ps1
# → publish\ExLlamaSharp-Setup-win-x64.exe
# → publish\ExLlamaSharp-Setup-win-x64.zip

Details: docs/INSTALL.md · packaging/README.md.

Data directory: %ProgramData%\ExLlamaSharp\ (app.db, models\, logs\, backups\).

Override for tests: environment variable EXLLAMASHARP_DATA_ROOT.

┌─────────────────────────────────────────┐
│  Browser PWA (any PC on the LAN)        │
│  Blazor UI (Kortexio theme)             │
└─────────────────────────────────────────┘
              ↓ HTTP
┌─────────────────────────────────────────┐
│  Windows Service / console (GPU host)   │
│  ┌───────────────────────────────────┐  │
│  │ Kestrel .NET 10                   │  │
│  │ ├─ OpenAI /v1                     │  │
│  │ ├─ Admin /api/v1                  │  │
│  │ └─ Blazor Server + PWA            │  │
│  └───────────────────────────────────┘  │
│  ┌───────────────────────────────────┐  │
│  │ ExLlamaV3WorkerEngine (Python)    │  │
│  │ tools/exl3_worker → ExLlamaV3     │  │
│  │ Real EXL3 CUDA kernels            │  │
│  └───────────────────────────────────┘  │
│              ↓ optional P/Invoke         │
│  ┌───────────────────────────────────┐  │
│  │ exllamasharp_native.dll (C++/CUDA)│  │
│  │ Scheduler, PageTable, EXL3 check  │  │
│  └───────────────────────────────────┘  │
│  SQLite · audit · backup · live logs    │
└─────────────────────────────────────────┘

More detail: docs/architecture.md.

Important: Do not push changes to upstream turboderp-org/exllamav3. Keep a local third_party/exllamav3 tree only (not in this git repo).

• User Manual
• Admin Guide
• API Reference
• Troubleshooting
• Architecture
• Packaging
• Swagger UI in Development: /swagger

*Typical production installs often use containers or Linux hosts.

# Prerequisites (as needed)
# - .NET 10 SDK
# - Visual Studio 2022+ with C++ (native builds)
# - CUDA Toolkit 12.8+ (native CUDA DLL)
# - CMake 3.25+
# - Python 3.11+ (real EXL3 worker)

dotnet restore ExLlamaSharp.slnx
dotnet build ExLlamaSharp.slnx -c Release

# Native stub (CI / no GPU toolkit)
.\packaging\build-native-stub.ps1

# Native CUDA (optional; validates EXL3 dirs)
.\packaging\build-native-cuda.ps1

# Real EXL3 path
.\packaging\Setup-Exl3Python.ps1
.\packaging\Download-DemoModel.ps1

dotnet run --project src/ExLlamaSharp.Server/ExLlamaSharp.Server.csproj
dotnet test ExLlamaSharp.slnx

# E2E feature matrix
dotnet test --filter FullyQualifiedName~E2eFeatureMatrix -c Release

Publish + Windows install:

.\packaging\Build-Installer.ps1

See third_party/README.md and packaging/install-cuda-libtorch.md.

Apache 2.0

• ExLlamaV3 — EXL3 CUDA kernels and model format
• vLLM — serving / scheduler inspiration
• Ollama — UX inspiration