| Docs | Roadmap | Recipes | Examples | Prebuilt Containers | Digest | Design Proposals | How to Contribute |

The open-source, datacenter-scale inference stack. Dynamo is the orchestration layer above inference engines — it doesn't replace SGLang, TensorRT-LLM, or vLLM, it turns them into a coordinated multi-node inference system. Disaggregated serving, intelligent routing, multi-tier KV caching, and automatic scaling work together to maximize throughput and minimize latency for LLM, reasoning, multimodal, and video generation workloads.

Built in Rust for performance, Python for extensibility.

• You're serving LLMs across multiple GPUs or nodes and need to coordinate them
• You want KV-aware routing to avoid redundant prefill computation
• You need to independently scale prefill and decode (disaggregated serving)
• You want automatic scaling that meets latency SLAs at minimum total cost of ownership (TCO)
• You need fast cold-starts when spinning up new replicas

If you're running a single model on a single GPU, your inference engine alone is probably sufficient.

Feature support at a glance:

Full Feature Matrix → — LoRA, request migration, speculative decoding, and feature interactions.

Most inference engines optimize a single GPU or a single node. Dynamo is the orchestration layer above them — it turns a cluster of GPUs into a coordinated inference system.

Architecture Deep Dive →

• Zero-config deploy (DGDR) (beta): Specify model, HW, and SLA in one YAML — AIConfigurator auto-profiles the workload, Planner optimizes the topology, and Dynamo deploys
• Agentic inference: Per-request hints for priority, expected output length, and speculative prefill, plus session metadata for tracing and SGLang subagent KV isolation. LangChain + NeMo Agent Toolkit integrations
• Multimodal E/P/D: Disaggregated encode/prefill/decode with embedding cache — 30% faster TTFT on image workloads
• Video generation: Native FastVideo + SGLang Diffusion support — real-time 1080p on single B200
• K8s Inference Gateway plugin: KV-aware routing inside the standard Kubernetes gateway
• Storage-tier KV offload: S3/Azure blob support + global KV events for cluster-wide cache visibility

Dynamo can expose traffic through two Kubernetes request routing topologies. Both expose an OpenAI-compatible API and support the same backends, disaggregated serving, and KV-aware routing.

Request flow for the Dynamo-native path is client → Frontend → Router → workers. Request flow for the Gateway API path is client → Gateway → EPP → Frontend sidecar (direct) → workers.

See the Gateway API Inference Extension (GAIE) guide for the Gateway API setup, supported features, and configuration.

# Pull a prebuilt container (SGLang example)
docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.2.1

# Inside the container — start frontend and worker
python3 -m dynamo.frontend --http-port 8000 --discovery-backend file > /dev/null 2>&1 &
python3 -m dynamo.sglang --model-path Qwen/Qwen3-0.6B --discovery-backend file &

# Send a request
curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-0.6B",
  "messages": [{"role": "user", "content": "Hello!"}],
  "max_tokens": 100
}' | jq

Also available: tensorrtllm-runtime:1.2.1 and vllm-runtime:1.2.1.

Install uv (curl -LsSf https://astral.sh/uv/install.sh | sh), then:

uv pip install --prerelease=allow "ai-dynamo[sglang]"   # or [vllm]

Note: TensorRT-LLM requires pip with --extra-index-url https://pypi.nvidia.com. See the install guide for TRT-LLM-specific instructions.

Then start the frontend and a worker as shown above. See the full installation guide for system dependencies and backend-specific notes.

For production multi-node clusters, install the Dynamo Platform and deploy with a single manifest:

# Zero-config deploy: specify model + SLA, Dynamo handles the rest
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: my-model
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  sla:
    ttft: 200.0   # ms
    itl: 20.0     # ms
  autoApply: true

Pre-built recipes for common models:

See recipes/ for the full list. Cloud-specific guides: AWS EKS · Google GKE · Azure AKS · Amazon ECS

For contributors who want to build and develop locally. See the full build guide for details.

# Install system deps (Ubuntu 24.04)
sudo apt install -y build-essential libhwloc-dev libudev-dev pkg-config libclang-dev protobuf-compiler python3-dev cmake

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh && source $HOME/.cargo/env

# Create venv and build
uv venv dynamo && source dynamo/bin/activate
uv pip install pip 'maturin[patchelf]'
cd lib/bindings/python && maturin develop --uv && cd $PROJECT_ROOT
uv pip install -e lib/gpu_memory_service
uv pip install -e .

VSCode/Cursor users: see the .devcontainer for a pre-configured dev environment.

Dynamo is built in the open with an OSS-first development model. We welcome contributions of all kinds.

• Contribution Guide — How to contribute code, docs, and recipes
• Design Proposals — RFCs for major features
• Office Hours — Biweekly calls
• Community Meetings (Youtube) – Weekly (Wed 10:30 AM PT) development community meetings
• Discord — Chat with the team and community
• Dynamo Day Recordings — Deep dives from production users

• [03/15] Dynamo 1.0 is here — production-ready with strong community adoption
• [03/15] NVIDIA Blackwell Ultra sets new inference records in MLPerf
• [03/15] NVIDIA Blackwell leads on SemiAnalysis InferenceMax benchmarks
• [12/05] Moonshot AI's Kimi K2 achieves 10x inference speedup with Dynamo on GB200
• [12/02] Mistral AI runs Mistral Large 3 with 10x faster inference using Dynamo
• [11/20] Dell integrates PowerScale with NIXL for 19x faster TTFT

Dynamo provides comprehensive benchmarking tools:

• Benchmarking Guide – Compare deployment topologies using AIPerf
• SLA-Driven Deployments – Optimize deployments to meet SLA requirements

The OpenAI-compatible frontend exposes an OpenAPI 3 spec at /openapi.json. To generate without running the server:

cargo run -p dynamo-llm --bin generate-frontend-openapi

This writes to docs/reference/api/openapi.json.

Dynamo uses TCP for inter-component communication. On Kubernetes, native resources (CRDs + EndpointSlices) handle service discovery. External services are optional for most deployments:

Note: KV-aware routing does not require NATS. Enable KV events when you need event-backed cache-state tracking, or use --no-router-kv-events for prediction-based routing without external event infrastructure.

For Slurm or other distributed deployments that choose etcd or NATS JetStream-backed modes:

• etcd can be run directly as ./etcd.
• nats needs JetStream enabled: nats-server -js.

To quickly setup both: docker compose -f dev/docker-compose.yml up -d

• [11/20] Dell integrates PowerScale with Dynamo's NIXL for 19x faster TTFT
• [11/20] WEKA partners with NVIDIA on KV cache storage for Dynamo
• [11/13] Dynamo Office Hours Playlist
• [10/16] How Baseten achieved 2x faster inference with NVIDIA Dynamo
• [12/01] InfoQ: NVIDIA Dynamo simplifies Kubernetes deployment for LLM inference

• Support Matrix — Hardware, OS, CUDA, and backend versions
• Feature Matrix — Detailed backend compatibility
• Release Artifacts — Containers, wheels, Helm charts
• Service Discovery — K8s-native vs etcd vs file-based discovery
• Benchmarking Guide — Compare deployment topologies with AIPerf