FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

llm-serving · GitHub Topics · GitHub

Explore Topics Trending Collections Events GitHub Sponsors
#

llm-serving

Star

Here are 366 public repositories matching this topic...

Language: All
Filter by language
All 366 Python 231 Jupyter Notebook 21 Go 19 Rust 14 Shell 13 C++ 9 TypeScript 9 HTML 5 JavaScript 5 Dockerfile 3
Sort: Most stars
Sort options
Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

vllm-project / vllm

Sponsor
Star 90.3k

A high-throughput and memory-efficient inference and serving engine for LLMs

amd cuda inference pytorch transformer openai moe llama gpt model-serving tpu kimi blackwell llm llm-serving qwen deepseek deepseek-v3 qwen3 gpt-oss
  • Updated Aug 28, 2026
  • Python

ray-project / ray

Star 43.6k

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

python data-science machine-learning reinforcement-learning deep-learning deployment tensorflow optimization parallel pytorch distributed hyperparameter-optimization ray hyperparameter-search serving rllib large-language-models llm llm-serving llm-inference
  • Updated Aug 28, 2026
  • Python

liguodongiot / llm-action

Star 25k

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

llm llmops llm-serving llm-training llm-inference
  • Updated Jul 19, 2026
  • HTML

NVIDIA / TensorRT-LLM

Star 14.5k

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

cuda pytorch moe blackwell llm-serving
  • Updated Aug 28, 2026
  • Python

bentoml / OpenLLM

Star 12.5k

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

llama mistral fine-tuning mlops bentoml vicuna llm model-inference llmops llm-serving llm-inference open-source-llm llama2 openllm llm-ops llama3-1 llama3-2 llama3-2-vision
  • Updated Aug 24, 2026
  • Python

skypilot-org / skypilot

Star 10.5k

The AI Compute Platform for frontier teams. SkyPilot turns fragmented AI compute into one AI supercomputer, so frontier AI teams build custom intelligence faster.

machine-learning deep-learning gpu job-scheduler slurm cloud-management spot-instances cloud-computing job-queue hyperparameter-tuning cost-optimization distributed-training multicloud ml-infrastructure tpu mlops ml-platform llm-serving llm-training
  • Updated Aug 28, 2026
  • Python

bentoml / BentoML

Star 8.8k

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

python machine-learning deep-learning model-serving multimodal mlops ml-engineering ai-inference llm generative-ai llmops llm-serving model-inference-service llm-inference inference-platform
  • Updated Aug 28, 2026
  • Python

gpustack / gpustack

Star 5.6k

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

cuda inference openai llama maas rocm ascend llm llm-serving vllm genai llm-inference qwen deepseek sglang distributed-inference high-performance-inference mindie
  • Updated Aug 28, 2026
  • Python

superduper-io / superduper

Star 5.3k

Superduper: End-to-end framework for building custom AI applications and agents.

python data database ai mongodb chatbot ml transformers inference torch pytorch pretrained-models semantic-search rag mlops distributed-ml vector-search llmops llm-serving llm-inference
  • Updated Sep 1, 2025
  • Python

predibase / lorax

Star 3.8k

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

transformers pytorch llama gpt lora model-serving fine-tuning llm llmops llm-serving llm-inference
  • Updated May 28, 2026
  • Python

PaddlePaddle / FastDeploy

Star 3.7k

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

inference openai serving ernie llm llm-serving vllm ernie-45 ernie-45-vl
  • Updated Aug 26, 2026
  • Python

thu-pacman / chitu

Star 3k

High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.

gpu pytorch model-serving llm llm-serving deepseek
  • Updated Aug 28, 2026
  • Python

vllm-project / vllm-ascend

Sponsor
Star 2.7k

Community maintained hardware plugin for vLLM on Ascend

inference transformer model-serving mlops ascend llm llmops llm-serving vllm
  • Updated Aug 28, 2026
  • C++

MoonshotAI / MoBA

Star 2.2k

MoBA: Mixture of Block Attention for Long-Context LLMs

pytorch transformer moe llm llm-serving llm-training flash-attention
  • Updated Apr 3, 2025
  • Python

microsoft / aici

Star 2.1k

AICI: Prompts as (Wasm) Programs

rust ai wasm inference transformer language-model model-serving wasmtime llm llmops llm-serving llm-inference llm-framework
  • Updated Jan 22, 2025
  • Rust

GradientHQ / parallax

Star 1.4k

Parallax is a distributed model serving framework that lets you build your own AI cluster anywhere

python distributed-systems chatbot pytorch transformer llama glm minimax kimi blackwell large-language-models llm llm-serving qwen deepseek oss-gpt decentralized-inference
  • Updated Jul 1, 2026
  • Python

alibaba / rtp-llm

Star 1.3k

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

inference llama gpt model-serving llm llmops llm-serving
  • Updated Aug 28, 2026
  • Cuda

ray-project / ray-llm

Star 1.3k

RayLLM - LLMs on Ray (Archived). Read README for more info.

ray llm llm-serving
  • Updated Mar 13, 2025

ovg-project / kvcached

Star 1.1k

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

serverless inference-engine llm llm-serving vllm llm-inference ollama llm-framework sglang kvcache gpu-sharing kvcached gpu-mutiplexing kvcache-optimization elastic-kvcache online-offline-coserve
  • Updated Aug 23, 2026
  • Python

efeslab / Nanoflow

Star 974

A throughput-oriented high-performance serving framework for LLMs

cuda inference model-serving llm llm-serving llama2
  • Updated Mar 29, 2026
  • Jupyter Notebook

Improve this page

Add a description, image, and links to the llm-serving topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the llm-serving topic, visit your repo's landing page and select "manage topics."

Learn more

Back | FazBrowse Home | New Git URL