ModelTC/LightLLM: LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance. · GitHub

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance. LightLLM harnesses the strengths of numerous well-regarded open-source implementations, including but not limited to FasterTransformer, TGI, vLLM, and FlashAttention.

English Docs | 中文文档 | Blogs

• [2025/11] 🚀 Prefix KV Cache Transfer between DP rankers is now supported! Check out the technical deep dive in our blog post.

• [2025/09] 🔥 LightLLM v1.1.0 release!
• [2025/08] Pre $^3$ achieves the outstanding paper award of ACL2025.
• [2025/05] LightLLM paper on constrained decoding accepted by ACL2025 (Pre $^3$: Enabling Deterministic Pushdown Automata for Faster Structured LLM Generation). For a more accessible overview of the research with key insights and examples, check out our blog post: LightLLM Blog
• [2025/04] LightLLM paper on request scheduler published in ASPLOS’25 (Past-Future Scheduler for LLM Serving under SLA Guarantees)
• [2025/02] 🔥 LightLLM v1.0.0 release, achieving the fastest DeepSeek-R1 serving performance on single H200 machine.

• Install LightLLM
• Quick Start
• TuTorial

Learn more in the release blogs: v1.1.0 blog.

Please refer to the FAQ for more information.

We welcome any coopoeration and contribution. If there is a project requires LightLLM's support, please contact us via email or create a pull request.

Projects based on LightLLM or referenced LightLLM components:

• LoongServe, Peking University
• vLLM (some LightLLM's kernel used)
• SGLang (some LightLLM's kernel used)
• ParrotServe, Microsoft
• Aphrodite (some LightLLM's kernel used)
• S-LoRA
• OmniKV, Ant Group
• Lab4AI LightLLM+LlamaIndex, Lab4AI LightLLM+Qwen3-8B
• LazyLLM

Also, LightLLM's pure-python design and token-level KC Cache management make it easy to use as the basis for research projects.

Academia works based on or use part of LightLLM:

• ParrotServe (OSDI’24)
• SLoRA (MLSys’24)
• LoongServe (SOSP’24)
• ByteDance’s CXL (Eurosys’24)
• VTC (OSDI’24)
• OmniKV (ICLR’25)
• CaraServe, LoRATEE, FastSwitch ...

For further information and discussion, join our discord server. Welcome to be a member and look forward to your contribution!

This repository is released under the Apache-2.0 license.

We learned a lot from the following projects when developing LightLLM.

• Faster Transformer
• Text Generation Inference
• vLLM
• SGLang
• flashinfer
• Flash Attention 1&2
• OpenAI Triton

We have published a number of papers around components or features of LightLLM, if you use LightLLM in your work, please consider citing the relevant paper.

constrained decoding: accepted by ACL2025 and achieved the outstanding paper award.

@inproceedings{chen-etal-2025-pre3,
    title = "Pre$^3$: Enabling Deterministic Pushdown Automata for Faster Structured {LLM} Generation",
    author = "Chen, Junyi and Bai, Shihao and Wang, Zaijun and Wu, Siyu and Du, Chuheng and Yang, Hailong and Gong, Ruihao and Liu, Shengzhong  and Wu, Fan and Chen, Guihai",
    editor = "Che, Wanxiang and Nabende, Joyce and Shutova, Ekaterina and Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.acl-long.551/",
    doi = "10.18653/v1/2025.acl-long.551",
    pages = "11253--11267",
    ISBN = "979-8-89176-251-0",
}

Request scheduler: accepted by ASPLOS’25:

@inproceedings{gong2025past,
  title={Past-Future Scheduler for LLM Serving under SLA Guarantees},
  author={Gong, Ruihao and Bai, Shihao and Wu, Siyu and Fan, Yunqian and Wang, Zaijun and Li, Xiuhong and Yang, Hailong and Liu, Xianglong},
  booktitle={Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2},
  pages={798--813},
  year={2025}
}