LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance. LightLLM harnesses the strengths of numerous well-regarded open-source implementations, including but not limited to FasterTransformer, TGI, vLLM, and FlashAttention.
English Docs | 中文文档 | Blogs
- [2025/11] 🚀 Prefix KV Cache Transfer between DP rankers is now supported! Check out the technical deep dive in our blog post.
- [2025/09] 🔥 LightLLM v1.1.0 release!
- [2025/08] Pre
$^3$ achieves the outstanding paper award of ACL2025. - [2025/05] LightLLM paper on constrained decoding accepted by ACL2025 (Pre
$^3$ : Enabling Deterministic Pushdown Automata for Faster Structured LLM Generation). For a more accessible overview of the research with key insights and examples, check out our blog post: LightLLM Blog - [2025/04] LightLLM paper on request scheduler published in ASPLOS’25 (Past-Future Scheduler for LLM Serving under SLA Guarantees)
- [2025/02] 🔥 LightLLM v1.0.0 release, achieving the fastest DeepSeek-R1 serving performance on single H200 machine.
Learn more in the release blogs: v1.1.0 blog.
Please refer to the FAQ for more information.
We welcome any coopoeration and contribution. If there is a project requires LightLLM's support, please contact us via email or create a pull request.
Projects based on LightLLM or referenced LightLLM components:
- LoongServe, Peking University
- vLLM (some LightLLM's kernel used)
- SGLang (some LightLLM's kernel used)
- ParrotServe, Microsoft
- Aphrodite (some LightLLM's kernel used)
- S-LoRA
- OmniKV, Ant Group
- Lab4AI LightLLM+LlamaIndex, Lab4AI LightLLM+Qwen3-8B
- LazyLLM
Also, LightLLM's pure-python design and token-level KC Cache management make it easy to use as the basis for research projects.
Academia works based on or use part of LightLLM:
- ParrotServe (OSDI’24)
- SLoRA (MLSys’24)
- LoongServe (SOSP’24)
- ByteDance’s CXL (Eurosys’24)
- VTC (OSDI’24)
- OmniKV (ICLR’25)
- CaraServe, LoRATEE, FastSwitch ...
For further information and discussion, join our discord server. Welcome to be a member and look forward to your contribution!
This repository is released under the Apache-2.0 license.
We learned a lot from the following projects when developing LightLLM.
- Faster Transformer
- Text Generation Inference
- vLLM
- SGLang
- flashinfer
- Flash Attention 1&2
- OpenAI Triton
We have published a number of papers around components or features of LightLLM, if you use LightLLM in your work, please consider citing the relevant paper.
constrained decoding: accepted by ACL2025 and achieved the outstanding paper award.
@inproceedings{chen-etal-2025-pre3,
title = "Pre$^3$: Enabling Deterministic Pushdown Automata for Faster Structured {LLM} Generation",
author = "Chen, Junyi and Bai, Shihao and Wang, Zaijun and Wu, Siyu and Du, Chuheng and Yang, Hailong and Gong, Ruihao and Liu, Shengzhong and Wu, Fan and Chen, Guihai",
editor = "Che, Wanxiang and Nabende, Joyce and Shutova, Ekaterina and Pilehvar, Mohammad Taher",
booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.acl-long.551/",
doi = "10.18653/v1/2025.acl-long.551",
pages = "11253--11267",
ISBN = "979-8-89176-251-0",
}Request scheduler: accepted by ASPLOS’25:
@inproceedings{gong2025past,
title={Past-Future Scheduler for LLM Serving under SLA Guarantees},
author={Gong, Ruihao and Bai, Shihao and Wu, Siyu and Fan, Yunqian and Wang, Zaijun and Li, Xiuhong and Yang, Hailong and Liu, Xianglong},
booktitle={Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2},
pages={798--813},
year={2025}
}