Awesome Chatgpt

A high-throughput and memory-efficient inference and serving engine for LLMs

SGLang is a high-performance serving framework for large language models and multimodal models.

Machine Learning Engineering Open Book

Nano vLLM

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

Production ready toolkit to run AI locally

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

Easily fine-tune, evaluate and deploy Qwen, Gemma, or any open weight LLM!

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

A programmable Mixture-of-Models router for heterogeneous LLM inference

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

Achieve state of the art inference performance with modern accelerators on Kubernetes

CSGHub is a brand-new open-source platform for managing LLMs, developed by the OpenCSG team. It offers both open-source and on-premise/SaaS solutions, with features comparable to Hugging Face. Gain full control over the lifecycle of LLMs, datasets, and agents, with Python SDK compatibility with Hugging Face. Join us! ⭐️

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

The fastest local AI engine for Apple Silicon. 4.2x faster than Ollama, 0.08s cached TTFT, 100% tool calling. 17 tool parsers, prompt cache, reasoning separation, cloud routing. Drop-in OpenAI replacement. Works with Claude Code, Cursor, Aider.

Open-source inference server and production cluster for all the models your agent needs.

Community maintained hardware plugin for vLLM on Ascend

Mastering Applied AI, One Concept at a Time

Continual learning infra for self-improving agents

🏗️ Fine-tune, build, and deploy open-source LLMs easily!