LLM engines

Inference engines powering LLMs.

Generic

• SGLang a high-performance serving framework. See also SGLang Cookbook
• TensorFold : LLM Inference Engine for Metal and CUDA.
• Lucebox : LLM speculative inference server for heterogeneous hardware & consumer GPUs (esp. AMD GPUs).

llama.cpp

llama.cpp : LLM inference in C/C++ supporting GGUF format.

• beellama.cpp : llama.cpp fork supporting the KVarN KV cache format.
• ik_llama.cpp : llama.cpp fork with new quants and improved performance for MoE models.
• KoboldCpp : llama.cpp fork with text, image, speech, music, and video generation.

vLLM

vllm : A high-throughput and memory-efficient inference and serving engine for LLMs.

• Intel vllm a vLLM docker image for Intel GPUs.
• spark-vllm-docker : a vLLM docker image for DGX sparks.
• vllm-radiance : A vLLM inference-server image for the AMD Radeon AI PRO R9700.

Mixture of Experts (MoE)

Tiered data access: hot experts on the GPU (VRAM), cold ones in the RAM, even data streaming from the SSD.

• Basalt : Local inference engine for Qwen3.8-Flash-Next on NVIDIA Blackwell.
• FreeToken : Local inference engine with Bandwidth-Adaptive Execution.
• PolyStrata : supports various MoE models.
• Project Maya : supports GLM-5.3-Flash (321B MoE), DeepSeek-V4.1-Flash-1017 (748B MoE), and MiniMax-M3.1-Flash (428B MoE).
• Strata : runs Qwen3.8-Flash-Next and DeepSeekV4.1-Flash-1017 from 12GB+ VRAM with smart expert caching.

Specialized

• TabbyAPI the official API server for exllamav3 backend, an inference library for the EXL3 quantization format (smaller bit per weight).
• ninfer, a C++/CUDA inference engine for explicitly registered Qwen checkpoints on a single NVIDIA GeForce RTX 5090.