github.com · ML Frameworks
Inference framework that splits large mixture of experts models between CPU and GPU, keeping experts in system RAM so one consumer GPU can serve very large open weight models.
KTransformers is one of 21 ML framework tools on Axiomi. See all ML framework tools or how we check facts.
// guides
Run GLM 5.3 Flash locally with llama.cpp and Unsloth GGUFs, KTransformers or vLLM. VRAM and RAM requirements, which quant fits 128 GB, MTP and settings.
// related
Stratagithub.comMIT licensed local inference app and terminal that runs very large mixture of experts models on one consumer GPU by keeping busy experts in RAM and streaming the rest from SSD.ML FrameworksOpen source★ 38.9k
vLLMvllm.aiHigh throughput open source inference and serving engine built around PagedAttention and continuous batching, with an OpenAI compatible server that scales from one GPU to clusters.ML FrameworksOpen source★ 96.2k
Transformershuggingface.coHugging Face's library defining thousands of model architectures, the de facto reference implementation for open models.ML FrameworksOpen source★ 168.5kllama.cppgithub.comPlain C/C++ LLM inference engine that popularised GGUF quantised models running on everyday hardware.ML FrameworksOpen source★ 133.2k
SGLangsglang.ioOpen source (Apache 2.0) server for language, vision and diffusion models. RadixAttention reuses KV cache across prompts that share a prefix; v0.5.22 is current as of October 2026.ML FrameworksOpen source★ 38.4k
Ollamaollama.comOpen source local model runner with a one line pull and run CLI, a desktop app and a local HTTP API. Uses llama.cpp-based and MLX engines; optional paid cloud models.ML FrameworksFreemium★ 185.8k