Engine favorites
The ranking is the community's — your votes decide the order, and we don't play favorites.
The reference GGUF runtime with the widest platform reach — CPU and Apple Metal are its strong suit, and most other tools wrap it.
Benchmarks run with it →A polished GUI that packages llama.cpp and MLX behind point-and-click — the least-friction way to try GGUF models, with fewer tuning knobs.
Benchmarks run with it →Apple's own framework that parks big models on unified memory via 4-bit MLX quants — the pragmatic choice when your only GPU is a Mac.
Benchmarks run with it →A serving framework tuned for structured output and tool calling at high throughput — genuinely fast JSON and function-calling, at the price of more moving parts.
Benchmarks run with it →Purpose-built for high-throughput multi-user serving with PagedAttention — overkill for a single interactive session, but the standard for production APIs.
Benchmarks run with it →The fastest EXL2/GPTQ runtime on NVIDIA consumer cards — great throughput per watt, but NVIDIA VRAM and a narrow model zoo.
Benchmarks run with it →A leaner, faster rework of ExLlama's NVIDIA kernels — its new EXL3 quant is supported only here, at the cost of ecosystem compatibility.
Benchmarks run with it →An offline-first desktop chat client with a local-first philosophy and pluggable engines — privacy-minded, though runtime updates can lag upstream.
Benchmarks run with it →A llama.cpp-based sandbox aimed at writers and game masters — rich sampler controls and a browser UI, though not a general-purpose API server.
Benchmarks run with it →Mozilla's single-executable GGUF packs the model and runtime into one chmod-and-run file — ideal for sharing demos, less for fleet serving.
Benchmarks run with it →An OpenAI-compatible API server that routes several backends and formats — handy for swapping models behind one endpoint, without tight coupling to any single runtime.
Benchmarks run with it →A one-command model manager and OpenAI-style server built on llama.cpp — unbeatable for quick LLM starts, with the flexibility of the underlying runtime.
Benchmarks run with it →NVIDIA's compiled-engine server that extracts peak throughput on its data-center GPUs — fast, but NVIDIA-only and demanding to build and maintain.
Benchmarks run with it →