ExLlamaV2
The fastest EXL2/GPTQ runtime on NVIDIA consumer cards — great throughput per watt, but NVIDIA VRAM and a narrow model zoo.
https://github.com/turboderp/exllamav2
cudaEXL2GPTQFP8
No benchmarks pasted on ExLlamaV2 yet. Paste one — it takes one keystroke and no form.