ExLlamaV3
A leaner, faster rework of ExLlama's NVIDIA kernels — its new EXL3 quant is supported only here, at the cost of ecosystem compatibility.
https://github.com/turboderp-org/exllama
cudaEXL3EXL2
No benchmarks pasted on ExLlamaV3 yet. Paste one — it takes one keystroke and no form.