TensorRT-LLM
NVIDIA's compiled-engine server that extracts peak throughput on its data-center GPUs — fast, but NVIDIA-only and demanding to build and maintain.
https://github.com/NVIDIA/TensorRT-LLM
cudasafetensorsFP8AWQGPTQ
No benchmarks pasted on TensorRT-LLM yet. Paste one — it takes one keystroke and no form.