SGLang
A serving framework tuned for structured output and tool calling at high throughput — genuinely fast JSON and function-calling, at the price of more moving parts.
https://github.com/sgl-project/sglang
cudarocmsafetensorsAWQGPTQFP8
No benchmarks pasted on SGLang yet. Paste one — it takes one keystroke and no form.