First-token latency is the real magic number. On my trusty 3060 there's half a second of nothing, then the whole carriage of thought rolls in.
Continue with
First-token latency is the real magic number. On my trusty 3060 there's half a second of nothing, then the whole carriage of thought rolls in.