Your speech models are LLMs now, and it breaks how you size a GPU

There was a stretch where speech systems and language models were separate crafts. ASR was an encoder with a CTC or attention decoder, TTS was an acoustic model feeding a vocoder, and both were served like ordinary neural networks: fixed input, fixed compute, a latency you could estimate from FLOPs and batch size.

That is no longer what most teams are running. The systems people actually deploy today — Whisper-family ASR, CosyVoice, Fish Speech, VoxCPM, the omni-style models that take audio in and emit audio out — are autoregressive transformers that predict discrete audio tokens, one step at a time, with a KV cache. The vocoder at the end is a rounding error in the compute budget. Everything expensive about them is decode.

The modelling win is obvious and well covered: cloning a voice from a few seconds of reference audio, prosody that survives long sentences, one architecture for many languages. The operational consequence is less discussed, and it is the one that shows up in your on-call rotation: you are no longer running a speech service. You are running an LLM inference service that happens to emit audio. Most of what you know about sizing the first thing is wrong about the second.

RTF stopped being a property of the model

Real-time factor — seconds of compute per second of audio produced — is the number every speech vendor publishes and every buyer plans with. An RTF of 0.15 sounds like a clean unit of capacity: one GPU produces audio about seven times faster than real time, so call it six or seven concurrent streams with headroom.

That arithmetic assumed RTF was a constant. For an autoregressive model served with continuous batching, it is a function of what else is on the card.

Decode is memory-bandwidth bound. Running one stream alone, you are pulling the entire weight matrix through memory to produce a single token, and the GPU is mostly idle — which is exactly why the published single-stream RTF looks so good, and also why it is nearly free to add a second stream. Batch eight streams and each step still costs roughly what one step cost, so aggregate throughput climbs almost linearly while per-stream RTF barely moves. This is the regime where capacity planning feels easy and cheap.

Then it stops. Past some batch size the step time starts growing with the batch, and per-stream RTF degrades in lockstep for every stream on the card, including the ones that were already running and were previously fine. The curve is not a gentle slope into a wall; it is flat and then it isn't, and the knee position depends on model size, sequence length, and how much KV cache you have room for.

So the honest version of the vendor number is: RTF 0.15 at concurrency 1 on an idle card. It tells you the shape of the model. It does not tell you how many users fit on a GPU.

What saturation sounds like

Here is the part that makes speech different from chat, and the reason you cannot reuse an LLM serving playbook unchanged.

When a text LLM endpoint goes past its knee, everyone's tokens arrive a bit slower. The UI still streams, the answer still lands, and most users register nothing worse than "it's a little sluggish today". Degradation is proportional and it is survivable.

Voice has a hard threshold underneath it. Once playback has started, the client is consuming audio at exactly one second per second, and generation has to stay ahead of that or the buffer runs dry. Cross the line and you do not get slightly slower audio — you get a stutter in the middle of a word, which every listener reads as a broken product rather than a busy one. The same 20% slowdown that a chat user would not notice takes a stream that was generating at 1.3× real time and puts it under 1.0×.

And it does not hit one user. Continuous batching means the marginal stream taxes every stream sharing the batch. The user who tips you over the knee is fine. Everyone already mid-sentence is the one who hears it break. That inversion — new load damaging existing sessions rather than queueing behind them — is the single most important operational difference between serving speech and serving text.

The capacity number you actually want

Given that, throughput is the wrong target. The number to plan with is: the highest concurrency at which p99 per-chunk generation still beats real-time playback, with margin.

Measuring it takes an afternoon:

1. Pick a representative workload — real sentence lengths, real reference audio, your actual voices. Long-tail sequence lengths matter here, because KV cache scales with them.
2. Ramp concurrency: 1, 2, 4, 8, 16 streams, holding each level long enough to reach steady state.
3. At every level, log per-chunk arrival timestamps for every stream, and compute audio seconds delivered per wall-clock second, per stream, at p99 rather than mean.
4. Find the concurrency where p99 crosses about 1.2×. That is your ceiling. Not 1.0× — that is the cliff edge, and you need room for a network hiccup, a longer-than-usual sentence, and a GC pause.

Two things about the result usually surprise people. It is often meaningfully lower than throughput-based sizing suggests, because throughput saturates well after quality does. And at that concurrency the GPU may sit around half utilised, which will look like waste on a dashboard and is in fact the cost of not stuttering. If someone asks you to push utilisation up, that is the trade you are declining.

Admission control beats graceful degradation

The corollary: when you are at the ceiling, refuse the next stream. Do not accept it and let everyone slow down together.

This is the opposite instinct from text serving, where admitting one more request and spreading the pain is usually right. For streaming audio, spreading the pain converts one rejected session into ten audibly broken ones. A rejection at connect time is a retry, a fallback voice, a queue position — all recoverable. A stutter at second nine of a sentence is not recoverable, and it is what the user remembers.

That means a real queue in front of the model, with a bounded concurrency limit, rather than relying on the inference engine's internal scheduler to sort it out. And it means watching the right signal: with continuous batching, the engine's queue depth stays at zero while quality collapses, because nothing is queueing — everything is admitted and everything is slower. Queue depth looks healthy right up until users start complaining. Per-chunk inter-arrival time at p99 is the metric that actually moves before the phone rings.

Two smaller traps from the same root cause

Co-location doesn't work the way you'd hope. LLM serving engines typically pre-allocate a large fraction of VRAM for KV cache at startup, so a chat model and a TTS model on one card is not a negotiation — whichever starts first takes what it asked for and the other fails or thrashes. This is mundane and it still costs teams a day, usually the day of a launch. Give real-time speech its own card, or pin explicit memory fractions and test the interaction before you need it.

ASR inherited the same properties. Streaming recognition through an autoregressive decoder has the same batch-dependent step time, plus a knob TTS doesn't have: how much right-hand context each chunk gets before emitting. More context, better accuracy, more latency — and that setting interacts with load, because a chunk that arrives late has already spent its latency budget. If your pipeline is ASR into an LLM into TTS, all three stages now share a failure mode, and the end-to-end budget is not the sum of three independently measured means. It is dominated by whichever stage is closest to its own knee.

The takeaway

The shift to token-based speech models bought real quality, and it moved the operational problem from "is the model fast enough" to "how many streams can this card hold before the audio breaks". Those are different questions with different answers.

Stop planning from a model card's RTF. Measure p99 per-chunk delivery under a concurrency ramp, set your limit where it stays comfortably above real time, and cap admissions there. Then accept the utilisation you get, because the alternative is a GPU that looks efficient on a graph and sounds broken in an ear.

If you would rather not own that curve at all, that is roughly what a hosted endpoint is for — MiraEcho's streaming synthesis is free to start, and the concurrency limits are ours to keep honest.