End-to-end speech models did not kill the ASR–LLM–TTS cascade
For a couple of years the standard way to build a voice agent has been three boxes in a row: ASR turns speech into text, an LLM decides what to say, TTS turns that back into speech. Then speech-native models arrived — audio in, audio out, one network — and the framing became "the cascade is legacy". The demos are genuinely impressive: the model laughs, it hesitates, it changes tone mid-sentence, it lets you interrupt.
Most of the arguments made for replacing the cascade are about latency, and most of them are wrong about where the latency is. Meanwhile the strongest argument for replacing it is rarely stated clearly, and the cost of doing so is usually understated. It is worth separating those.
The latency argument is mostly about endpointing
The intuitive case against the cascade is that three models in series must be slower than one. Add up ASR, LLM, TTS, and you get a number.
Look at where the time actually goes in a well-built cascade and that sum is not the problem. Streaming ASR is producing a stable transcript within a couple hundred milliseconds of the audio arriving. The LLM can be prompted with a partial transcript, so its prefill overlaps with the user still speaking. TTS, if it streams, returns its first packet in well under a hundred milliseconds. The hops are pipelined, not stacked.
The dominant term is something else entirely: deciding that the user has stopped talking. Classic voice pipelines do this with silence detection — wait N milliseconds of quiet and declare the turn over. Set N low and you cut people off mid-thought every time they pause to think. Set N high and every exchange carries that delay. Teams typically land somewhere around half a second, and that single parameter is usually a bigger share of perceived response time than the three models combined.
This is the thing speech-native models genuinely improve, and it is worth being precise about why. Turn-taking in human conversation is not signalled by silence; it is signalled by intonation, by syntactic completion, by pitch falling at the end of a clause. A model that sees the raw audio can use those cues and respond at the point a human would, including responding fast after a completed short utterance and waiting patiently through a mid-sentence pause.
But notice that this is a property of turn detection, not of end-to-end generation. You can build a semantic endpointing model — one that takes acoustics plus the partial transcript and predicts "the turn is over" — and drop it into a cascade. Several teams have. It recovers most of the conversational feel without touching the rest of the architecture. If latency is your complaint, that is the cheap fix, and it is a much smaller change than replacing all three stages.
The real argument: text is a lossy bottleneck
Here is the case that actually holds up. In a cascade, everything the user conveys has to survive being flattened into text, and everything the system conveys has to be reconstructed from text. Twice through that bottleneck, and what falls out is everything about how something was said.
Text does not carry that the user sounded frustrated, or hesitant, or that they were half-joking, or that they trailed off unsure. It does not carry that they are a child, or that there is a second person in the room, or that they are in a car. On the way back out, TTS has to guess the emotion for a sentence the LLM wrote without knowing how the words would land. Sarcasm becomes flat. A condolence and a confirmation get read the same way.
For a large class of products this loss is irrelevant. If someone is checking an order status, nothing about the emotional register changes the correct behaviour. For another class it is the entire product: companionship, language practice, coaching, anything where the system's job is to respond to a person rather than to a request. There, the cascade is not slightly worse — it is structurally unable to do the job, and no amount of engineering on the three boxes fixes it.
That is the honest dividing line. Not latency. Does your product's value live in what gets said, or in how it gets said?
What you give up
Replacing the cascade costs more than the demos suggest, and most of the cost is operational rather than perceptual.
Text is the only substrate you can audit. A cascade produces a transcript of everything the user said and everything the system said. You can log it, search it, redact PII from it, run compliance checks on it, sample it for quality, and hand it to an analyst. An end-to-end system's intermediate state is a sequence of audio tokens. You can transcribe the audio afterwards to get something similar, but that is a reconstruction after the fact, and it is not what the model actually conditioned on.
You lose per-stage substitution. In a cascade, each box is a separate procurement decision. You can move to a better ASR for one language, keep your TTS, and switch the LLM when a stronger one ships next quarter — which, in this field, is roughly always. An end-to-end model is one vendor for all three capabilities simultaneously, and you upgrade all of them or none.
The reasoning gap is real. Speech-native models are, as of now, well behind frontier text models on instruction following, tool use, and multi-step reasoning. If your agent has to call functions reliably, follow a long policy document, or stay accurate on domain facts, the cascade lets you put the best available text model at the centre of it. That gap will narrow. It has not closed.
Control over exact wording. Regulated products often need certain sentences said verbatim, and need certain sentences never said. A text stage is where you enforce that — you can inspect the string before it is spoken. With audio tokens there is no equivalent checkpoint.
Debuggability. When a cascade misbehaves you can tell which box did it: the transcript was wrong, or the transcript was right and the reply was wrong, or both were right and it sounded odd. That triage takes minutes. End-to-end, you have audio in and audio out, and a bad response has no intermediate evidence to inspect.
Barge-in is a transport problem either way
One thing worth flagging, because both architectures get it wrong in the same way: interruption is not solved by the model choice.
When a user speaks over the agent, you have to stop playback immediately and discard whatever audio is already buffered on the client. That part everyone gets. The subtle part is what the model believes happened afterwards. If you generated a three-sentence answer and the user cut in after four words, the model's context now claims it said three sentences, while the user heard four words. Every subsequent turn is built on a false premise about shared state, and the conversation quietly desynchronises — the agent refers back to things the user never heard.
The fix is to truncate the model's own history to what was actually played, which means the client has to report the playback position at the moment of interruption, and your session state has to be able to rewrite the last turn to match. This is fiddly in a cascade and fiddly in an end-to-end system. It is also the single most common reason a voice agent that demos well feels wrong in extended use.
How to choose
If the product is transactional — support, booking, ordering, data lookup, anything with tools and business rules — build the cascade. Fix turn-taking with a semantic endpointing model rather than a silence timer, stream both ends, and you will get response times that feel immediate, plus auditability and per-stage freedom you would otherwise be paying for later.
If the product is relational — the point is that the user feels heard — the cascade's text bottleneck is a ceiling you cannot engineer past, and end-to-end is the right architecture despite the operational cost.
And if you are in the middle, the useful hybrid is not "half a model". It is a cascade with an acoustic side channel: run a lightweight classifier on the input audio for emotion, energy, and speaker traits, pass those to the LLM as structured context, and let it choose delivery hints for the TTS stage. You recover a meaningful fraction of the paralinguistic loop while keeping text in the middle where you can see it. That is where most production systems should sit today, and it is a considerably less dramatic conclusion than the demos imply.