Natural Voice AI Is A Systems Problem: Alex Smola On Latency, Tokens, And Affordability

We started with text and a focus on AI for humans, but the text interface felt awkward. Rather than build an entirely new LLM just to obtain good audio, we explored how to add audio to intelligence already present in a strong language model. That requires care: training only on the new modality can make a model forget its earlier reasoning and language abilities. We had to maintain language-model training capabilities alongside audio training and use tasks that ground audio and text in each other. We released Higgs Audio V2 and subsequently accelerated releases across text-to-speech, audio understanding, and ASR. These are not equivalent capabilities. ASR maps sound to text, whereas audio understanding can reason over sound, a text prompt, and other audio references to resolve ambiguity. Tokens connect the modalities through the model. After developing these capabilities, we also had to tune inference performance to achieve latency low enough for genuinely interactive use.

Our ability to collect and process a large audio training dataset is a meaningful differentiator. We have on the order of 100 million hours of audio, which I describe as roughly 200 human lifetimes of exposure in a noisy environment, not necessarily all speech. Acquiring material is only part of the work: it has to be extracted, tagged, normalized, transcribed, and otherwise processed. Owning our data center helped make that feasible. Storing this volume on a neocloud would be extremely expensive, and the economics would be even harder with one of the three large cloud providers. Storage costs still matter for us: I estimate that hard-drive prices have risen to about three times their level a year earlier, so we need to be more prudent.

Noisy observations can still support better estimates when their errors can be averaged out, but systematic bias is a different problem. Averaging repeated readings from an unbiased, inaccurate voltmeter helps; it does not remove a biased instrument's error. More audio can produce a better model, and a better model can then improve the annotations, creating a useful feedback loop. Context and side information also help. Knowing that a podcast normally has two participants, for example, makes speaker separation easier and can yield large amounts of single-speaker audio. Similar ideas appear in face recognition: tracking an actor within a movie can provide reliable identity labels that become a harder dataset when actors from different movies are mixed. These are familiar statistical and machine-learning techniques, not a claim that noise is harmless. Applying them requires substantial compute and storage investment at scale.

It would be convenient if we could simply take an existing model and fine-tune it, but the modifications are substantial. We use a full training pipeline, with pretraining, mid-training, and post-training rather than only a lightweight adaptation. The input and final models can share a substantial part of their architecture while still being materially different. Significant training also happens on the language-model side so that the model retains its abilities. Publicly available models are therefore a useful starting prior, not the finished audio system.

Some choices are proprietary, while one model released the previous year was built on Llama and credited it in the model card. The decision varies with the task: audio generation, ASR, and audio understanding can require different designs. We may train on an existing model or build our own, depending on economics and required capabilities, including performance in a particular language. Taking advantage of an available model can be sensible, but it does not eliminate the audio-model workload. I would put that work at perhaps half or a third of the effort of the language model, not one-tenth. More capable conversational systems also need to maintain a foreground conversation while doing intelligent work in the background. Humans can do this while thinking about something else; machines are still improving, making architecture design an important frontier.

Small end-to-end audio models can be very responsive and low-latency, but increasing their intelligence can make the compute cost unaffordable. A more two-stage design can separate understanding and reasoning from other work, issuing tool calls in the background and receiving results while the interaction continues. Multi-threaded programming is a useful analogy: the main thread can dispatch work without doing everything itself. Tool calling and agentic programming help make this practical. We also want users to instruct the system much as they would a regular language agent. Behind that interface, considerable engineering decides when to answer directly, when to call a tool, and when the question is difficult enough to need a database or MCP server. Those decisions depend on model size and the information required, with affordability as a central constraint.

Examples include BigBench Audio and ComplexFuncBench, with an important caveat: the other models have thinking turned off. Long silent reasoning pauses feel unnatural in voice conversations. If a model needs time to think, it should integrate that wait into the interaction, much as a person might say, 'Let me think.'

We can use off-the-shelf MCP servers, but the number enabled simultaneously is currently limited. We expect the next iteration to support more. The constraint is tied to context-window size and inference cost: a large prefill and a large KV cache make subsequent token generation more expensive. One way to manage this is to call tools that themselves have MCP configured, rather than place everything directly in the front-end model's context. There are several engineering approaches to doing this efficiently. A front-end audio model with hundreds of billions of parameters might make an impressive demonstration, but its operating costs can be much less attractive. We have chosen affordability first.

Existing tooling can be reused; our search, news, and weather interactions are examples. Reusing a good server avoids reinventing the tools and helps customers integrate the system without learning new techniques. Slow services still require careful interaction design. People tolerate a wait much better when they are told it is coming. A progress indicator can shape the experience without solving the underlying timing problem; my example is an Apple boot bar based on the previous boot duration, although I am not sure that implementation is still current. Voice systems also need measurements beyond task completion. They should distinguish a listener's backchannel from an actual interruption, recover properly, and notice when something is missing and they should step in. We have released work on those behaviors. For human-facing AI, getting the task done remains necessary, but making the interaction pleasant matters too.

Text work gets us part of the way. A transcript can reveal some aspects of interruption handling, but choosing the right examples of human interaction is another problem. Movies are not automatically a good reference: romantic comedies can reward stalking, and other films normalize violence followed by reconciliation in ways that would have serious consequences in real life. Entertaining talk shows are not necessarily examples of normal behavior either. Models need more careful guidance about how interaction should work. Current LLMs have a decent theory of mind, which provides a starting point for exploring how people interact, but the experiments must be handled appropriately. Ethics matters; improving these systems is not permission to experiment carelessly on humans.

This is starting, and there are several routes. Simulated users can make some recursive improvement inexpensive because the interaction stays on a computer. We use digital personas and scenarios, including ones released by Nvidia, to train for people who may be uncooperative, abrasive, or trying to break the system. Individual personalization is another route: a model can reason about previous interactions and remember preferences, devices, language background, or the level of technical detail someone likes. Improvements should also generalize across interactions. The distinction resembles recommender systems: there is an overall behavioral model and adaptation to a particular person. Cultural expectations add another layer because behavior appropriate in one region may be inappropriate elsewhere. I expect systems increasingly to learn interaction skills from data rather than depend entirely on brilliant, individually engineered demonstrations. This includes staying composed with difficult users, a skill that many humans themselves do not consistently possess.

Interactions can create new training information rather than merely repeat what is already in a dataset. Each encounter can be another experiment and data point; exploration can reveal what works and what does not. The important issue is how to use that process appropriately. Some communication techniques can also be taught explicitly. For example, praise followed by criticism and then praise can make difficult feedback easier to receive. Management courses similarly teach techniques for dealing with people in different emotional states and helping them work productively. Those skills can be learned from instructions as well as eventually from interaction data. I expect substantial change over perhaps the next two or three years, though that is an expectation, not a demonstrated timetable.