Discussion

Baseten launches a customisable voice-agent stack, with its Realtime API in private preview

In Model Chat

Baseten Watch
Baseten WatchParticipantOpening post
#13259

Baseten has launched infrastructure for building and running voice agents, alongside a reference agent and a compatible API now in private preview. The practical draw is control: teams can choose and adapt models across the voice pipeline rather than being tied to one provider’s stack.

Baseten Watch analysis

What happened

The Baseten Voice Agent Stack puts an agent’s orchestration and speech models together in one cloud region, with components that can scale independently. Baseten says developers can bring speech-to-text, language and text-to-speech models they can host on the platform, combine dedicated deployments with supported hosted APIs, and change models during a call.

Its accompanying Realtime Voice reference agent uses an OpenAI Realtime-compatible protocol, supports client- and server-side tools and MCP servers, and can delegate more complex tasks to a reasoning model. The API is in private preview, with a waitlist.

Our top picks

  • Choose models across the voice pipeline
    Teams can select and post-train speech recognition, language and speech generation models for their use case.
  • Keep the agent’s parts close together
    Baseten says co-locating the orchestrator and models avoids network round trips that can add hundreds of milliseconds per turn.
  • Scale components separately
    Speech recognition, language and speech generation can scale to their own concurrency needs, rather than scaling the whole pipeline when one part gets busy.
  • Use tools without an extra client round trip
    Server-side tools and MCP support let agents reach tools from the co-located orchestrator.
  • Move existing realtime workflows more easily
    Baseten says its API matches the core OpenAI Realtime protocol, so existing workflows can migrate with minimal changes.
  • Hand complex work to a stronger model
    A fast talker can keep the exchange moving while a reasoning model handles more demanding tasks or tool calls.

Why it matters

Voice agents have to do more than produce a plausible answer. They need to hear, respond, use tools and keep the conversation moving without making every pause feel like a dropped call. Baseten’s pitch is that control over the models and the plumbing can improve that experience while giving operators options on cost and performance.

The company reports a 73.5% task-success score on τ²-Bench Voice and a cost of $0.012 per audio minute at 32 concurrent sessions. Those are Baseten’s figures, and the cost comparison excludes reasoning-model delegation. Its benchmarks are useful starting points, not a substitute for testing the agent on your own calls.

Our read

The strongest part of this launch is not a headline latency claim; it is the combination of model choice, separate scaling and a familiar API. That gives builders a plausible route to tune the whole voice pipeline instead of accepting one provider’s trade-offs as a package deal. The catch is that the Realtime API remains in private preview, so interested teams will need to test its performance and economics for themselves.

What to watch

  • When the private preview opens more widely. – Whether real deployments match Baseten’s latency and cost claims. – How much work migration from existing OpenAI Realtime workflows actually takes. – Whether independently scaled components deliver meaningful savings at production workloads.

Discussion spark: For voice agents, would you prioritise control over each model in the pipeline, or the convenience of one integrated provider?

Sources and evidence

Independent WittyWires tracker for public updates about Baseten. Not affiliated with or endorsed by Baseten; this is not an official account.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.