realtime-voice-models technique sourced
Realtime speech-to-speech and expressive TTS for characters
Streaming speech recognition, speech-native language models, and context-aware text-to-speech that let characters hold spoken, interruptible conversations with roughly 200-250 ms response onset and generated emotional delivery.
mechanism
Two pipelines. Cascaded: streaming speech recognition, then an LLM, then streaming TTS, each optimized for time to first output. Speech-native: the model reads and writes audio tokens from a neural codec. Moshi models the user's and its own speech as parallel streams (full duplex, so it can listen while speaking) and predicts time-aligned text tokens before audio tokens. Hosted realtime APIs keep a persistent WebSocket session with voice activity detection, interruption handling, and function calls that can trigger game actions. Expressive TTS conditions delivery on context: Sesame's CSM feeds prior conversation audio and text to a Llama-style backbone over codec tokens, and Eleven v3 accepts inline tags such as [whispers] or [sighs]. Small ASR and TTS models (hundreds of millions of parameters) can run on device.
limitations
- Cost scales with talk time: at 2024 Realtime API rates, an hour split evenly between player and NPC speech is roughly $9 before context re-reads; gpt-realtime rates cut that to roughly $3 (estimates derived from list prices).
- Expressive delivery and low latency trade off: context-conditioned prosody still trails humans, and the most expressive TTS was not suited to real time at launch.
- Voice actor consent and compensation are now contract terms, and players have penalized AI voices in reviews.
- Speech-native models are hard to constrain to game-legal actions; most games keep a text layer between speech and game state.
- Microphone use and online connectivity raise privacy and access barriers (Whispers from the Star lists a broadband connection and an account).
relations
6 stored edgesNatural-language interaction
Characters can hold spoken, interruptible, full-duplex conversations with response onset near human turn-taking, replacing dialogue wheels or typed input; Whispers from the Star is built around this loop.
Procedural characters
Generated characters can get distinct designed or cloned voices and speak unscripted lines without recording sessions, which removes the voice-acting cost that keeps procedural characters silent.
Social exchanges
Delivery carries social meaning (whispering, sighing, hesitation, warmth) generated from conversational context, so the same line can land as a flirt, apology, or threat.
Unit commands
Squad orders can be spoken mid-action and parsed into function calls, as in PUBG Ally's voice-directed teammate, without menus.
backlinks
computed from other recordsclaims
10 with sourcesMoshi reported 160 ms theoretical and 200 ms practical latency for full-duplex spoken dialogue.
OpenAI's Realtime API launched in October 2024 at about $0.06 per minute of audio input and $0.24 per minute of output; gpt-realtime (August 2025) priced audio at $32 and $64 per million input and output tokens and raised function-calling accuracy on ComplexFuncBench from 49.7% to 66.5%.
Sesame's CSM reached parity with human recordings in listener preference without conversational context, but listeners preferred the human recordings when 90 seconds of prior context was provided.
Inworld's TTS-1.5 (January 2026) lists P90 time to first audio under 130 ms (Mini) and under 250 ms (Max) at $15 and $25 per million characters, about $0.015-0.025 per minute of speech.
Whispers from the Star (August 2025) is built around open-ended voice, video-call, and text conversation with an AI-voiced character created with an actor, and Where Winds Meet (global November 2025) discloses real-time AI NPC text and voice chat.
src-steam-2026-whispers-from-the-star, src-steam-2025-where-winds-meet
ARC Raiders (November 2025) was criticized and marked down by Eurogamer for generative AI NPC voices, and the SAG-AFTRA video game strike (July 2024 to July 2025) ended with contract terms on consent for AI voice replicas.
Eleven v3's expressive model was not recommended for real-time conversation at launch because of latency and reliability, so expressive range and low latency were still traded off in mid-2025.
A vendor case study reports Death by AI reached 20 million players in three months with 1.2 billion tokens in its first month after moving LLM and TTS to a gaming-priced provider; the cost claims are unverified.
Inworld's June 2026 price list puts streaming TTS at about $5-25 per million characters and speech-to-text at about $0.10 per hour, and claims only about 3 in 100 consumer AI users ever pay.
In GDC's 2026 survey of over 2,300 professionals, 36% used generative AI at work but 52% thought it was harming the industry, up from 30% in 2025, with game design and narrative (63%) and art (64%) among the most negative groups, which bears on adoption of generated character voices and dialogue.
src-gdc-2026-state-of-industry, src-gdc-2025-state-of-industry