Moshi: a speech-text foundation model for real-time dialogue

https://arxiv.org/abs/2410.00037

Full-duplex spoken dialogue as speech-to-speech generation from a text LM backbone; models user and system speech as parallel codec token streams; Inner Monologue predicts time-aligned text before audio. Latency 160 ms theoretical, 200 ms practical.

Realtime speech-to-speech and expressive TTS for characters