Paper 2512.23576
LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation
- Published
- Dec 2025
- Research lab
- Independent
- Citations
- 7
- GitHub
- 331 stars
01 In brief
Summary
The paper introduces LiveTalk, a real-time multimodal interactive video diffusion system.
It addresses the high inference cost of diffusion models by distilling a bidirectional, many-step model into a causal, 4-step autoregressive one.
The authors identify that the leading on-policy distillation method, Self Forcing, suffers from training instability and visual artifacts when applied to multimodal conditioning (text, image, audio).
They propose three improvements: refining multimodal conditions (e.g., using high-quality reference images and motion-focused text prompts), training ODE initialization to convergence, and using aggressive learning rates and tuned classifier-free guidance scales.
The distilled 1.3B model achieves 24.82 FPS (20x speedup) and sub-second first-frame latency, matching or exceeding larger baselines on HDTF, AVSpeech, and CelebV-HQ.
LiveTalk integrates the model with Qwen3-Omni for reasoning and speech, and uses a training-free Anchor-Heavy Identity Sinks (AHIS) technique for long-horizon identity preservation.
On a new multi-turn interaction benchmark, LiveTalk outperforms Sora2 and Veo3 in coherence and content quality while reducing latency from minutes to real-time.
02 From the paper
Abstract
Real-time video generation via diffusion is essential for building general-purpose multimodal interactive AI systems. However, the simultaneous denoising of all video frames with bidirectional attention via an iterative process in diffusion models prevents real-time interaction. While existing distillation methods can make the model autoregressive and reduce sampling steps to mitigate this, they focus primarily on text-to-video generation, leaving the human-AI interaction unnatural and less efficient. This paper targets real-time interactive video diffusion conditioned on a multimodal context, including text, image, and audio, to bridge the gap. Given the observation that the leading on-policy distillation approach Self Forcing encounters challenges (visual artifacts like flickering, black frames, and quality degradation) with multimodal conditioning, we investigate an improved distillation recipe with emphasis on the quality of condition inputs as well as the initialization and schedule for the on-policy optimization. On benchmarks for multimodal-conditioned (audio, image, and text) avatar video generation including HDTF, AVSpeech, and CelebV-HQ, our distilled model matches the visual quality of the full-step, bidirectional baselines of similar or larger size with 20x less inference cost and latency. Further, we integrate our model with audio language models and long-form video inference technique Anchor-Heavy Identity Sinks to build LiveTalk, a real-time multimodal interactive avatar system. System-level evaluation on our curated multi-turn interaction benchmark shows LiveTalk outperforms state-of-the-art models (Sora2, Veo3) in multi-turn video coherence and content quality, while reducing response latency from 1 to 2 minutes to real-time generation, enabling seamless human-AI multimodal interaction.