The Year in AI Papers
The year/Independent research

Paper 2606.14777

JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence

Dingyu Yao, Junhao Zhou, Chenxu Yang, Chuanyu Qin, Haowen Hou, Zheming Liang, Congcong Wang, Yuhang Cao, Shenglong Ye, Shuai Xie, Shuhuan Gu, Haoyang Huang, et al.

Published
Jun 2026
Research lab
Independent
Citations
2
GitHub
1.7K stars

01 In brief

Summary

JoyAI-VL-Interaction introduces a paradigm shift from turn-based to proactive, streaming interaction for vision-language models.

The authors release an 8B-scale, vision-first model that continuously watches live video and decides each second to stay silent, respond, or delegate complex tasks to a background model.

Built on JoyAI-VL 1.0 with AdaCodec for efficient video tokenization, the model is trained on over 4M time-aligned clips covering six families of interaction data, using a weighted SFT loss and GRPO reinforcement learning.

The accompanying deployable system includes pluggable ASR/TTS, memory, visualization UI, and a background bridge, supporting hours of continuous video with sub-second latency.

In human evaluations across six real-world scenarios, JoyAI-VL-Interaction is preferred over Doubao's and Gemini's video-call assistants by wide margins (77.6% and 87.9% wins, respectively), with perfect scores in monitoring and alerting.

The model also exhibits emergent capabilities like guiding app navigation and improvising lectures.

All model weights, data, and system code are open-sourced, with release scheduled for June 20, 2026.

02 From the paper

Abstract

Many moments in the real world do not wait for a user to ask. A fire starts on a security monitor, an expression flickers across a video call, or a product a viewer wants flashes by in a livestream. Yet today's large models remain mostly turn-based by design: they answer only when addressed, and even video-call apps that appear interactive still operate as question-answer systems, reacting only when polled or prompted. We argue for a different paradigm: a model that is present in the world like a person. It continuously watches what is happening now, decides on its own whether to speak or stay silent, interacts in real time, and delegates to a background model when the problem is hard. To advance interaction models and their adoption across domains, we make two fully open-sourced contributions. First, we release JoyAI-VL-Interaction, an 8B-scale, vision-first VL-interaction model. The model makes the response decision internally, choosing each second to stay silent, respond, or delegate to a background model, and it excels at vision-triggered responsiveness and time awareness. We pair it with a transferable training recipe, from which capabilities we never trained for emerge, such as guiding a shopper through changing app screens or improvising a lecture from a slide deck. Second, we release a complete, deployable system built around that model. The system streams any ongoing video into the model, making it genuinely present in the world. All other components are pluggable, including ASR/TTS modules, memory, visualization UI, and a background brain that can connect to any API or agent. Across six real-world scenarios, human raters prefer JoyAI-VL-Interaction over the in-app video-call assistants of Doubao and Gemini by a wide margin. To our knowledge, this is the first open, vision-driven interaction model released together with its training recipe, data, and complete deployable system.