AV-HuBERT · Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
2022 · ICLR 2022 · Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, Abdelrahman Mohamed
音视频联合表征的锚点：掩码多模态聚类预测，后面 AVSR / 唇形 / talking face 几乎都从这里分叉。
音视频联合
九篇。表征、AV-LLM、说话人脸。
ImageBind · ImageBind: One Embedding Space To Bind Them All
2023 · CVPR 2023 · Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, Ishan Misra
用图像当枢纽，把文本、音频、深度、热成像、IMU 绑进同一空间。早期 Video-LLM 的音频塔常用它。
Video-LLaMA · Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
2023 · arXiv 2023 · Hang Zhang, Xin Li, Lidong Bing
早期端到端 AV-LLM：Video Q-Former + ImageBind 音频编码器。
AV Speech Expert · Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation
2024 · arXiv 2024 · Dogucan Yaman, Fevziye Irem Eyiokur, Leonard Bärmann, Seymanur Aktı, Hazım Kemal Ekenel, Alexander Waibel
把 AV-HuBERT 从识别搬到生成：当唇形同步损失和三项新评测。
VideoLLaMA 2 · VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs
2024 · arXiv 2024 · Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, Lidong Bing
在 Video-LLaMA 上加 STC 连接器和联合训练的音频支路。
AV-Flow · AV-Flow: Transforming Text to Audio-Visual Human-like Interactions
2025 · arXiv 2025 · Aggelina Chatziagapi, Louis-Philippe Morency, Hongyu Gong, Michael Zollhöfer, Dimitris Samaras, Alexander Richard
文本直接出语音+画面，并行扩散 Transformer + flow matching，支持双人听。
INFP · INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations
2025 · CVPR 2025 · Yongming Zhu, Longhao Zhang, Zhengkun Rong, Tianshu Hu, Shuang Liang, Zhipeng Ge
对口型升级成双人对话：听/说角色由双轨音频自动切换。
VisG AV-HuBERT · VisG AV-HuBERT: Viseme-Guided AV-HuBERT
2026 · arXiv 2026 · Aristeidis Papadopoulos, Rishabh Jain, Naomi Harte
给 AV-HuBERT 加 viseme 辅助任务，针对「编码器本身」而不是再叠一层 LLM。
Zero-AV AVSR · Bootstrapping Audiovisual Speech Recognition in Zero-AV-Resource Scenarios with Synthetic Visual Data
2026 · arXiv 2026 · Pol Buitrago Gàlvez, Oriol Pareras, Javier Hernando et al.
没有真视频语料时，用 Wav2Lip 合成说话人脸，再微调 AV-HuBERT。