Actions that Sound · Swoosh! Rattle! Thump! — Actions that Sound
2020 · RSS · Dhiraj Gandhi, Abhinav Gupta, Lerrel Pinto
碰撞声理解物体动力学
听着做
接触声、操作、VLA。 14 篇。
ObjectFolder 2.0 · ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer
2022 · CVPR · Ruohan Gao, Zilin Si, Yen-Yu Chang, Samuel Clarke, Jeannette Bohg, Li Fei-Fei, Wenzhen Yuan, Jiajun Wu
视听触物体神经先验
Play it by Ear · Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning
2022 · RSS · Maximilian Du, Olivia Y. Lee, Suraj Nair, Chelsea Finn
接触声解开遮挡操作
See Hear Feel · See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation
2022 · CoRL · Hao Li, Yizhi Zhang, Junzhe Zhu, Shaoxiong Wang, Michelle A. Lee, Huazhe Xu, Edward Adelson, Li Fei-Fei, Ruohan Gao, Jiajun Wu
视听触三模态融合操作
YCB-Impact · Recognizing object surface material from impact sounds for robot manipulation
2022 · IROS · Mariella Dimiccoli, Shubhan Patni, Matej Hoffmann, Francesc Moreno-Noguer
撞击声识别物体材料
AuRL · That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulation
2023 · CoRL · Abitha Thankaraj, Lerrel Pinto
听觉自监督动态操作
Hearing Touch · Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulation
2024 · ICRA · Jared Mejia, Victoria Dean, Tess Hellebrekers, Abhinav Gupta
接触麦当触觉预训练
ManiWAV · ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data
2024 · CoRL · Zeyi Liu, Cheng Chi, Eric Cousineau, Naveen Kuppuswamy, Benjamin Burchfiel, Shuran Song
野外手内音频示教
SonicSense · SonicSense: Object Perception from In-Hand Acoustic Vibration
2024 · CoRL · Jiaxun Liu, Boyuan Chen
手内振动感知物体
Audio-VLA · Audio-VLA: Adding Contact Audio Perception to Vision-Language-Action Model for Robotic Manipulation
2025 · arXiv · Xiangyi Wei, Haotian Zhang, Xinyi Cao, Siyu Xie, Weifeng Ge, Yang Li, Changbo Wang
接触音接入 VLA
VITA-E · VITA-E: Natural Embodied Interaction with Concurrent Seeing, Hearing, Speaking, and Acting
2025 · arXiv · Xiaoyu Liu, Chaoyou Fu, Chi Yan, Chu Wu, Haihan Gao, Yi-Fan Zhang, Shaoqi Dong, Cheng Qian, Bin Luo, Xiuyong Yang, Guanwu Li, Yusheng Cai, Yunhang Shen, Deqiang Jiang, Haoyu Cao, Xing Sun, Caifeng Shan, Ran He
边看边听边说边做
HEAR · Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation
2026 · IJRR · Chang Nie, Tianchen Deng, Guangming Wang, Zhe Liu, Hesheng Wang
声音中心 VSLA 范式
RoboOmni · RoboOmni: Proactive Robot Manipulation in Omni-modal Context
2026 · ICLR · Siyin Wang, Jinlan Fu, Feihong Liu, Xinzhe He, Huangxuan Wu, Junhao Shi, Kexin Huang, Zhaoye Fei, Jingjing Gong, Zuxuan Wu, Yu-Gang Jiang, See-Kiong Ng, Tat-Seng Chua, Xipeng Qiu
全模态主动推断操作
STMM · From Instruction to Event: Sound-Triggered Mobile Manipulation
2026 · arXiv · Hao Ju, Shaofei Huang, Hongyu Li, Zihan Ding, Si Liu, Meng Wang, Zhedong Zheng
声音事件触发移动操作