一种遥操作场景下的混合现实视觉增强交互系统

施乐洋; 李晓玲; 王龙; 陈漳沂; 郑帅

doi:10.7652/xjtuxb202310003

您当前的位置：

首页 >

文章列表页 >

一种遥操作场景下的混合现实视觉增强交互系统

专题医工交叉与人机交互 | 更新时间：2025-05-08

- 一种遥操作场景下的混合现实视觉增强交互系统
- A Mixed Reality Visual Augmentation and Interaction System in Teleoperation Scenarios
- 西安交通大学学报 2023年57卷第10期页码：20-29
- 作者机构：
  
  1. 西安交通大学机械工程学院,西安,710049
  2. 西安交通大学电子与信息学部,西安,710049
- 作者简介：
  
  : 2022-12-31。作者简介: 施乐洋(1997—),男,硕士生
  李晓玲(通信作者),女,教授,博士生导师。基金项目: 陕西省重点研发计划资助项目(2023-YBGY-505)。
- 基金信息：
  
  : 2022-12-31。作者简介: 施乐洋(1997—),男,硕士生;李晓玲(通信作者),女,教授,博士生导师。基金项目: 陕西省重点研发计划资助项目(2023-YBGY-505)。
- DOI：10.7652/xjtuxb202310003
  中图分类号： TP391.7
- 网络首发：2023-10-10，
  
  纸质出版：2023
- 稿件说明：
移动端阅览
施乐洋, 李晓玲, 王龙, 等. 一种遥操作场景下的混合现实视觉增强交互系统[J]. 西安交通大学学报, 2023,57(10):20-29.

SHI Yueyang, LI Xiaoling, WANG Long, et al. A Mixed Reality Visual Augmentation and Interaction System in Teleoperation Scenarios[J]. 2023, 57(10): 20-29.
施乐洋, 李晓玲, 王龙, 等. 一种遥操作场景下的混合现实视觉增强交互系统[J]. 西安交通大学学报, 2023,57(10):20-29. DOI： 10.7652/xjtuxb202310003.

SHI Yueyang, LI Xiaoling, WANG Long, et al. A Mixed Reality Visual Augmentation and Interaction System in Teleoperation Scenarios[J]. 2023, 57(10): 20-29. DOI： 10.7652/xjtuxb202310003.

摘要

针对遥操作机器人系统在高风险任务中近端交互临场感差、操作者难以获得真实体验感的问题

提出了一种适用于遥操作场景的混合现实视觉增强交互系统。首先

针对远程场景传输和机械臂的远程控制这两个重要环节

通过分析遥操作交互任务中的目标需求

从人体视觉感知与认知特性的角度构建并设计了遥操作场景下的视觉增强交互系统框架; 然后根据系统功能需求

研究了混合现实场景下的远程视景实时呈现技术

设计了混合现实场景下的机械臂交互控制策略

实现了混合现实全息对象与真实物理场景相映射的虚实融合效果

并以此搭建了遥操作场景下的混合现实视觉增强交互系统

实现了混合现实终端的可视化与系统端到端的网络通信。实验结果表明:与传统的二维系统相比

该交互系统所采用视觉增强形式的任务完成时间减少了17.3%

准确度提高了7%。证明该系统具有较好的应用前景。

Abstract

To address the challenges of poor situational awareness and lack of realistic experience for operators in high-risk teleoperation tasks faced by robotic systems

a mixed reality visual augmentation and interaction system applicable to teleoperation was introduced. Firstly

the system framework was designed from the perspective of human visual perception and cognitive characteristics and based on requirements for teleoperation interaction tasks

with a specific focus on remote scene transmission and robotic arm control in teleoperation scenarios. Secondly

according to the system's functional requirements

extensive research was conducted on real-time rendering techniques in mixed reality environments

and an interactive control strategy of the robotic arm was developed for mixed reality environments. This effort enabled a fusion effect between virtual and physical scenes by mapping holographic objects to the real-world environment. This supports the establishment of the mixed reality visual augmentation and interaction system that allows operators to visualize the mixed reality terminal and facilitates seamless end-to-end network communication. Experimental results show that the interaction system has better performance than a traditional system

as manifested by a remarkable 17.3% reduction in task completion time and a notable 7% improvement in accuracy. These findings suggest the promising application prospects of the system.

关键词

Keywords

references

张宇, 李宇翔, 张贺飞, 等. 撼地者: 基于遥操作与自主导航的移动救援机器人 [J]. 中国科学技术大学学报, 2023, 53(1): 37-49, 71-72.

ZHANG Yu, LI Yuxiang, ZHANG Hefei, et al. Earthshaker: a mobile rescue robot for emergencies and disasters through teleoperation and autonomous navigation [J]. Journal of University of Science and Technology of China, 2023, 53(1): 37-49, 71-72.

曾建军. 排爆机器人遥操作性能研究 [D]. 上海: 上海交通大学, 2010.

SUN Zeyuan, YANG Hong, MA Yifan, et al. BIT-DMR: a humanoid dual-arm mobile robot for complex rescue operations [J]. IEEE Robotics and Automation Letters, 2022, 7(2): 802-809.

龚朋朋, 李笑. 遥操作工程机器人双目视觉定位 [J]. 机械设计与制造, 2016(9): 120-123.

GONG Pengpeng, LI Xiao. Binocular vision location of Tele-Operated construction robot [J]. Machinery Design Manufacture, 2016(9): 120-123.

张鸿健, 韩捷, 蒋维涛, 等. 柔性水下悬停机器人的液气相变驱动及制造 [J]. 西安交通大学学报, 2021, 55(2): 10-17.

ZHANG Hongjian, HAN Jie, JIANG Weitao, et al. Liquid-gas phase change driving and manufacturing of flexible underwater hovering robot [J]. Journal of Xi'an Jiaotong University, 2021, 55(2): 10-17.

CHEN Haifei, XU Yong. High telepresence control for the relay communication based space teleoperation [J]. Acta Astronautica, 2023, 208: 96-104.

程瑞洲, 黄攀峰, 刘正雄, 等. 一种面向在轨服务的空间遥操作人机交互方法 [J]. 宇航学报, 2021, 42(9): 1187-1196.

CHENG Ruizhou, HUANG Panfeng, LIU Zhengxiong, et al. A human-robot interaction method of on-orbit service-oriented space teleoperation [J]. Journal of Astronautics, 2021, 42(9): 1187-1196.

WHITNEY D, YAMANASHI LEIB A. Ensemble perception [J]. Annual Review of Psychology, 2018, 69(1): 105-129.

KIM G, JANG J, BAEK S, et al. Visual number sense in untrained deep neural networks [J]. Science advances, 2021, 7(1): 6127.

GIBBS J K, GILLIES M, PAN Xueni. A comparison of the effects of haptic and visual feedback on presence in virtual reality [J]. International Journal of Human-Computer Studies, 2022, 157: 102717.

BARUA H B, SAU A, DEV ROYCHOUDHURY R. A perspective on robotic telepresence and teleoperation using cognition: are we there yet? [EB/OL].(2022-03-06)[2022-05-20]. https://arxiv.org/abs/2203.02959.

陶建华, 巫英才, 喻纯, 等. 多模态人机交互综述 [J]. 中国图象图形学报, 2022, 27(6): 1956-1987.

TAO Jianhua, WU Yingcai, YU Chun, et al. A survey on multi-modal human-computer interaction [J]. Journal of Image and Graphics, 2022, 27(6): 1956-1987.

AZOFEIFA J D, NOGUEZ J, RUIZ S, et al. Systematic review of multimodal human-computer interaction [J]. Informatics, 2022, 9(1): 13.

LI Ke, SCHMIDT S, BACHER R, et al. Mixed reality tunneling effects for stereoscopic untethered video-see-through head-mounted displays [C]//2022 IEEE International Symposium on Mixed and Augmented Reality(ISMAR). Piscataway, NJ, USA: IEEE, 2022: 44-53.

COELHO L P, FREITAS I, KAMINSKA D U, et al. Virtual and augmented reality awareness tools for universal design: towards active preventive healthcare [M]//Emerging Advancements for Virtual and Augmented Reality in Healthcare. Hershey, PA, USA: IGI Global, 2022: 11-24.

TORRENCE B, DRESSEL J. Critical review of extended reality applications in aviation [C]//Virtual, Augmented and Mixed Reality: Applications in Education, Aviation and Industry. Cham: Springer International Publishing, 2022: 270-288.

SU Yunpeng, CHEN Xiaoqi, ZHOU T, et al. Mixed reality-enhanced intuitive teleoperation with hybrid virtual fixtures for intelligent robotic welding [J]. Applied Sciences, 2021, 11(23): 11280.

AGUAYO C, EAMES C. Using mixed reality(XR)immersive learning to enhance environmental education [J]. The Journal of Environmental Education, 2023, 54(1): 58-71.

AYYANCHIRA A, MAHFOUD E, WANG Weichao, et al. Toward cross-platform immersive visualization for indoor navigation and collaboration with augmented reality [J]. Journal of Visualization, 2022, 25(6): 1249-1266.

GALLALA A, KUMAR A A, HICHRI B, et al. Digital twin for human-robot interactions by means of industry 4.0 enabling technologies [J]. Sensors, 2022, 22(13): 4950.

占宏, 梁聪垣, 杨辰光. 基于混合现实的机器人遥操作实验平台 [J]. 实验技术与管理, 2021, 38(8): 114-117.

ZHAN Hong, LIANG Congyuan, YANG Chenguang. Robot teleoperation experimental platform based on mixed reality [J]. Experimental Technology and Management, 2021, 38(8): 114-117.[22] LIVATINO S, GUASTELLA D C, MUSCATO G, et al. Intuitive robot teleoperation through multi-sensor informed mixed reality visual aids [J]. IEEE Access, 2021, 9: 25795-25808.

940-941.

LIN T C, KRISHNAN A U, LI Zhi. Comparison of haptic and augmented reality visual cues for assisting tele-manipulation [C]//2022 International Conference on Robotics and Automation(ICRA). Piscataway, NJ, USA: IEEE, 2022: 9309-9316.

PAMPARU C, SCHIPOR O A, DANCU A, et al. SAPIENS in XR: operationalizing interaction-attention in extended reality [J/OL]. Virtual Reality, 2023: 1-17 [2022-08-28]. https://doi.org/10.1007/s10055-023-00776-1.

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

3D图形预测仿真及虚拟夹具的大时延遥操作技术

多工况下离心压气机叶轮-机匣一体联合设计优化

隧道内燃料电池汽车氢气泄漏扩散特性及影响因素

超临界水异常区域热力学特性研究

融合物理信息与深度算子网络的变压器绕组热-流场快速计算方法