A Mixed Reality Visual Augmentation and Interaction System in Teleoperation Scenarios

SHI Yueyang; LI Xiaoling; WANG Long; CHEN Zhangyi; ZHENG Shuai

doi:10.7652/xjtuxb202310003

您当前的位置：

首页 >

文章列表页 >

A Mixed Reality Visual Augmentation and Interaction System in Teleoperation Scenarios

更新时间：2025-05-08

- A Mixed Reality Visual Augmentation and Interaction System in Teleoperation Scenarios
- Vol. 57, Issue 10, Pages: 20-29(2023)
- 作者机构：
  
  1. 西安交通大学机械工程学院,西安,710049
  2. 西安交通大学电子与信息学部,西安,710049
- 作者简介：
- 基金信息：
- DOI：10.7652/xjtuxb202310003
  CLC： TP391.7
- Online First：10 October 2023，
  
  Published：2023
- 稿件说明：
移动端阅览
SHI Yueyang, LI Xiaoling, WANG Long, et al. A Mixed Reality Visual Augmentation and Interaction System in Teleoperation Scenarios[J]. 2023, 57(10): 20-29.
DOI：

SHI Yueyang, LI Xiaoling, WANG Long, et al. A Mixed Reality Visual Augmentation and Interaction System in Teleoperation Scenarios[J]. 2023, 57(10): 20-29. DOI： 10.7652/xjtuxb202310003.

摘要

针对遥操作机器人系统在高风险任务中近端交互临场感差、操作者难以获得真实体验感的问题

提出了一种适用于遥操作场景的混合现实视觉增强交互系统。首先

针对远程场景传输和机械臂的远程控制这两个重要环节

通过分析遥操作交互任务中的目标需求

从人体视觉感知与认知特性的角度构建并设计了遥操作场景下的视觉增强交互系统框架; 然后根据系统功能需求

研究了混合现实场景下的远程视景实时呈现技术

设计了混合现实场景下的机械臂交互控制策略

实现了混合现实全息对象与真实物理场景相映射的虚实融合效果

并以此搭建了遥操作场景下的混合现实视觉增强交互系统

实现了混合现实终端的可视化与系统端到端的网络通信。实验结果表明:与传统的二维系统相比

该交互系统所采用视觉增强形式的任务完成时间减少了17.3%

准确度提高了7%。证明该系统具有较好的应用前景。

Abstract

To address the challenges of poor situational awareness and lack of realistic experience for operators in high-risk teleoperation tasks faced by robotic systems

a mixed reality visual augmentation and interaction system applicable to teleoperation was introduced. Firstly

the system framework was designed from the perspective of human visual perception and cognitive characteristics and based on requirements for teleoperation interaction tasks

with a specific focus on remote scene transmission and robotic arm control in teleoperation scenarios. Secondly

according to the system's functional requirements

extensive research was conducted on real-time rendering techniques in mixed reality environments

and an interactive control strategy of the robotic arm was developed for mixed reality environments. This effort enabled a fusion effect between virtual and physical scenes by mapping holographic objects to the real-world environment. This supports the establishment of the mixed reality visual augmentation and interaction system that allows operators to visualize the mixed reality terminal and facilitates seamless end-to-end network communication. Experimental results show that the interaction system has better performance than a traditional system

as manifested by a remarkable 17.3% reduction in task completion time and a notable 7% improvement in accuracy. These findings suggest the promising application prospects of the system.

关键词

Keywords

references

张宇, 李宇翔, 张贺飞, 等. 撼地者: 基于遥操作与自主导航的移动救援机器人 [J]. 中国科学技术大学学报, 2023, 53(1): 37-49, 71-72.

ZHANG Yu, LI Yuxiang, ZHANG Hefei, et al. Earthshaker: a mobile rescue robot for emergencies and disasters through teleoperation and autonomous navigation [J]. Journal of University of Science and Technology of China, 2023, 53(1): 37-49, 71-72.

曾建军. 排爆机器人遥操作性能研究 [D]. 上海: 上海交通大学, 2010.

SUN Zeyuan, YANG Hong, MA Yifan, et al. BIT-DMR: a humanoid dual-arm mobile robot for complex rescue operations [J]. IEEE Robotics and Automation Letters, 2022, 7(2): 802-809.

龚朋朋, 李笑. 遥操作工程机器人双目视觉定位 [J]. 机械设计与制造, 2016(9): 120-123.

GONG Pengpeng, LI Xiao. Binocular vision location of Tele-Operated construction robot [J]. Machinery Design Manufacture, 2016(9): 120-123.

张鸿健, 韩捷, 蒋维涛, 等. 柔性水下悬停机器人的液气相变驱动及制造 [J]. 西安交通大学学报, 2021, 55(2): 10-17.

ZHANG Hongjian, HAN Jie, JIANG Weitao, et al. Liquid-gas phase change driving and manufacturing of flexible underwater hovering robot [J]. Journal of Xi'an Jiaotong University, 2021, 55(2): 10-17.

CHEN Haifei, XU Yong. High telepresence control for the relay communication based space teleoperation [J]. Acta Astronautica, 2023, 208: 96-104.

程瑞洲, 黄攀峰, 刘正雄, 等. 一种面向在轨服务的空间遥操作人机交互方法 [J]. 宇航学报, 2021, 42(9): 1187-1196.

CHENG Ruizhou, HUANG Panfeng, LIU Zhengxiong, et al. A human-robot interaction method of on-orbit service-oriented space teleoperation [J]. Journal of Astronautics, 2021, 42(9): 1187-1196.

WHITNEY D, YAMANASHI LEIB A. Ensemble perception [J]. Annual Review of Psychology, 2018, 69(1): 105-129.

KIM G, JANG J, BAEK S, et al. Visual number sense in untrained deep neural networks [J]. Science advances, 2021, 7(1): 6127.

GIBBS J K, GILLIES M, PAN Xueni. A comparison of the effects of haptic and visual feedback on presence in virtual reality [J]. International Journal of Human-Computer Studies, 2022, 157: 102717.

BARUA H B, SAU A, DEV ROYCHOUDHURY R. A perspective on robotic telepresence and teleoperation using cognition: are we there yet? [EB/OL].(2022-03-06)[2022-05-20]. https://arxiv.org/abs/2203.02959.

陶建华, 巫英才, 喻纯, 等. 多模态人机交互综述 [J]. 中国图象图形学报, 2022, 27(6): 1956-1987.

TAO Jianhua, WU Yingcai, YU Chun, et al. A survey on multi-modal human-computer interaction [J]. Journal of Image and Graphics, 2022, 27(6): 1956-1987.

AZOFEIFA J D, NOGUEZ J, RUIZ S, et al. Systematic review of multimodal human-computer interaction [J]. Informatics, 2022, 9(1): 13.

LI Ke, SCHMIDT S, BACHER R, et al. Mixed reality tunneling effects for stereoscopic untethered video-see-through head-mounted displays [C]//2022 IEEE International Symposium on Mixed and Augmented Reality(ISMAR). Piscataway, NJ, USA: IEEE, 2022: 44-53.

COELHO L P, FREITAS I, KAMINSKA D U, et al. Virtual and augmented reality awareness tools for universal design: towards active preventive healthcare [M]//Emerging Advancements for Virtual and Augmented Reality in Healthcare. Hershey, PA, USA: IGI Global, 2022: 11-24.

TORRENCE B, DRESSEL J. Critical review of extended reality applications in aviation [C]//Virtual, Augmented and Mixed Reality: Applications in Education, Aviation and Industry. Cham: Springer International Publishing, 2022: 270-288.

SU Yunpeng, CHEN Xiaoqi, ZHOU T, et al. Mixed reality-enhanced intuitive teleoperation with hybrid virtual fixtures for intelligent robotic welding [J]. Applied Sciences, 2021, 11(23): 11280.

AGUAYO C, EAMES C. Using mixed reality(XR)immersive learning to enhance environmental education [J]. The Journal of Environmental Education, 2023, 54(1): 58-71.

AYYANCHIRA A, MAHFOUD E, WANG Weichao, et al. Toward cross-platform immersive visualization for indoor navigation and collaboration with augmented reality [J]. Journal of Visualization, 2022, 25(6): 1249-1266.

GALLALA A, KUMAR A A, HICHRI B, et al. Digital twin for human-robot interactions by means of industry 4.0 enabling technologies [J]. Sensors, 2022, 22(13): 4950.

占宏, 梁聪垣, 杨辰光. 基于混合现实的机器人遥操作实验平台 [J]. 实验技术与管理, 2021, 38(8): 114-117.

ZHAN Hong, LIANG Congyuan, YANG Chenguang. Robot teleoperation experimental platform based on mixed reality [J]. Experimental Technology and Management, 2021, 38(8): 114-117.[22] LIVATINO S, GUASTELLA D C, MUSCATO G, et al. Intuitive robot teleoperation through multi-sensor informed mixed reality visual aids [J]. IEEE Access, 2021, 9: 25795-25808.

940-941.

LIN T C, KRISHNAN A U, LI Zhi. Comparison of haptic and augmented reality visual cues for assisting tele-manipulation [C]//2022 International Conference on Robotics and Automation(ICRA). Piscataway, NJ, USA: IEEE, 2022: 9309-9316.

PAMPARU C, SCHIPOR O A, DANCU A, et al. SAPIENS in XR: operationalizing interaction-attention in extended reality [J/OL]. Virtual Reality, 2023: 1-17 [2022-08-28]. https://doi.org/10.1007/s10055-023-00776-1.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Teleoperation with Large Time Delay Based on 3D Graphic Predictive Simulation and Virtual Fixture

Integrated and Joint Design Optimization of Centrifugal Compressor Impeller and Casing Treatment at Multiple Working Points

I/O Performance Modeling and Prediction Method for Large-Scale Parallel Applications Using Machine Learning

Energy Efficiency Optimization of the Cryogenic Air Separation Process for Electronic-Grade Ultra Pure Nitrogen

Turbulent Kinetic Energy Dissipation Mechanism of Supercritical Carbon Dioxide Leakage Flow in a Scallop Bionic Seal Cavity

Related Author

蒋再男 1

刘宏 1

谢宗武 1

赵京东 1

Related Institution

School of Mechatronic Engineering, Harbin Institute of Technology

⁰