Telepresence augmentation for visual and haptic guided immersive teleoperation of industrial manipulator

Fanghao Huang; Xiao Yang; Tuoyu Yan; Zheng Chen

doi:10.1016/j.isatra.2024.05.003

Telepresence augmentation for visual and haptic guided immersive teleoperation of industrial manipulator

ISA Trans. 2024 May 8:S0019-0578(24)00206-4. doi: 10.1016/j.isatra.2024.05.003. Online ahead of print.

Authors

Fanghao Huang¹, Xiao Yang², Tuoyu Yan³, Zheng Chen⁴

Affiliations

¹ State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou, 310027, China; Key Laboratory of Intelligent Robot for Operation and Maintenance of Zhejiang Province, Hangzhou, 311121, China; Ocean College, Zhejiang University, Zhoushan, 316021, China.
² State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou, 310027, China; Ocean College, Zhejiang University, Zhoushan, 316021, China.
³ Ocean College, Zhejiang University, Zhoushan, 316021, China.
⁴ State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou, 310027, China; Key Laboratory of Intelligent Robot for Operation and Maintenance of Zhejiang Province, Hangzhou, 311121, China; Ocean College, Zhejiang University, Zhoushan, 316021, China. Electronic address: zheng_chen@zju.edu.cn.

PMID: 38749885
DOI: 10.1016/j.isatra.2024.05.003

Abstract

Teleoperation under human guidance has become an effective solution to extend human's reach in various environments. However, the teleoperation system still faces challenges of insufficient sense of both visual and haptic feedback from remote environments, which results in the inadequate guidance for the operator. In this paper, a visual/haptic integrated perception and reconstruction system (VHI-PRS) is developed to provide the operator with 3D visual information and effective haptic guidance. Specifically, a visual-based telepresence augmentation method is proposed to provide the operator with virtual-reality combined visual telepresence, where the real point cloud model is directly superimposed on virtual manipulator to avoid the time-consuming process of mesh model rendering. With the utilization of visual information, a haptic-based telepresence augmentation method is proposed to provide the operator with comprehensive force feedback, including the virtual guiding force, virtual repulsive force and real-time interactive force, which greatly helps reduce the workload of operator. Finally, a user study on grab-place task is carried out to verify the effectiveness of proposed system.

Keywords: Fusion haptic force guidance; Point cloud model; Telepresence augmentation; Visual and haptic integration.