Embodied robot teleoperation based on high-fidelity visual-haptic simulator: Pipe-fitting example

Robot teleoperation, a control method allowing human operators to manipulate robotic systems remotely, has become increasingly popular in construction applications. A significant challenge is the disconnection between the robot sensor data and the human operator's sensory processes, creating a sensorimotor mismatch in motor-intensive activities. This disconnection is particularly challenging in motor-intensive activities that require accurate perception and response. Researchers have started investigating haptic interactions to enhance the control feedback loop, including simulating contacts, motions, and tactile input. However, although current methodologies have advanced the field, they often focused on certain aspects and could be further expanded to provide a more comprehensive simulation of the physical interaction that occurs in typical construction operations. This study designs and tests a comprehensive high-fidelity embodied teleoperation method that simulates complete real-world physical processes via the physics engine. The proposed method captures all categories of physical interaction in typical motor-intensive construction tasks, including weight, texture, inertia, impact, balance, rotation, and spring. A human-subject experiment shows that the proposed method substantially improves performance and human functions in a teleoperated pipe-fitting task. The results indicate that the proposed multisensory augmentation method significantly enhances performance and user experience, offering valuable insights for designing innovative robot teleoperation systems for future construction applications.