Internship FUSION : Integration of XR-based Shared-Control Teleoperation system on a Real Industrial Robot

Laboratory presentation

CESI LINEACT (UR 7527), Laboratory for Digital Innovation for Businesses and Learning to Support the Competitiveness of Territories, anticipates and accompanies the technological mutations of sectors and services related to industry and construction. The historical proximity of CESI to companies is a determining factor in our research activities. It has led us to focus our efforts on applied research close to companies and in partnership with them. A human-centered approach coupled with the use of technologies, as well as territorial networking and links with training, has enabled the construction of cross-cutting research; it puts humans, their needs, and their uses at the center of its issues and addresses the technological angle through these contributions.

Its research is organized according to two interdisciplinary scientific teams and several application areas.

• Team 1 "Learning and Innovating" mainly concerns Cognitive Sciences, Social Sciences, and Management Sciences, Training Techniques, and those of Innovation. The main scientific objectives are the understanding of the effects of the environment, and, more particularly, of situations instrumented by technical objects (platforms, prototyping workshops, immersive systems...) on learning, creativity, and innovation processes.

• Team 2 "Engineering and Digital Tools" mainly concerns Digital Sciences and Engineering. The main scientific objectives focus on modeling, simulation, optimization, and data analysis of cyber-physical systems. Research also focuses on decision-support tools and on the study of human-system interactions, particularly through digital twins coupled with virtual or augmented environments.

These two teams develop and cross their research in application areas such as

• Industry 5.0,

• Construction 4.0 and Sustainable City,

• Digital Services.

Areas supported by research platforms, mainly the one in Rouen dedicated to Factory 5.0 and the one in Nanterre dedicated to Factory 5.0 and Construction 4.0.

Abstract

This internship is part of the FUSION project, which aims to democratize robot programming and operation by leveraging Extended Reality (XR) and digital twins.

While industrial robots are increasingly deployed in Industry 5.0 contexts, existing proprietary programming environments often limit access to low-level control and remain inaccessible to non-expert users. XR-enabled teleoperation provides promising solutions to overcome these barriers by offering immersive interfaces and improved situational awareness for remote manipulation tasks.

However, fully manual teleoperation can increase cognitive workload and reduce precision in constrained environments. Shared-control strategies address this challenge by combining operator intent with robotic autonomy, enabling the robot to assist motion execution while keeping the human in the control loop. The main objective of this internship is to integrate an XR-based shared-control teleoperation system on a real industrial robot, transitioning from digital-twin simulation to safe physical execution.

The work will include implementing assistance strategies such as virtual fixtures (e.g., guidance tubes), attractive/repulsive fields, and workspace safety constraints. Camera-based perception will be integrated into XR to ensure visibility of the real scene and robot limitations. Finally, the intern will define an industrial use case (e.g., pick-and-place with obstacles, part insertion, trajectory following), design an experimental protocol, and conduct user tests to evaluate performance and workload.

Research Work

The explosion of robotics and its applications in industry 5.0 has led robot manufacturers to develop their own programming software to equip their robots with intelligent capabilities. However, these programming environments have certain limitations: they are often proprietary and do not provide access to low-level programming layers. To address these constraints, robotic middleware emerged in the early 2000s. The concept is simple: robotic middleware is an additional software layer that can be installed on an existing operating system (such as Linux or Windows), which natively contains generic tools and libraries to facilitate the programming of robot intelligence capabilities (Sahni, Cao and Jiang, 2019). Despite these middlewares, programming robotic missions remains largely inaccessible to non-experts. This is the context in which the FUSION project is positioned, with its main objective being to democratize the use of robotics, particularly through the introduction of XR for the design of robotic missions via digital twins.

The development of the Tactile Internet (IEEE 1918.1) aims to create networks enabling real-time remote access, perception, manipulation, or control of real or virtual objects or processes by humans or machines. This is crucial for applications requiring high precision and low latency. In highly constrained environments where physical presence is impractical and where robots are sent instead, teleoperation becomes necessary. Digital Twins (DT)—virtual replicas of physical systems—combined with Extended Reality (XR) can be an effective solution to enhance and assist human-robot interactions (Havard et al., 2023).

Teleoperation assisted through XR is providing innovative methods for humans to interact with robots. Research demonstrates the efficiency of XR interfaces in teleoperating industrial robot manipulators. Moreover, combining Extended Reality with Digital Twin (DTXR) techniques in multi-agents systems enable intuitive human-in-the-loop control to improve efficiency in industrial environments. XR addresses issues like distance and robots' limitations perception, improving the immersive experience for operators engaged in remote manipulation tasks. This technology holds potential applications across social and high-risk environments, offering robust solutions for complex operational challenges.

However, robots also have autonomous capabilities which allows them to freely execute their missions and thus alleviates teleoperator’s cognitive workload while doing it. But, in highly constraint environment and fine manipulation, teleoperator must be in control even partially of the robot. This approach, called shared-control leverages the strengths of both human intuition and machine precision, enhancing task efficiency and flexibility (Li et al., 2023; Y. Zhu et al.,2023; Luo et al., 2024 ).

Key challenges in DTXR-assisted teleoperation include issues related to perception of robotic autonomy during fine manipulation tasks and the need to ensure workspace visibility and limitations inoperational contexts (Pryor et al.,2023).

Addressing these challenges is crucial for optimizing operator load, enhancing task performance, and improving overall user experience in DTXR-assisted teleoperation systems.

Work Program

This internship aims to design, implement, and validate XR interfaces for shared-control teleoperation of industrial robots linked to digital twins. The goal is to connect a virtual reality environment to a real robotic system, enabling the operator to control the robot partially while receiving intuitive feedback about its capabilities, and limitations. The work is expected to enhance efficiency, safety, and operator comfort in teleoperation, particularly in constrained industrial contexts.

The internship specifically targets the transition from digital-twin simulation to control of a physical robot, while maintaining shared-control assistance strategies and ensuring compliance with safety constraints.

• Study and Familiarization (M1)

• Study of shared control principles.

• Discovery of the CESI LINEACT middleware and tools.

• Familiarization with the digital twin and XR teleoperation system.

2. Development of XR System with Shared Control (M1–M2)

• Coupling the XR teleoperation interface with the real robot using shared-control concepts:

• Tube virtual fixtures.

• Attractive and repulsive fields.

• Safety fixtures and workspace limits.

• Integrate camera views in XR to visualize the real environment

3. Design of Industrial Use Case and Experimental Protocol (M2–M3–M4)

• Definition of an industrial use case including unit tasks (with and without shared control) :

• Pick and Place with Obstacles: Use DTXR to guide UR robots through pick-and-place tasks involving obstacles.

• Part Insertion: Enable precise part insertion using DTXR to visualize accurate alignment and force application.

• Trajectory Following: Support accurate trajectory following by providing real-time visual feedback on robot paths and capabilities within DTXR.

• Design of the corresponding experimental protocol.

Depending on progress:

• Extend pick-and-place to multiple objects.

• Implement intention-prediction algorithms.

• Generate robot trajectories based on operator intention.

4. Experimentation and Analysis (M4–M5)

• Conduction of user tests.

• Collection and analysis of results.

• Comparison between control conditions (direct control / shared control).

5. Finalization and Reporting (M6)

• Consolidation of developments.

• Writing of the internship report and presentation.