Internship FUSION : Development of advanced interaction for remote control and mission authoring of autonomous robots through XR digital twin (XRDT)

Laboratory presentation

CESI LINEACT (UR 7527), Laboratory for Digital Innovation for Businesses and Learning to Support the Competitiveness of Territories, anticipates and accompanies the technological mutations of sectors and services related to industry and construction. The historical proximity of CESI to companies is a determining factor in our research activities. It has led us to focus our efforts on applied research close to companies and in partnership with them. A human-centered approach coupled with the use of technologies, as well as territorial networking and links with training, has enabled the construction of cross-cutting research; it puts humans, their needs, and their uses at the center of its issues and addresses the technological angle through these contributions.

Its research is organized according to two interdisciplinary scientific teams and several application areas.

• Team 1 "Learning and Innovating" mainly concerns Cognitive Sciences, Social Sciences, and Management Sciences, Training Techniques, and those of Innovation. The main scientific objectives are the understanding of the effects of the environment, and, more particularly, of situations instrumented by technical objects (platforms, prototyping workshops, immersive systems...) on learning, creativity, and innovation processes.

• Team 2 "Engineering and Digital Tools" mainly concerns Digital Sciences and Engineering. The main scientific objectives focus on modeling, simulation, optimization, and data analysis of cyber-physical systems. Research also focuses on decision-support tools and on the study of human-system interactions, particularly through digital twins coupled with virtual or augmented environments.

These two teams develop and cross their research in application areas such as

• Industry 5.0,

• Construction 4.0 and Sustainable City,

• Digital Services.

Areas supported by research platforms, mainly the one in Rouen dedicated to Factory 5.0 and the one in Nanterre dedicated to Factory 5.0 and Construction 4.0.

Research proposal

Context

The explosion of robotics and its uses has led robot manufacturers to develop their own programming software to equip their machines with intelligence capabilities. However, these programming environments suffer from certain limitations: they are very often proprietary and therefore not open to access the lower layers of programming. To overcome these constraints, robotic middleware appeared in the early 2000s. The concept is simple: the robotic middleware is a software overlay, installable on an existing operating system (Linux or Windows type), which contains generic tools and libraries to facilitate programming of robot intelligence capabilities (Sahni, Cao and Jiang, 2019). Despite these middlewares, programming of robotic missions is still not very accessible to the uninitiated. This is the framework in which this project positions FUSION; the main objective is to democratize the use of robotics, especially with the introduction of XR for the design of robotic missions via the digital twin.

In this context and within the framework of its research on digital twins, virtual reality and robotics, the CESI LINEACT laboratory conducts research activities on human-systems interfaces allowing to edit business procedures, training content or virtual reality robotic missions (Richard et al., 2021; Havard et al., 2023). In addition, the first works have made it possible to architect the NEURONE framework allowing remote control.

Objectives

The objectives of this internship focus first on integrating and comparing existing interactions within the FUSION project. Then, the evolution of the work in collaboration with CONSCIENCE ROBOTICS will require to design and develop new functionalities in XR such as: integration of LIDAR 3D Scan mapping and visualization, definition of the mission of recognition and reading elements of a remote scene coupling robotic architecture dedicated to one API with our intern architecture. The proposed developments will be integrated into the DTAPI Framework, implemented at the LINEACT laboratory, and connected to the API implemented in the Conscience Robotics environment. This work will be based on the future industry platform present at the CESI Rouen campus and its digital twin. A proof of concept integrating one of the robotic components of the platform (mobile robot or robotic arm) or micro learning factory will be studied during the internship.