Towards Ontology-based interactive / immersive Task and Motion Planning for the assistance to the validation of manipulation tasks under strong spatial constraints in VR

Title: Towards Ontology-based interactive and immersive Task and Motion Planning for the assistance to the validation of manipulation tasks under strong spatial and geometric constraints in Virtual Reality

Keywords: Virtual Reality, assistance to manipulation, interactive and immersive task and motion planning, visuo-haptic guidance, ontologies, Industry 4.0, Silver Economy

Scientific and applicative context

This PhD proposal is funded by the Occitanie French region (call Emergence 2024) and the University of Technology Tarbes Occitanie Pyrénées.

The proposed work lies within the context of assistance to manipulation in Virtual Reality (VR), in the following two applicative frameworks:

1. Society is increasingly recognizing the necessity and emergency to provide services allowing elderly people to continue living at home with a satisfactory autonomy level. Challenges for assisted living include the performance of daily activities in the face of reduced mobility, manual dexterity, and cognitive capacities.
2. On the other hand, in industry, up-to-date products are more and more integrated and the tasks related to their life cycle [1] (assembly, maintenance, etc.) are often complex and to be executed under strong spatial and geometric constraints. In the framework of the Factory of the Future [2] and with the emergence of digital simulation techniques, industrial companies express the need to validate these tasks from design stage on, before building the physical prototypes. This would allow to reduce development time and cost, to detect errors as early as possible, and to target more environment-friendly processes, as less ill-conditioned prototypes would be built.

In order to address these issues, we propose, in this PhD work, the development of new techniques based on the novel and joint usage of human-centred simulation techniques in Virtual Reality (VR), robotics and Artificial Intelligence (AI). The PhD work will benefit from the experience built through former and current multidisciplinary scientific works at the crossroads of VR and robotics at the LGP-ENIT laboratory, in the context of Industry 4.0.

The research activities in VR at the LGP started in order to answer the industrial needs expressed above (validation, from design stage on, of complex industrial scenarios to be executed under strong geometric constraints). When validating such processes through simulation, showing the feasibility of motion is key. Motion planning techniques, developed by the robotics community from the 1980s [3], are largely used. The limitations of these techniques are mainly linked to the complexity of the environment models manipulated, which are traditionally purely geometric; the environment models and the methods for exploring these environments generally do not use information related to the task to be performed. In a complex environment, automatic path planning techniques may lead to high computation time, and may fail or provide solutions of little relevance to the simulated task.

In parallel, the emergence of VR in recent years has made it possible to envisage interactive and immersive simulation.

For over 15 years, the LGP has been developing an original approach based on the joint usage of VR and motion planning techniques; interactive and immersive motion planning (through visuo-haptic guidance) has allowed to improve the state of the art of the assistance to the validation of complex scenarios, and to address control sharing between the motion planning algorithms involved and the operator in the loop [4]. Then, the approach was improved by the involvement of higher abstraction level information (topological and semantic data) than the purely geometric data traditionally involved [5]. Finally, in order to be able to reason at the task level, it is necessary to consider jointly task and motion planning [6] for the assistance to the operator, and to model and use task-related information. Ontologies are a promising tool [7].

The LGP now wants to extend the approach developed by the integration of AI techniques. This new exploratory step will allow, through joint, interactive and immersive ontology-based task and motion planning, to move from the “virtual experience” to the “task- or trade-oriented experience” through a ground-breaking methodological innovation based on the joint and ontology-based involvement of robotics, VR and AI techniques. To the best of our knowledge, the proposed approach has not been investigated in and interactive and immersive context.

Proposed work

The objectives of this PhD work are the following:

1) To model the semantics of the collaboration between task and motion planning by exploring the possibilities of the tools of the cognitive science domain (more specifically ontologies).

The work developed will rely upon the state of the art in the domains of:

• The mdelling of industrial [8] and/or medical manipulation tasks and task planning [9].
• The representatin of concepts associated to the 3D modelling of the environment where the considered tasks take place, and motion planning [7] [10] [11] [12] [13].

2) To study the possibility of representing spatial and geometric constraints in a generic way (a problematic challenge, as the literature offers various - even contradictory - classifications of these constraints [14] [15]) and to reason at the level of these constraints to automatically generate task or trajectory plans.

3) To study novel interaction and/or immersion modalities between the operator in the loop and a “task-motion” assistance module in VR.

The concepts developed will be validated by full-scale experimentation on the VR platform of the LGP-ENIT of two case studies (an industrial manipulation task, and a task relating to home support or physical or cognitive rehabilitation), to be defined during the thesis.

Scientific challenges

The proposed PhD work addresses the following scientific challenges:

• The first challenge relates to the study of strategies for joint Task And Motion Planning (TAMP) [16].
• The second challenge deals with the use of ontologies in the robotics field [17].

These scientific challenges have been emerging in the recent years at the crossroads of the robotics and AI scientific fields. While recent work has demonstrated the necessity to consider jointly task and motion planning [18] [19] and the interest for using ontologies in robotic applications [17] [20] (often to bridge the gap between low level motion planning and high level task planning), the integration of these AI and robotics techniques is still a challenge [16].

• The third scientific challenge concerns the study of TAMP in an interactive and immersive context. To the best of our knowledge, this issue has not been addressed by the scientific community.

Scientific cooperation context

The PhD works will rely upon former research carried on at the LGP-ENIT in interactive and immersive motion planning, AI and distributed planning, in the context of a scientific collaboration between the MAVRICS (robotics and VR) and the PICS (AI) teams of the laboratory.

They will also be undertaken in the context of an emerging scientific cooperation with the Semantic Technologies Laboratory at the University of Toronto [21], which specialises in knowledge representation and automatic reasoning based on the use of ontologies. This collaboration focuses so far on the application of recent developments in the concepts of mereotopology (a theory of first-order logic incorporating the notions of topological relations and parthood) to the generic expression of spatial constraints and the reasoning on them.

Bibliography

[1]        Fillatreau Philippe. "SimProVR : Plateforme de SIMulation Interactive en RV pour les PROcessus de l'Usine du Futur". Présentation lors du 9ème Forum Technique des adhérents Aerospace Valley. Projet lauréat de l'Appel à Idées de Démonstrateurs Technologiques "Usine du Futur" (UF) de la journée du DAS UF du Pôle de Compétitivité Aerospace Valley, Arcachon, Juin 2014.

[2]        Matthew D. Jones, Scott Hutcheson, Jorge D. Camba, Past, present, and future barriers to digital transformation in manufacturing: A review, Journal of Manufacturing Systems, Volume 60, 2021, Pages 936-948, ISSN 0278-6125, https://doi.org/10.1016/j.jmsy.2021.03.006. (https://www.sciencedirect.com/science/article/pii/S0278612521000613).

[3]        J.-C. Latombe. Robot motion planning, vol. 124. Springer Science & Business Media, 2012.

[4]        N. Ladeveze, &al. Interactive path planning for haptic assistance in assembly tasks. Computers & Graphics,  2010.
[5]        S. Cailhol &al. Multi-layer path planning control for the simulation of manipulation tasks: involving semantics and topology. Robotics, Computers and Integrated Manufacturing,  2019.

[6]        J. Akbari &al. Combined heuristic task and motion planning for bi-manual robots. Auton Robot 2019.

[7]        Muhayyuddin &al. Ontological physics-based motion planning for manipulation, in IEEE 20th Conference on Emerging Technologies Factory Automation (ETFA), 2015.

[8]        N. Lhose & al. An ontology for the definition and validation of assembly processes for evolvable assembly systems. Presented at the 6th IEEE Symposium on Assembly and Task Planning: From Nano to Macro Assembly and Manufacturing, (ISATP 2005), Montreal, Canada, 19-21st July, pp. 242-247.

[9]        Rajpathak, Dnyanesh and Motta, Enrico (2004). An Ontological formalization of the planning task. In: International Conference on Formal Ontology in Information Systems, 4-6 Nov 2004, Torino, Italy.

[10]      3D Modeling ontology. http://3dontology.org

[11]      Yingshen Zhao, Arkopaul Sarkar, Linda Elmhadhbi, Mohamed Hedi Karray, Philippe Fillatreau, Bernard Archimede, (2024), "An ontology of 3d environment where a simulated manipulation task takes place (ENVON)", Semantic Web Journal, Vol. 15(2), pp. 613-640, 10.3233/SW-233460, (JCR)

[12]      Alexander Perzylo, Nikhil Somani, Markus Rickert and Alois Knoll. An Ontology for CAD Data and Geometric Constraints as a Link Between Product Models and Semantic Robot Task Descriptions. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany, September 2015.

[13]      R. Gayathri, V. Uma. Ontology based knowledge representation technique, domain modeling languages and planners for robotic path planning: A survey. ICT Express, Volume 4, Issue 2, June 2018, pages 69-74.

[14]      Belouaer L., Bouzid M. and Mouaddib A.. Spatial Knowledge in Planning Language. Int. Conf. on Knowledge Engineering and Ontology Development (KEOD-2011), pp. 71-80.

[15]      Touya, G., Balley, S., Duchêne, C., Jaara, K., Regnauld, N., Gould, N., 2012. "Towards an Ontology of Spatial Relations and Relational Constraints". 15th ICA Workshop on Generalisation and Multiple Representation. ICA, Istanbul, Turkey.

[16]      H. Guo &al. Recent Trends in Task and Motion Planning for Robotics: A Survey. ACM Comput Surv. 2023.

[17]      R. Bernardo &al. A novel framework to improve motion planning of robotic systems through semantic knowledge-based reasoning. Computers and Industrial Engineering. 2023.

[18]      A. M. Wells, N. T. Dantam, A. Shrivastava, L. E. Kavraki, Learning feasibility for task and motion planning in tabletop environments, IEEE Robotics and Automation Letters 4 (2019).

[19]      T. Pan, A. M. Wells, R. Shome, L. E. Kavraki, A general task and motion planning framework for multiple manipulators, in: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE Press, 2021, p. 3168–3174. URL: https://doi.org/10.1109/IROS51168.2021.9636119. doi:10.1109/IROS51168.2021.9636119. 1255–1262.

[20]      A. Olivares-Alarcos, S. Foix, S. Borgo, Guillem Aleny.,, Ocra – an ontology for collaborative robotics and adaptation, Computers in Industry 138 (2022) 103627. URL: https://www.sciencedirect.com/science/article/pii/S0166361522000227. doi:https://doi. org/10.1016/j.compind.2022.103627.

[21]     F. Leoty &al. "Using Mereotopology for Automated Spatial Inference in Task and Motion Planning", The 4th RobOntics Workshop, IEEE RO-MAN 2023.

PhD supervision

The proposed PhD works will be supervised by:

• Professor Bernard ARCHIMEDE (LGPENIT - PICS Research Group)
• Associate Professor Philippe FILLATREAU (LGPENIT – MAVRICS Research Group)

PhD candidate profile

Graduate from Master, research Master, in a University or Engineering School.

The PhD candidate should have part or all of the following skills :

• Technical and scientific domains:
  • Infrmatics, AI (Ontologies, Task Planning)
  • Rbotics (Motion Planning)
• Tools, programming languages, development environments
  • C++,  UML, Visual Studi
  • VR envirnments : Virtools, Unity
• Good English level

Research experience would be appreciated.

Application procedure

The recruitment procedure is open.

Applications to be sent as soon as possible.

Typical PhD start date: 1^st October 2024

Contract duration: 3 years

PhD candidates should send the following information:

• Resume, motivation letter, academic records, description of research experience if any
• English level assessment

Address of the laboratory

Laboratoire Génie de Production

Ecole Nationale d’Ingénieurs de Tarbes

Université de Technologie Tarbes Occitanie Pyrénées

47 Avenue d’Azereix

65016 TARBES Cedex

FRANCE

The LGP is a multidisciplinary laboratory that develops research activities in materials, mechanics, automation, computer science, electrical engineering, robotics and production sciences and techniques in the field of Systems Science and Engineering. Given its affiliation with UTTOP, its mission is to conduct and develop research in areas related to the training of ENIT general engineers and thus promote the link between training, research and technology transfer. The LGP is also part of the Université Fédérale de Toulouse Midi Pyrénées and shares a number of activities with the Institut National Polytechnique de Toulouse (Toulouse INP)

The research is organized most often in close connection with real problems of the socio-economic world by following a unit of view of the product/process type throughout the life cycle of the product, from design to dismantling.

The LGP relies on remarkable equipment, consistent with the needs of companies and the profile of engineers trained at ENIT.

Following its restructuring, in July 2021, two scientific departments will group the teacher-researchers and researchers of UTTOP according to two major themes:

    Scientific Department Mechanics-Materials-Processes

    Scientific Department Systems

The main activities of the department are related to complex and dynamic systems. A system is a set of interacting components whose individual behaviors are different from the behavior of the whole system. The developments aim at establishing models to understand, predict and act on the behavior of these systems. The department relies on five research groups oriented around the sciences of automation, robotics, mechatronics, electronics and electrical conversion systems, artificial intelligence and industrial engineering.

PhD candidate profile

Graduate from Master, research Master, in a University or Engineering School.

The PhD candidate should have part or all of the following skills :

• Technical and scientific domains:
  • Informatics, AI (Ontologies, Task Planning)
  • Rbotics (Motion Planning)
• Tools, programming languages, development environments
  • C++,  UML, Visual Studi
  • VR envirnments : Virtools, Unity
• Good English level

Research experience would be appreciated.