Physics aware Human Action Recognition from Monocular Videos (PhARMov)

While current methods have shown promising progress in estimating 3D human motion from monocular videos (video captured using a single camera from a single viewpoint), their motion estimates are often physically. Human body kinematics refers to studying and analyzing human movement without considering the forces or torques that cause it. Kinematics focuses on describing motion in terms of measurable parameters such as position, velocity, acceleration, angles, and trajectories of body parts (joint angles based on observed or measured motion data; e.g. tracking how the knee joint moves during running, i.e. trajectories of joints and body postures). Furthermore, the estimation of kinematics highly depends on the correct detection of the human skeleton from the image where the detection of landmarks plays a pivotal role in the construction of the skeleton. Firstly, we aim to enhance the accuracy of landmark detection by utilizing human body kinematics, such as positions, velocities, accelerations, and joint angles, guided by established rigid body equations from physics. Secondly, we want to improve the performance of estimating 3D human motion by combining kinematics with dynamics. Since kinematics itself doesn’t involve force analysis (which belongs to dynamics), we would like to propose techniques that can estimate forces based on the observed human motions (kinematics). For example, if we know the trajectory and speed of a leg during a kick (kinematics) then we can infer the muscular forces needed to produce that motion using the existing mathematical model of bio-mechanics and physics. The underlying idea is to incorporate physics principles governing human motions where we would like to build a physics-based body representation and contact force model (to capture the physical properties of the human body and the forces it experiences). Thirdly, our focus is on leveraging “Active Learning” techniques to minimize the annotation costs for new real-time video frames. Specifically, we aim to develop innovative methods for selecting the optimal set of frames that can most effectively improve the model's training performance. Fourthly, we intend to deploy our model or algorithm on edge devices (e.g., NVIDIA Jetons Nano, Google Coral, Raspberry Pi, etc.) equipped with camera systems. To achieve this, we aim to develop compact, efficient, and task-specific algorithms (e.g. detecting abnormal falls of isolated elderly persons, specific actions like ball pass, off-side, shooting goal in soccer, etc. ) capable of performing real-time inference.

The FOX research group is part of the CRIStAL laboratory (University of Lille, CNRS), located in Lille, France. We focus on video analysis for human behavior understanding. Specifically, we develop spatio-temporal models of motions for tasks such as abnormal event detection, emotion recognition, and face alignment. Our work is published in major journals (Pattern Recognition, IEEE Trans. on Affective Computing) and conferences (WACV, IJCNN).

https://www.cristal.univ-lille.fr/FOX/

Candidates must hold a Master degree in Computer Science, Statistics, Applied Mathematics or a related field. Experience in one or more of the following is a plus:

•    image processing, computer vision;

•    machine learning;

•    research methodology (literature review, experimentation…).

Candidates should have the following skills:

•    good proficiency in English, both spoken and written;

•    scientific writing;

•    programming (experience in C++ is a plus, but not mandatory).