Deep learning for 3d human motion capture

3D motion capture (3D MoCap) is a critical technology used in industries such as animation, gaming, sports science, healthcare, and robotics to accurately track and reconstruct human motion. Traditional marker-based systems, while precise, are expensive and require controlled environments, making them inaccessible for many applications. Deep learning offers a transformative solution by enabling markerless motion capture using RGB cameras, dramatically reducing costs and setup complexity. By leveraging neural networks, we can achieve accurate pose estimation, reconstruct detailed 3D human meshes, and even capture subtle movements like soft-tissue dynamics. This makes deep learning-based 3D MoCap a powerful tool for real-time applications, from creating immersive AR/VR experiences to enhancing biomechanical analysis in healthcare.

In this talk, I will review state-of-the-art methods such as HMR (Human Mesh Recovery), SPIN, and VIBE, which combine pose estimation and human mesh reconstruction using advanced architectures like graph neural networks and transformers. Despite their impressive results, these methods face challenges such as handling occlusions, generalizing across diverse body shapes, and achieving real-time performance. My research aims to address these gaps by developing lightweight and efficient models capable of real-time markerless 3D MoCap. Specifically, I will focus on improving soft-tissue dynamics, enhancing model robustness for diverse scenarios, and reducing dependency on labeled data with semi-supervised learning. My contributions aim to push the boundaries of what is possible with deep learning in 3D motion capture, paving the way for broader adoption in dynamic and real-world applications.