Using GANs to Synthesize Videos of Physiotherapy Rehab Exercises from Motion Capture Data

---

Proposed by Richard Blythman
Email: blythmar(at)tcd.ie

Motion capture systems are often used to capture ground truth 3D pose labels in combination with RGB videos for training deep computer vision models for human pose estimation. To date, these datasets have tended to capture generic human motion. In contrast, MoCap data of motions specific to applications such as physiotherapy rehabilitation exercises are common in the field of biomechanics, although simultaneous multiview RGB video footage is rare. The aim of this project is synthesise video sequences of rehab exercies from image-less MoCap data using generative adversarial networks (GANs).

Recently, a dataset of physiotherapy rehabilitation exercises has been acquired in a motion capture laboratory [1], although the single-view RGB video is not shared. Several GAN models for pose-guided image generation have been developed for the application of fashion dataset synthesis [2, 3]. The generation of coherent videos from pose sequences [4] is relatively unexplored.

The initial focus of the project will be on visualisation of the MoCap data (possibly using some computer graphics engines) and establishing the 3D geometry of the problem (e.g. by choosing several camera positions and parameters and inspecting the projected 2D pose sequence on the image plane). The indiviudal frames of the 2D pose sequences will be passed as input to a number of publicly-available GANs, giving the student practical experience with the use of pretrained deep learning models. The student will then re-train a variety of human pose models on the synthetic data before testing on real data, and assess any drop in performance. The student may also be in a position to explore other areas, such as the addition of contraints to ensure temporal coherence or some ideas of their own.

References:

[1] Vakanski, A., Jun, H. P., Paul, D., & Baker, R. (2018). A data set of human body movements for physical rehabilitation exercises. Data, 3(1), 2.

[2] https://github.com/charliememory/Pose-Guided-Person-Image-Generation

[3] https://github.com/tengteng95/Pose-Transfer

[4] Yang, C., Wang, Z., Zhu, X., Huang, C., Shi, J., & Lin, D. (2018). Pose guided human video generation. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 201-216).