As a precious art in China, shadow play has been spreading for more than 2000 years. But with the development of society, shadow play is no longer popular and attractive to the younger generation. Therefore, digital shadow play plays a very important role in the protection and dissemination of shadow play. In spite of this, there are still some deficiencies in this way of spreading shadow play culture today. It is usually used by placing the shadow puppet model in a designated position in a pre-built place and making adjustments. Of course, this method takes a lot of time, and compared with traditional shadow play, this kind of shadow play is not flexible and vivid. This paper proposes to use motion capture technology to make digital shadow play. The method adopted in this paper is based on the open source code of AlphaPose, Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time. First of all, the action of the characters is captured by the live action and high-definition camera, and finally the shadow puppet animation is generated according to the captured action.

Zhengyu Lin, Kaiyi Tong, Tianyi Yang. Virtual Shadow Puppet Play Generation Based on Alphapose. Frontiers in Art Research (2023) Vol. 5, Issue 16: 73-77. https://doi.org/10.25236/FAR.2023.051613.

[1] Gkioxari, G., Hariharan, B., Girshick, R., & Malik, J. (2014). Using k-poselets for detecting people and localizing their keypoints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3582-3589). 

[2] Pishchulin, L., Insafutdinov, E., Tang, S., Andres, B., Andriluka, M., Gehler, P. V., & Schiele, B. (2016). Deepcut: Joint subset partition and labeling for multi person pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4929-4937).

[3] Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., & Schiele, B. (2016). Deepercut: A deeper, stronger, and faster multi-person pose estimation model. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14 (pp. 34-50). Springer International Publishing.

[4] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).

[5] Cao, Z., Simon, T., Wei, S. E., & Sheikh, Y. (2017). Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7291-7299).

[6] Newell, A., Huang, Z., & Deng, J. (2017). Associative embedding: End-to-end learning for joint detection and grouping. Advances in neural information processing systems, 30. 

[7] Roth, D., Lugrin, J. L., Büser, J., Bente, G., Fuhrmann, A., & Latoschik, M. E. (2016, March). A simplified inverse kinematic approach for embodied VR applications. In 2016 IEEE Virtual Reality (VR) (pp. 275-276). IEEE.

[8] Villegas, R., Yang, J., Ceylan, D., & Lee, H. (2018). Neural kinematic networks for unsupervised motion retargetting. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8639-8648). 

[9] H.-S. Fang, S. Xie, Y.-W. Tai, and C. Lu, “Rmpe: Regional multiperson pose estimation,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2334–2343. 

[10] Xiao, B., Wu, H., & Wei, Y. (2018). Simple baselines for human pose estimation and tracking. In Proceedings of the European conference on computer vision (ECCV) (pp. 466-481).

[11] Sun, K., Xiao, B., Liu, D., & Wang, J. (2019). Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5693-5703).

[12] Bertasius, G., Feichtenhofer, C., Tran, D., Shi, J., & Torresani, L. (2019). Learning temporal pose estimation from sparsely-labeled videos. Advances in neural information processing systems, 32. 

[13] Jin, S., Xu, L., Xu, J., Wang, C., Liu, W., Qian, C., ... & Luo, P. (2020). Whole-body human pose estimation in the wild. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16 (pp. 196-214). Springer International Publishing.

[14] Yang, Y., Ren, Z., Li, H., Zhou, C., Wang, X., & Hua, G. (2021). Learning dynamics via graph neural networks for human pose estimation and tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8074-8084).

[15] Zhang, J., Zhu, Z., Zou, W., Li, P., Li, Y., Su, H., & Huang, G. (2019). Fastpose: Towards real-time pose estimation and tracking via scale-normalized multi-task networks. arXiv preprint arXiv:1908.05593. 

[16] Liu, Z., Zhou, L., Leung, H., & Shum, H. P. (2015). Kinect posture reconstruction based on a local mixture of gaussian process models. IEEE transactions on visualization and computer graphics, 22(11), 2437-2450. 

[17] Mousas, C. (2018, March). Performance-driven dance motion control of a virtual partner character. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (pp. 57-64). IEEE. 

[18] Fang, A. C., & Pollard, N. S. (2003). Efficient synthesis of physically valid human motion. Acm transactions on graphics (tog), 22(3), 417-426.

[19] Lee, J., Chai, J., Reitsma, P. S., Hodgins, J. K., & Pollard, N. S. (2002, July). Interactive control of avatars animated with human motion data. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques (pp. 491-500). 

[20] Grochow, K., Martin, S. L., Hertzmann, A., & Popović, Z. (2004). Style-based inverse kinematics. In ACM SIGGRAPH 2004 Papers (pp. 522-531).

[21] Lv, X., Chai, J., & Xia, S. (2016). Data-driven inverse dynamics for human motion. ACM Transactions on Graphics (TOG), 35(6), 1-12.