Similar articles

Abstract: The deformability and high degree of freedom of mollusks bring challenges in mathematical modeling and synthesis of motions. Traditional analytical and statistical models are limited by either rigid skeleton assumptions or model capacity, and have difficulty in generating realistic and multi-pattern mollusk motions. In this work, we present a large-scale dynamic pose dataset of Drosophila larvae and propose a motion synthesis model named Path2Pose to generate a pose sequence given the initial poses and the subsequent guiding path. The Path2Pose model is further used to synthesize long pose sequences of various motion patterns through a recursive generation method. Evaluation analysis results demonstrate that our novel model synthesizes highly realistic mollusk motions and achieves state-of-the-art performance. Our work proves high performance of deep neural networks for mollusk motion synthesis and the feasibility of long pose sequence synthesis based on the customized body shape and guiding path.

基于路径引导的果蝇幼虫运动合成

[1]Aksan E, Kaufmann M, Hilliges O, 2019. Structured prediction helps 3D human motion modelling. Proc IEEE/CVF Int Conf on Computer Vision, p.7143-7152.

[2]Aksan E, Kaufmann M, Cao P, et al., 2021. A spatio-temporal transformer for 3D human motion prediction. Proc Int Conf on 3D Vision, p.565-574.

[3]Barsoum E, Kender J, Liu ZC, 2018. HP-GAN: probabilistic 3D human motion prediction via GAN. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition Workshops, p.1499-1508.

[4]Bhattacharya U, Rewkowski N, Banerjee A, et al., 2021. Text2Gestures: a transformer-based network for generating emotive body gestures for virtual agents. Proc IEEE Virtual Reality and 3D User Interfaces, p.1-10.

[5]Busso C, Deng ZG, Neumann U, et al., 2005. Natural head motion synthesis driven by acoustic prosodic features. Comput Anim Virtual Worlds, 16:283-290.

[6]Cao JK, Tang HY, Fang HS, et al., 2019. Cross-domain adaptation for animal pose estimation. Proc IEEE/CVF Int Conf on Computer Vision, p.9497-9506.

[7]Carreira J, Zisserman A, 2017. Quo Vadis, action recognition? A new model and the kinetics dataset. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.4724-4733.

[8]Coros S, Beaudoin P, van de Panne M, 2010. Generalized biped walking control. Proc ACM SIGGRAPH, p.130.

[9]Cui QJ, Sun HJ, 2021. Towards accurate 3D human motion prediction from incomplete observations. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.4799-4808.

[10]Dang Q, Yin JQ, Wang B, et al., 2019. Deep learning based 2D human pose estimation: a survey. Tsinghua Sci Technol, 24(6):663-676.

[11]Dong R, Chang Q, Ikuno S, 2021. A deep learning framework for realistic robot motion generation. Neur Comput Appl, p.1-14.

[12]Eberly D, 2007. 3D Game Engine Design: a Practical Approach to Real-Time Computer Graphics (2^nd Ed.). CRC Press, Boca Raton, USA.

[13]Fragkiadaki K, Levine S, Felsen P, et al., 2015. Recurrent network models for human dynamics. Proc IEEE Int Conf on Computer Vision, p.4346-4354.

[14]Ghosh P, Song J, Aksan E, et al., 2017. Learning human motion models for long-term predictions. Proc Int Conf on 3D Vision, p.458-466.

[15]Goodfellow I, Pouget-Abadie J, Mirza M, et al., 2014. Generative adversarial networks. Commun ACM, 63(11):139-144.

[16]Guo X, Choi J, 2019. Human motion prediction via learning local structure representations and temporal dependencies. Proc 33^rd AAAI Conf on Artificial Intelligence, p.‍2580-2587.

[17]He KM, Gkioxari G, Dollár P, et al., 2017. Mask R-CNN. Proc IEEE Int Conf on Computer Vision, p.2980-2988.

[18]Heusel M, Ramsauer H, Unterthiner T, et al., 2017. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Proc 31^st Int Conf on Neural Information Processing Systems, p.6629-6640.

[19]Holden D, Saito J, Komura T, 2016. A deep learning framework for character motion synthesis and editing. ACM Trans Graph, 35(4):138.

[20]Ionescu C, Papava D, Olaru V, et al., 2014. Human3.6M: large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Trans Patt Anal Mach Intell, 36(7):1325-1339.

[21]Jain A, Zamir AR, Savarese S, et al., 2016. Structural-RNN: deep learning on spatio-temporal graphs. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.‍5308-5317.

[22]Jain DK, Zareapoor M, Jain R, et al., 2020. GAN-Poser: an improvised bidirectional GAN model for human motion prediction. Neur Comput Appl, 32(18):14579-14591.

[23]Ji SW, Xu W, Yang M, et al., 2013. 3D convolutional neural networks for human action recognition. IEEE Trans Patt Anal Mach Intell, 35(1):221-231.

[24]Kalman RE, 1960. A new approach to linear filtering and prediction problems. J Basic Eng, 82(1):35-45.

[25]Kingma DP, Ba J, 2015. Adam: a method for stochastic optimization. Proc 3rd Int Conf on Learning Representations.

[26]Kundu JN, Gor M, Babu RV, 2019. BiHMP-GAN: bidirectional 3D human motion prediction GAN. Proc 33^rd AAAI Conf on Artificial Intelligence, p.8553-8560.

[27]Lehrmann AM, Gehler PV, Nowozin S, 2013. A non-parametric Bayesian network prior of human pose. Proc IEEE Int Conf on Computer Vision, p.1281-1288.

[28]Li C, Zhang Z, Lee WS, et al., 2018. Convolutional sequence to sequence model for human dynamics. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.5226-5234.

[29]Li MS, Chen SH, Zhao YH, et al., 2020. Dynamic multiscale graph neural networks for 3D skeleton based human motion prediction. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.211-220.

[30]Li RL, Yang S, Ross DA, et al., 2021. AI choreographer: music conditioned 3D dance generation with AIST++. Proc IEEE/CVF Int Conf on Computer Vision, p.13381-13392.

[31]Li YR, Wang Z, Yang XS, et al., 2019. Efficient convolutional hierarchical autoencoder for human motion prediction. Vis Comput, 35(6):1143-1156.

[32]Liu C, Wang DL, Zhang H, et al., 2022. Using simulated training data of voxel-level generative models to improve 3D neuron reconstruction. IEEE Trans Med Imaging, 41(12):3624-3635.

[33]Liu LB, Yin KK, van de Panne M, et al., 2010. Sampling-based contact-rich motion control. ACM Trans Graph, 29(4):128.

[34]Liu XL, Yin JQ, Liu J, et al., 2021. TrajectoryCNN: a new spatio-temporal feature learning network for human motion prediction. IEEE Trans Circ Syst Video Technol, 31(6):2133-2146.

[35]Mao W, Liu MM, Salzmann M, et al., 2019. Learning trajectory dependencies for human motion prediction. Proc IEEE/CVF Int Conf on Computer Vision, p.9488-9496.

[36]Martinez J, Black MJ, Romero J, 2017. On human motion prediction using recurrent neural networks. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.4674-4683.

[37]Miyato T, Kataoka T, Koyama M, et al., 2018. Spectral normalization for generative adversarial networks. Proc 6^th Int Conf on Learning Representations.

[38]Mourot L, Hoyet L, Le Clerc F, et al., 2022. A survey on deep learning for skeleton-based human animation. Comput Graph Forum, 41(1):122-157.

[39]Negrete SB, Labuguen R, Matsumoto J, et al., 2021. Multiple monkey pose estimation using OpenPose.

[40]Okajima S, Tournier M, Alnajjar FS, et al., 2018. Generation of human-like movement from symbolized information. Front Neurorobot, 12:43.

[41]Pavllo D, Grangier D, Auli M, 2018. QuaterNet: a quaternion-based recurrent model for human motion. Proc British Machine Vision Conf.

[42]Pavlovic V, Rehg JM, MacCormick J, 2000. Learning switching linear models of human motion. Proc 13^th Int Conf on Neural Information Processing Systems, p.942-948.

[43]Sha T, Zhang W, Shen T, et al., 2021. Deep person generation: a survey from the perspective of face, pose and cloth synthesis.

[44]Shooter M, Malleson C, Hilton A, 2021. SyDog: a synthetic dog dataset for improved 2D pose estimation.

[45]Sok KW, Kim M, Lee J, 2007. Simulating biped behaviors from human motion data. ACM Trans Graph, 26(3):‍107.1-107.9.

[46]Stephens GJ, Johnson-Kerner B, Bialek W, et al., 2008. Dimensionality and dynamics in the behavior of C. elegans. PLoS Comput Biol, 4(4):e1000028.

[47]Sun K, Xiao B, Liu D, et al., 2019. Deep high-resolution representation learning for human pose estimation. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.5686-5696.

[48]Wang YC, Wang X, Jiang PL, et al., 2019. RNN-based human motion prediction via differential sequence representation. Proc IEEE 6^th Int Conf on Cloud Computing and Intelligence Systems, p.138-143.

[49]Yan SJ, Li ZZ, Xiong YJ, et al., 2019. Convolutional sequence generation for skeleton-based action synthesis. Proc IEEE/CVF Int Conf on Computer Vision, p.4393-4401.

[50]Yekutieli Y, Sagiv-Zohar R, Hochner B, et al., 2005. Dynamic model of the octopus arm. II. Control of reaching movements. J Neurophysiol, 94(2):1459-1468.

[51]Yin KK, Loken K, van de Panne M, 2007. SIMBICON: simple biped locomotion control. ACM Trans Graph, 26(3):105-es.

[52]Yin KK, Coros S, Beaudoin P, et al., 2008. Continuation methods for adapting simulated skills. ACM Trans Graph, 27(3):1-7.

[53]Yu SZ, 2010. Hidden semi-Markov models. Artif Intell, 174(2):215-243.

[54]Zhang DJ, Wu YQ, Guo MY, et al., 2021. Deep learning methods for 3D human pose estimation under different supervision paradigms: a survey. Electronics, 10(18):2267.

[55]Zhang H, Starke S, Komura T, et al., 2018. Mode-adaptive neural networks for quadruped motion control. ACM Trans Graph, 37(4):145.

[56]Zhao R, Ji Q, 2018. An adversarial hierarchical hidden Markov model for human pose modeling and generation. Proc 32^nd AAAI Conf on Artificial Intelligence, p.2636-2643.