CLC number: TP39
On-line Access: 2024-02-19
Received: 2023-05-21
Revision Accepted: 2024-02-19
Crosschecked: 2023-10-29
Cited: 0
Clicked: 873
Citations: Bibtex RefMan EndNote GB/T7714
Weining WANG, Jiahui LI, Yifan LI, Xiaofen XING. Style-conditioned music generation with Transformer-GANs[J]. Frontiers of Information Technology & Electronic Engineering, 2024, 25(1): 106-120.
@article{title="Style-conditioned music generation with Transformer-GANs",
author="Weining WANG, Jiahui LI, Yifan LI, Xiaofen XING",
journal="Frontiers of Information Technology & Electronic Engineering",
volume="25",
number="1",
pages="106-120",
year="2024",
publisher="Zhejiang University Press & Springer",
doi="10.1631/FITEE.2300359"
}
%0 Journal Article
%T Style-conditioned music generation with Transformer-GANs
%A Weining WANG
%A Jiahui LI
%A Yifan LI
%A Xiaofen XING
%J Frontiers of Information Technology & Electronic Engineering
%V 25
%N 1
%P 106-120
%@ 2095-9184
%D 2024
%I Zhejiang University Press & Springer
%DOI 10.1631/FITEE.2300359
TY - JOUR
T1 - Style-conditioned music generation with Transformer-GANs
A1 - Weining WANG
A1 - Jiahui LI
A1 - Yifan LI
A1 - Xiaofen XING
J0 - Frontiers of Information Technology & Electronic Engineering
VL - 25
IS - 1
SP - 106
EP - 120
%@ 2095-9184
Y1 - 2024
PB - Zhejiang University Press & Springer
ER -
DOI - 10.1631/FITEE.2300359
Abstract: Recently, various algorithms have been developed for generating appealing music. However, the style control in the generation process has been somewhat overlooked. Music style refers to the representative and unique appearance presented by a musical work, and it is one of the most salient qualities of music. In this paper, we propose an innovative music generation algorithm capable of creating a complete musical composition from scratch based on a specified target style. A style-conditioned linear transformer and a style-conditioned patch discriminator are introduced in the model. The style-conditioned linear transformer models musical instrument digital interface (MIDI) event sequences and emphasizes the role of style information. Simultaneously, the style-conditioned patch discriminator applies an adversarial learning mechanism with two innovative loss functions to enhance the modeling of music sequences. Moreover, we establish a discriminative metric for the first time, enabling the evaluation of the generated music’s consistency concerning music styles. Both objective and subjective evaluations of our experimental results indicate that our method’s performance with regard to music production is better than the performances encountered in the case of music production with the use of state-of-the-art methods in available public datasets.
[1]Brunner G, Konrad A, Wang YY, et al., 2018. MIDI-VAE: modeling dynamics and instrumentation of music with applications to style transfer. Proc 19th Int Society for Music Information Retrieval Conf, p.747-754.
[2]Choi K, Hawthorne C, Simon I, et al., 2020. Encoding musical style with Transformer autoencoders. Proc 37th Int Conf on Machine Learning, p.1899-1908.
[3]Chou YH, Chen IC, Chang CJ, et al., 2021. MidiBERT-Piano: large-scale pre-training for symbolic music understanding.
[4]Delgado M, Fajardo W, Molina-Solana M, 2009. Inmamusys: intelligent multiagent music system. Expert Syst Appl, 36(3):4574-4580.
[5]Devlin J, Chang MW, Lee K, et al., 2019. BERT: pre-training of deep bidirectional Transformers for language understanding. Proc Conf of the North American Chapter of the Association for Computational Linguistics, p.4171-4186.
[6]Dong HW, Yang YH, 2018. Convolutional generative adversarial networks with binary neurons for polyphonic music generation. Proc 19th Int Society for Music Information Retrieval Conf, p.190-196.
[7]Dong HW, Hsiao WY, Yang LC, et al., 2018. MuseGAN: multi-track sequential generative adversarial networks for symbolic music generation and accompaniment. Proc 32nd AAAI Conf on Artificial Intelligence, Article 5.
[8]Dong HW, Chen K, McAuley JJ, et al., 2020. MusPy: a toolkit for symbolic music generation. Proc 21st Int Society for Music Information Retrieval Conf, p.101-108.
[9]Dosovitskiy A, Beyer L, Kolesnikov A, et al., 2021. An image is worth 16x16 words: Transformers for image recognition at scale. Proc 9th Int Conf on Learning Representations.
[10]Ferreira LN, Whitehead J, 2021. Learning to generate music with sentiment. Proc 29th Int Society for Music Information Retrieval Conf, p.384-390.
[11]Goodfellow I, Pouget-Abadie J, Mirza M, et al., 2020. Generative adversarial networks. Commun ACM, 63(11):139-144.
[12]Herremans D, Chew E, 2019. MorpheuS: generating structured music with constrained patterns and tension. IEEE Trans Affect Comput, 10(4):510-523.
[13]Hsiao WY, Liu JY, Yeh YC, et al., 2021. Compound word Transformer: learning to compose full-song music over dynamic directed hypergraphs. Proc 35th AAAI Conf on Artificial Intelligence, p.178-186.
[14]Huang CZA, Vaswani A, Uszkoreit J, et al., 2019. Music Transformer: generating music with long-term structure. Proc 7th Int Conf on Learning Representations.
[15]Huang YS, Yang YH, 2020. Pop music Transformer: beat-based modeling and generation of expressive pop piano compositions. Proc 28th ACM Int Conf on Multimedia, p.1180-1188.
[16]Hung HT, Ching J, Doh S, et al., 2021. EMOPIA: a multi-modal pop piano dataset for emotion recognition and emotion-based music generation. Proc 22nd Int Society for Music Information Retrieval Conf, p.318-325.
[17]Jang E, Gu SX, Poole B, 2017. Categorical reparameterization with Gumbel-Softmax. Proc 5th Int Conf on Learning Representations.
[18]Jhamtani H, Berg-Kirkpatrick T, 2019. Modeling self-repetition in music generation using generative adversarial networks. Proc 36th Int Conf on Machine Learning.
[19]Jiang JY, Wang ZQ, 2019. Stylistic melody generation with conditional variational auto-encoder. Available fromhttps://www.cs.cmu.edu/~epxing/Class/10708-19/assets/project/final-reports/project8.pdf [Accessed on Oct. 28, 2023].
[20]Jiang JY, Xia GG, Carlton DB, et al., 2020. Transformer VAE: a hierarchical model for structure-aware and interpretable music representation learning. Proc IEEE Int Conf on Acoustics, Speech and Signal Processing, p.516-520.
[21]Kaliakatsos-Papakostas M, Floros A, Vrahatis MN, 2020. Artificial intelligence methods for music generation: a review and future perspectives. In: Yang XS (Ed.), Nature-Inspired Computation and Swarm Intelligence. Academic Press, Amsterdam, p.217-245.
[22]Katharopoulos A, Vyas A, Pappas N, et al., 2020. Transformers are RNNs: fast autoregressive transformers with linear attention. Proc 37th Int Conf on Machine Learning, p.5156-5165.
[23]Ke GL, He D, Liu TY, 2021. Rethinking positional encoding in language pre-training. Proc 9th Int Conf on Learning Representations.
[24]Leach J, Fitch J, 1995. Nature, music, and algorithmic composition. Comput Music J, 19(2):23-33.
[25]Liang X, Wu JM, Cao J, 2019. MIDI-Sandwich2: RNN-based hierarchical multi-modal fusion generation VAE networks for multi-track symbolic music generation.
[26]Liao YK, Yue W, Jian YQ, et al., 2022. MICW: a multi-instrument music generation model based on the improved compound word. Proc IEEE Int Conf on Multi- media and Expo Workshops, p.1-10.
[27]Lim YQ, Chan CS, Loo FY, 2020. Style-conditioned music generation. Proc IEEE Int Conf on Multimedia and Expo, p.1-6.
[28]Liu HM, Yang YH, 2018. Lead sheet generation and arrangement by conditional generative adversarial network. Proc 17th IEEE Int Conf on Machine Learning and Applications, p.722-727.
[29]Livingstone SR, Mühlberger R, Brown AR, et al., 2010. Changing musical emotion: a computational rule system for modifying score and performance. Comput Music J, 34(1):41-64.
[30]Lousseief E, Sturm BLT, 2019. MahlerNet: unbounded orchestral music with neural networks. Proc Nordic Sound and Music Computing Conf and the Interactive Sonification Workshop, p.58-64.
[31]Luo J, Yang XY, Ji SL, et al., 2020. MG-VAE: deep Chinese folk songs generation with specific regional styles. Proc 7th Conf on Sound and Music Technology, p.93-106.
[32]Mao HH, Shin T, Cottrell G, 2018. DeepJ: style-specific music generation. Proc IEEE 12th Int Conf on Semantic Computing, p.377-382.
[33]Mou LT, Sun YH, Tian YH, et al., 2023. MemoMusic 3.0: considering context at music recommendation and combining music theory at music generation. Proc IEEE Int Conf on Multimedia and Expo Workshops, p.296-301.
[34]Muhamed A, Li L, Shi XJ, et al., 2021. Symbolic music generation with Transformer-GANs. Proc 35th AAAI Conf on Artificial Intelligence, p.408-417.
[35]Nie WL, Narodytska N, Patel A, 2019. RelGAN: relational generative adversarial networks for text generation. Proc 7th Int Conf on Learning Representations.
[36]Oore S, Simon I, Dieleman S, et al., 2020. This time with feeling: learning expressive musical performance. Neur Comput Appl, 32(4):955-967.
[37]Ren Y, He JZ, Tan X, et al., 2020. PopMAG: pop music accompaniment generation. Proc 28th ACM Int Conf on Multimedia, p.1198-1206.
[38]Rivero D, Ramírez-Morales I, Fernandez-Blanco E, et al., 2020. Classical music prediction and composition by means of variational autoencoders. Appl Sci, 10(9):3053.
[39]Roberts A, Engel J, Raffel C, et al., 2018. A hierarchical latent vector model for learning long-term structure in music. Proc 35th Int Conf on Machine Learning, p.4364-4373.
[40]Shih YJ, Wu SL, Zalkow F, et al., 2022. Theme Transformer: symbolic music generation with theme-conditioned Transformer. IEEE Trans Multimed, 25:3495-3508.
[41]Sulun S, Davies MEP, Viana P, 2022. Symbolic music generation conditioned on continuous-valued emotions. IEEE Access, 10:44617-44626.
[42]Supper M, 2001. A few remarks on algorithmic composition. Comput Music J, 25(1):48-53.
[43]Trieu N, Keller RM, 2018. JazzGAN: improvising with generative adversarial networks. Proc 6th Int Workshop on Musical Metacreation.
[44]Vaswani A, Shazeer N, Parmar N, et al., 2017. Attention is all you need. Proc 31st Int Conf on Neural Information Processing Systems, p.6000-6010.
[45]Waite E, Eck D, Roberts A, et al., 2016. Project magenta: generating long-term structure in songs and stories. Available fromhttps://github.com/magenta/magenta/issues/1438 [Accessed on Oct. 28, 2023].
[46]Wang L, Zhao ZY, Liu HW, et al., 2023. A review of intelligent music generation systems.
[47]Wang WP, Li XB, Jin C, et al., 2022. CPS: full-song and style-conditioned music generation with linear transformer. Proc IEEE Int Conf on Multimedia and Expo Workshops, p.1-6.
[48]Williams RJ, Zipser D, 1989. A learning algorithm for continually running fully recurrent neural networks. Neur Comput, 1(2):270-280.
[49]Wu SL, Yang YH, 2020. The jazz Transformer on the front line: exploring the shortcomings of AI-composed music through quantitative measures. Proc 21st Int Society for Music Information Retrieval Conf, p.142-149.
[50]Wu XC, Wang CY, Lei QY, 2020. Transformer-XL based music generation with multiple sequences of time-valued notes.
[51]Yang LC, Lerch A, 2020. On the evaluation of generative models in music. Neur Comput Appl, 32(9):4773-4784.
[52]Yang LC, Chou SY, Yang YH, 2017. MidiNet: a convolutional generative adversarial network for symbolic-domain music generation. Proc 18th Int Society for Music Information Retrieval Conf, p.324-331.
[53]Yu BT, Lu PL, Wang R, et al., 2022. Museformer: Transformer with fine- and coarse-grained attention for music generation. Proc 36th Conf on Neural Information Processing Systems, p.1376-1388.
[54]Zhang N, 2023. Learning adversarial transformer for symbolic music generation. IEEE Trans Neur Netw Learn Syst, 34(4):1754-1763.
[55]Zhang XY, Zhang JC, Qiu Y, et al., 2022. Structure-enhanced pop music generation via harmony-aware learning. Proc 30th ACM Int Conf on Multimedia, p.1204-1213.
[56]Zhong K, Qiao TW, Zhang LQ, 2019. A study of emotional communication of emoticon based on Russell’s Circumplex Model of Affect. Proc 8th Int Conf on Human-Computer Interaction, p.577-596.
Open peer comments: Debate/Discuss/Question/Opinion
<1>