Similar articles

Abstract: Text generation is an essential research area in artificial intelligence (AI) technology and natural language processing and provides key technical support for the rapid development of AI-generated content (AIGC). It is based on technologies such as natural language processing, machine learning, and deep learning, which enable learning language rules through training models to automatically generate text that meets grammatical and semantic requirements. In this paper, we sort and systematically summarize the main research progress in text generation and review recent text generation papers, focusing on presenting a detailed understanding of the technical models. In addition, several typical text generation application systems are presented. Finally, we address some challenges and future directions in AI text generation. We conclude that improving the quality, quantity, interactivity, and adaptability of generated text can help fundamentally advance AI text generation development.

人工智能文本生成的进展与挑战

[1]Adamopoulou E, Moussiades L, 2020. Chatbots: history, technology, and applications. Mach Learn Appl, 2:100006.

[2]Akter M, Bansal N, Karmaker SK, 2022. Revisiting automatic evaluation of extractive summarization task: can we do better than ROUGE? Findings of the Association for Computational Linguistics, p.1547-1560.

[3]Albawi S, Mohammed TA, Al-Zawi S, 2017. Understanding of a convolutional neural network. Proc Int Conf on Engineering and Technology, p.1-6.

[4]Alexandr N, Irina O, Tatyana K, et al., 2021. Fine-tuning GPT-3 for Russian text summarization. Proc 5^th Computational Methods in Systems and Software, p.748-757.

[5]Barker E, Paramita ML, Aker A, et al., 2016. The SENSEI annotated corpus: human summaries of reader comment conversations in on-line news. Proc 17^th Annual Meeting of the Special Interest Group on Discourse and Dialogue, p.42-52.

[6]Barrera A, Verma R, 2012. Combining syntax and semantics for automatic extractive single-document summarization. Proc 13^th Int Conf on Computational Linguistics and Intelligent Text Processing, p.366-377.

[7]Birjali M, Kasri M, Beni-Hssane A, 2021. A comprehensive survey on sentiment analysis: approaches, challenges and trends. Knowl-Based Syst, 226:107134.

[8]Cai D, Wang Y, Liu L, et al., 2022. Recent advances in retrieval-augmented text generation. Proc 45^th Int ACM SIGIR Conf on Research and Development in Information Retrieval, p.3417-3419.

[9]Cao JR, Wang CW, 2018. Social media text generation based on neural network model. Proc 2^nd Int Conf on Computer Science and Artificial Intelligence, p.58-61.

[10]Chen YC, Gan Z, Cheng Y, et al., 2020. Distilling knowledge learned in BERT for text generation. Proc 58^th Annual Meeting of the Association for Computational Linguistics, p.7893-7905.

[11]Chen ZH, Shen YL, Song Y, et al., 2021. Cross-modal memory networks for radiology report generation. Proc 59^th Annual Meeting of the Association for Computational Linguistics and the 11^th Int Joint Conf on Natural Language Processing, p.5904-5914.

[12]Christian H, Agus MP, Suhartono D, 2016. Single document automatic text summarization using term frequency-inverse document frequency (TF-IDF). ComTech, 7(4):285-294.

[13]Coenen A, Davis L, Ippolito D, et al., 2021. Wordcraft: a human-AI collaborative editor for story writing.

[14]Croft R, Newlands D, Chen ZY, et al., 2021. An empirical study of rule-based and learning-based approaches for static application security testing. Proc 15^th ACM/IEEE Int Symp on Empirical Software Engineering and Measurement, Article 8.

[15]Dathathri S, Madotto A, Lan J, et al., 2020. Plug and play language models: a simple approach to controlled text generation. Proc 8^th Int Conf on Learning Representations.

[16]Devlin KJ, Chang MW, Lee K, 2019. BERT: pre-training of deep bidirectional Transformers for language understanding. Proc Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p.4171-4186.

[17]Dey R, Salem FM, 2017. Gate-variants of gated recurrent unit (GRU) neural networks. Proc IEEE 60^th Int Midwest Symp on Circuits and Systems, p.1597-1600.

[18]Ding HJ, Xu XL, 2023. SAN-T2T: an automated table-to-text generator based on selective attention network. Nat Lang Eng, First View.

[19]Doush IA, Alkhateeb F, Albsoul A, 2017. AraDaisy: a system for automatic generation of ARABIC daisy books. Int J Comput Appl Technol, 55(4):322-333.

[20]Dowling M, Lucey B, 2023. ChatGPT for (finance) research: the bananarama conjecture. Financ Res Lett, 53:103662.

[21]Evtikhiev M, Bogomolov E, Sokolov Y, et al., 2023. Out of the BLEU: how should we assess quality of the code generation models?J Syst Softw, 203:111741.

[22]Fan A, Gardent C, Braud C, et al., 2019. Using local knowledge graph construction to scale Seq2Seq models to multi-document inputs. Proc Conf on Empirical Methods in Natural Language Processing and the 9^th Int Joint Conf on Natural Language Processing, p.4186-4196.

[23]Fei H, Li CL, Ji DH, et al., 2022. Mutual disentanglement learning for joint fine-grained sentiment classification and controllable text generation. Proc 45^th Int ACM SIGIR Conf on Research and Development in Information Retrieval, p.1555-1565.

[24]Feng YS, Lapata M, 2013. Automatic caption generation for news images. IEEE Trans Patt Anal Mach Intell, 35(4):797-812.

[25]Floridi L, Chiriatti M, 2020. GPT-3: its nature, scope, limits, and consequences. Minds Mach, 30(4):681-694.

[26]Gao S, Ren ZC, Zhao YH, et al., 2019. Product-aware answer generation in e-commerce question-answering. Proc 12^th ACM Int Conf on Web Search and Data Mining, p.429-437.

[27]Garbacea C, Mei QZ, 2020. Neural language generation: formulation, methods, and evaluation.

[28]Glennie R, Adam T, Leos-Barajas V, et al., 2023. Hidden Markov models: pitfalls and opportunities in ecology. Methods Ecol Evol, 14(1):43-56.

[29]Goodfellow I, Pouget-Abadie J, Mirza M, et al., 2020. Generative adversarial networks. Commun ACM, 63(11):139-144.

[30]Guo ZX, Yan MX, Qi JX, et al., 2023. Few-shot table-to-text generation with prompt planning and knowledge memorization.

[31]Gupta A, Chugh D, Anjum, 2022. Automated news summarization using transformers. Int Conf on Sustainable Advanced Computing, p.249-259.

[32]He XL, Xu QK, Zeng Y, et al., 2022. CATER: intellectual property protection on text generation APIs via conditional watermarks. Proc 36^th Conf on Neural Information Processing Systems.

[33]Hua XY, Wang L, 2020. PAIR: planning and iterative refinement in pre-trained Transformers for long text generation. Proc Conf on Empirical Methods in Natural Language Processing, p.781-793.

[34]Imam AT, Rousan T, Aljawarneh S, 2014. An expert code generator using rule-based and frames knowledge representation techniques. Proc 5^th Int Conf on Information and Communication Systems, p.1-6.

[35]Jayasiriwardene TD, Ganegoda GU, 2020. Keyword extraction from tweets using NLP tools for collecting relevant news. Proc Int Research Conf on Smart Computing and Systems Engineering, p.129-135.

[36]Jin Q, Dhingra B, Liu ZP, et al., 2019. PubMedQA: a dataset for biomedical research question answering. Proc 9^th Int Joint Conf on Natural Language Processing, p.2567-2577.

[37]Jungwirth D, Haluza D, 2023. Artificial intelligence and the sustainable development goals: an exploratory study in the context of the society domain. J Softw Eng Appl, 16(4):91-112.

[38]Karakoylu PE, Kural AG, Gulden S, 2020. The effect of artificial intelligence (AI) on new product development (NPD): a future scenario. IOP Conf Ser Mater Sci Eng, 960:022026.

[39]Keskar NS, McCann B, Varshney LR, et al., 2019. CTRL: a conditional transformer language model for controllable generation.

[40]King MR, 2023. A place for large language models in scientific publishing, apart from credited authorship. Cell Mol Bioeng, 16(2):95-98.

[41]King MR, ChatGPT, 2023. A conversation on artificial intelligence, chatbots, and plagiarism in higher education. Cell Mol Bioeng, 16(1):1-2.

[42]Kingma DP, Welling M, 2014. Auto-encoding variational Bayes. https://arxiv.org/abs/1312.6114

[43]Koncel-Kedziorski R, Bekal D, Luan Y, et al., 2019. Text generation from knowledge graphs with graph transformers. Proc Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p.2284-2293.

[44]Koupaee M, Wang WY, 2018. WikiHow: a large scale text summarization dataset.

[45]Kraus S, Castellanos I, Albermann M, et al., 2016. Using Arden syntax for the generation of intelligent intensive care discharge letters. Stud Health Technol Inform, 228:471-475.

[46]Lai V, Smith-Renner A, Zhang K, et al., 2022. An exploration of post-editing effectiveness in text summarization. Proc Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p.475-493.

[47]Lee JS, Hsiang J, 2020. Patent claim generation by fine-tuning OpenAI GPT-2. World Patent Inform, 62:101983.

[48]Leppänen L, Munezero M, Granroth-Wilding M, et al., 2017. Data-driven news generation for automated journalism. Proc 10^th Int Conf on Natural Language Generation, p.188-197.

[49]Lewis M, Liu YH, Goyal N, et al., 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. Proc 58^th Annual Meeting of the Association for Computational Linguistics, p.7871-7880.

[50]Li B, Yang P, Zhao HL, et al., 2023. Hierarchical sliding inference generator for question-driven abstractive answer summarization. ACM Trans Inform Syst, 41(1):7.

[51]Li JJ, Song HZ, Li J, 2022. Transformer-based question text generation in the learning system. Proc 6^th Int Conf on Innovation in Artificial Intelligence, p.50-56.

[52]Li YJ, Choi D, Chung J, et al., 2022. Competition-level code generation with AlphaCode. Science, 378(6624):1092-1097.

[53]Liang ZY, Du JP, Shao YX, et al., 2021. Gated graph neural attention networks for abstractive summarization. Neurocomputing, 431:128-136.

[54]Lin J, Madnani N, Dorr BJ, 2010. Putting the user in the loop: interactive maximal marginal relevance for query-focused summarization. Proc Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p.305-308.

[55]Lin JY, Sun X, Ma SM, et al., 2018. Global encoding for abstractive summarization. Proc 56^th Annual Meeting of the Association for Computational Linguistics, p.163-169.

[56]Liu TY, Wang KX, Sha L, et al., 2018. Table-to-text generation by structure-aware Seq2Seq learning. Proc 32^nd AAAI Conf on Artificial Intelligencee, 32(1).

[57]Liu Y, Lapata M, 2019a. Hierarchical Transformers for multi-document summarization. Proc 57^th Annual Meeting of the Association for Computational Linguistics, p.5070-5081.

[58]Liu Y, Lapata M, 2019b. Text summarization with pretrained encoders. Proc Conf on Empirical Methods in Natural Language Processing and the 9^th Int Joint Conf on Natural Language Processing, p.3730-3740.

[59]Liu Y, Wan Y, He LF, et al., 2021. KG-BART: knowledge graph-augmented BART for generative commonsense reasoning. Proc 35^th AAAI Conf on Artificial Intelligence, p.6418-6425.

[60]Liu YH, Ott M, Goyal N, et al., 2019. RoBERTa: a robustly optimized BERT pretraining approach.

[61]Lund BD, Wang T, 2023. Chatting about ChatGPT: how may AI and GPT impact academia and libraries?Lib Hi Tech News, 40(3):26-29.

[62]Luo Y, Zhao SL, Li XC, et al., 2016. Text keyword extraction method based on word frequency statistics. J Comput Appl, 36(3):718-725(in Chinese).

[63]Ma C, Zhang S, Shen GH, et al., 2022. Switch-GPT: an effective method for constrained text generation under few-shot settings (student abstract). Proc 36^th AAAI Conf on Artificial Intelligence, p.13011-13012.

[64]Meng R, Zhao SQ, Han SG, et al., 2017. Deep keyphrase generation. Proc 55^th Annual Meeting of the Association for Computational Linguistics, p.582-592.

[65]Miller D, 2019. Leveraging BERT for extractive text summarization on lectures.

[66]Mukherjee S, 2021. Sentiment analysis. In: Mukherjee S (Ed.), MLNET Revealed: Simple Tools for Applying Machine Learning to Your Applications. Apress, Berkeley, p.113-127.

[67]Oh C, Choi J, Lee S, et al., 2020. Understanding user perception of automated news generation system. Proc CHI Conf on Human Factors in Computing Systems, p.1-13.

[68]Omar R, Mangukiya O, Kalnis P, et al., 2023. ChatGPT versus traditional question answering for knowledge graphs: current status and future directions towards knowledge graph chatbots.

[69]Özateş ŞB, Özgür A, Radev D, 2016. Sentence similarity based on dependency tree kernels for multi-document summarization. Proc 10^th Int Conf on Language Resources and Evaluation, p.2833-2838.

[70]Palangi H, Deng L, Shen YL, et al., 2016. Deep sentence embedding using long short-term memory networks: analysis and application to information retrieval. IEEE/ACM Trans Audio Speech Lang Process, 24(4):694-707.

[71]Procter R, Arana-Catania M, He YL, et al., 2023. Some observations on fact-checking work with implications for computational support.

[72]Rudin C, 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell, 1(5):206-215.

[73]Saeed A, Ilić S, Zangerle E, 2019. Creative GANs for generating poems, lyrics, and metaphors.

[74]See A, Liu PJ, Manning CD, 2017. Get to the point: summarization with pointer-generator networks. Proc 55^th Annual Meeting of the Association for Computational Linguistics, p.1073-1083.

[75]Selvadurai N, Matulionyte R, 2020. Reconsidering creativity: copyright protection for works generated using artificial intelligence. J Intellect Prop Law Pract, 15(7):536-543.

[76]Senftleben M, Buijtelaar L, et al., 2020. Robot creativity: an incentive-based neighbouring rights approach. Eur Intellect Prop Rev, 42(12):717.

[77]Smith S, Patwary M, Norick B, et al., 2022. Using DeepSpeed and megatron to train Megatron-Turing NLG 530B, a large-scale generative language model.

[78]Stevens K, 2022. Retrospective enhancement of bibliographic records with tables of contents and summary data and the effect on collection usage. J Aust Lib Inform Assoc, 71(4):379-387.

[79]Sun XF, Meng YX, Ao X, et al., 2022. Sentence similarity based on contexts. Trans Assoc Comput Ling, 10:573-588.

[80]Surameery NMS, Shakor MY, 2023. Use ChatGPT to solve programming bugs. Int J Inform Technol Comput Eng, 3(1):17-22.

[81]Svyatkovskiy A, Deng SK, Fu SY, et al., 2020. IntelliCode compose: code generation using transformer. Proc 28^th ACM Joint Meeting on European Software Engineering Conf and Symp on the Foundations of Software Engineering, p.1433-1443.

[82]Thoppilan R, De Freitas D, Hall J, et al., 2022. LaMDA: language models for dialog applications.

[83]Tian YF, Peng NY, 2022. Zero-shot sonnet generation with discourse-level planning and aesthetics features. Proc Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p.3587-3597.

[84]Tomczak JM, Welling M, 2016. Improving variational auto-encoders using householder flow.

[85]Tran OT, Bui VT, 2021. Neural text normalization in speech-to-text systems with rich features. Appl Artif Intell, 35(3):193-205.

[86]Trivedi A, Pant N, Shah P, et al., 2018. Speech to text and text to speech recognition systems—a review. IOSR J Comput Eng, 20(2):36-43.

[87]van der Lee C, Gatt A, van Miltenburg E, et al., 2019. Best practices for the human evaluation of automatically generated text. Proc 12^th Int Conf on Natural Language Generation, p.355-368.

[88]Vaswani A, Shazeer N, Parmar N, et al., 2017. Attention is all you need. Proc 31^st Int Conf on Neural Information Processing Systems, p.6000-6010.

[89]Vodolazova T, Lloret E, 2019. The impact of rule-based text generation on the quality of abstractive summaries. Proc Int Conf on Recent Advances in Natural Language Processing, p.1275-1284.

[90]Wang CH, Tang Y, Ma XT, et al., 2020. FAIRSEQ S2T: fast speech-to-text modeling with FAIRSEQ. Proc 1^st Conf of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10^th Int Joint Conf on Natural Language Processing: System Demonstrations, p.33-39.

[91]Wang TM, Wan XQ, Jin HQ, 2020. AMR-to-text generation with graph Transformer. Trans Assoc Comput Ling, 8:19-33.

[92]Wang WM, Li Z, Wang JW, et al., 2017. How far we can go with extractive text summarization? Heuristic methods to obtain near upper bounds. Exp Syst Appl, 90:439-463.

[93]Xu HH, Savelka J, Ashley KD, 2021. Accounting for Sentence Position and Legal Domain Sentence Embedding in Learning to Classify Case Sentences. IOS Press, Vilnius, Lithuania, p.33-42.

[94]Yellin DM, Weiss G, 2021. Synthesizing context-free grammars from recurrent neural networks. Proc 27^th Int Conf on Tools and Algorithms for the Construction and Analysis of Systems, p.351-369.

[95]Yu X, Vu NT, Kuhn J, 2019. Learning the dyck language with attention-based Seq2Seq models. Proc ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, p.138-146.

[96]Yu Y, Si XS, Hu CH, et al., 2019. A review of recurrent neural networks: LSTM cells and network architectures. Neur Comput, 31(7):1235-1270.

[97]Zhang TY, Kishore V, Wu F, et al., 2020. BERTScore: evaluating text generation with BERT. Proc 8^th Int Conf on Learning Representations.

[98]Zhang XY, Zou YY, Zhang HN, et al., 2022a. Automatic product copywriting for e-commerce. Proc 36^th AAAI Conf on Artificial Intelligence, p.12423-12431.

[99]Zhang XY, Shen K, Zhang C, et al., 2022b. Scenario-based multi-product advertising copywriting generation for e-commerce.

[100]Zhang YQ, Huang ML, 2019. Overview of the NTCIR-14 short text generation subtask: emotion generation challenge. Proc 14^th NTCIR Conf on Evaluation of Information Access Technologies, p.316-327.

[101]Zhang ZY, Han X, Liu ZY, et al., 2019. ERNIE: enhanced language representation with informative entities. Proc 57^th Annual Meeting of the Association for Computational Linguistics, p.1441-1451.

[102]Zhao GZ, Yang P, 2022. Table-based fact verification with self-labeled keypoint alignment. Proc 29^th Int Conf on Computational Linguistics, p.1401-1411.

[103]Zhao L, Xu JJ, Lin JY, et al., 2020. Graph-based multi-hop reasoning for long text generation.

[104]Zhao W, Peyrard M, Liu F, et al., 2019. MoverScore: text generation evaluating with contextualized embeddings and earth mover distance. Proc Conf on Empirical Methods in Natural Language Processing and the 9^th Int Joint Conf on Natural Language Processing, p.563-578.

[105]Zhao W, Strube M, Eger S, 2022. DiscoScore: evaluating text generation with BERT and discourse coherence. Proc 17^th Conf of the European Chapter of the Association for Computational Linguistics, p.3865-3883.

[106]Zheng LX, Ma S, Chen ZX, et al., 2021. Ensuring the correctness of regular expressions: a review. Int J Autom Comput, 18(4):521-535.

[107]Zhu CG, Yang ZY, Gmyr R, et al., 2021. Leveraging lead bias for zero-shot abstractive news summarization. Proc 44^th Int ACM SIGIR Conf on Research and Development in Information Retrieval, p.1462-1471.

[108]Zhu QH, Luo JX, 2023. Generative design ideation: a natural language generation approach. In: Gero JS (Ed.), Design Computing and Cognition’22. Springer, Cham, p.39-50.

[109]Zogopoulos V, Geurts E, Gors D, et al., 2022. Authoring tool for automatic generation of augmented reality instruction sequence for manual operations. Procedia CIRP, 106:84-89.