Abstract: The integration of large language models (LLMs) into financial applications has demonstrated remarkable potential for enhancing decision-making processes, automating operations, and delivering personalized services. However, the high-stakes nature of financial systems demands a very high level of trustworthiness that current LLMs often fail to meet. This study identifies and examines three major trustworthiness challenges in LLM-based financial systems: (1) jailbreak prompts that exploit vulnerabilities in model alignment to produce harmful or noncompliant responses; (2) hallucination, where models generate factually incorrect outputs that can mislead financial decision-making; (3) bias and fairness concerns, where demographic or institutional bias embedded in LLMs may result in unfair treatment of individuals or regions. To make these risks concrete, we design three finance-relevant probes and evaluate a set of mainstream LLMs spanning both proprietary and open-source families. Across models, we observe risky behavior in at least one scenario per probe. Based on these findings, we systematically summarize the existing mitigation strategies that aim to address these risks. We argue that resolving these issues is vital not only for ensuring the responsible use of artificial intelligence (AI) in the financial sector but also for enabling its safe and scalable deployment.

基于大语言模型的金融系统面临的三大可信度挑战：现实案例与缓解策略

[1]Andriushchenko M, Croce F, Flammarion N, 2025. Jailbreaking leading safety-aligned LLMs with simple adaptive attacks. 13^th Int Conf on Learning Representations.

[2]Authority FIR, 2009. Financial Industry Regulatory Authority. https://www.kurtalawfirm.com/wp-content/uploads/2019064126802-Clearview-Trading-Advisors-Inc.-CRD-142873-Gregg-H.-Ettin-CRD-1604260-AWC-geg-2022-1670804406029.pdf [Accessed on Mar. 23, 2025].

[3]Barry M, Caillaut G, Halftermeyer P, et al., 2025. GraphRAG: leveraging graph-based efficiency to minimize hallucinations in LLM-driven RAG for finance data. Proc Workshop on Generative AI and Knowledge Graphs, p.54-65. https://hal.science/hal-04907346

[4]Bowen DEIII, Price SM, Stein LCD, et al., 2025. Measuring and Mitigating Racial Disparities in LLMs: Evidence from a Mortgage Underwriting Experiment. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4812158 [Accessed on Mar. 23, 2025].

[5]Calver J, Church P, Ford J, et al., 2024. AI in financial services—the legal and regulatory landscape. In: Law 2024. Edward Elgar Publishing, Cheltenham, p.420-458.

[6]Choe J, Kim J, Jung W, 2025. Hierarchical retrieval with evidence curation for open-domain financial question answering on standardized documents. Findings of the Association for Computational Linguistics, p.16663-16681.

[7]Davis HA, 2007. Summary of selected FINRA regulatory notices. J Invest Compl, 8(4):60-67.

[8]Dong MM, Stratopoulos TC, Wang VX, 2024. A scoping review of ChatGPT research in accounting and finance. Int J Account Inform Syst, 55:100715.

[9]Gallegos IO, Rossi RA, Barrow J, et al., 2024. Bias and fairness in large language models: a survey. Comput Linguist, 50(3):1097-1179.

[10]Huang L, Yu WJ, Ma WT, et al., 2024. A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions.

[11]Kabra S, Jha A, Reddy CK, 2025. Reasoning towards fairness: mitigating bias in language models through reasoning-guided fine-tuning.

[12]Khachaturov D, Mullins R, 2025. Adversarial suffix filtering: a defense pipeline for LLMs.

[13]Kumar R, Kumar H, Shalini K, 2025. Detecting and mitigating bias in LLMs through knowledge graph-augmented training. Int Conf on Artificial Intelligence and Data Engineering, p.608-613.

[14]Lee J, Stevens N, Han SC, 2025. Language models in finance (FinLLMs). Neur Comput Appl, 37:24853-24867.

[15]Li XY, Chen ZP, Zhang JM, et al., 2024. Benchmarking bias in large language models during role-playing.

[16]Liu XG, Xu N, Chen MH, et al., 2024. AutoDAN: generating stealthy jailbreak prompts on aligned large language models. 12^th Int Conf on Learning Representations.

[17]Lundberg SM, Lee SI, 2017. A unified approach to interpreting model predictions. Proc 31^st Int Conf on Neural Information Processing Systems, p.4768-4777.

[18]Manakul P, Liusie A, Gales MJF, 2023. SelfCheckGPT: zero-resource black-box hallucination detection for generative large language models. Proc Conf on Empirical Methods in Natural Language Processing, p.9004-9017.

[19]Nakagawa K, Hirano M, Fujimoto Y, 2024. Evaluating company-specific biases in financial sentiment analysis using large language models. IEEE Int Conf on Big Data, p.6614-6623.

[20]Ribeiro MT, Singh S, Guestrin C, 2016. “Why should I trust you?”: explaining the predictions of any classifier. Proc 22^nd ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining, p.1135-1144.

[21]Sharma M, Tong M, Mu J, et al., 2025. Constitutional classifiers: defending against universal jailbreaks across thousands of hours of red teaming.

[22]Shen XY, Chen ZY, Backes M, et al., 2024. “Do anything now”: characterizing and evaluating in-the-wild jailbreak prompts on large language models. Proc ACM SIGSAC Conf on Computer and Communications Security, p.1671-1685.

[23]Simpson S, Nukpezah J, Brooks K, et al., 2025. Parity benchmark for measuring bias in LLMs. AI Ethics, 5(3):3087-3101.

[24]Tatsat H, Shater A, 2025. Beyond the black box: interpretability of LLMs in finance.

[25]Wu Z, Wang J, Zou C, et al., 2025. Towards competent AI for fundamental analysis in finance: a benchmark dataset and evaluation. https://arxiv.org/abs/2506.07315

[26]Yan SQ, Gu JC, Zhu Y, et al., 2024. Corrective retrieval augmented generation.

[27]Yu JH, Lin XW, Yu Z, et al., 2024. GPTFUZZER: red teaming large language models with auto-generated jailbreak prompts.

[28]Zhang YX, Zhou F, 2024. Bias mitigation in fine-tuning pre-trained models for enhanced fairness and efficiency.

[29]Zhou YJ, Han YF, Zhuang HM, et al., 2025. Defending jailbreak prompts via in-context adversarial game.

[30]Zou A, Wang Z, Kolter JZ, et al., 2023. Universal and transferable adversarial attacks on aligned language models.