Abstract: In practical reverse engineering, stripped and optimized binaries lack high-level semantics, hindering automated function understanding. This paper adopts binary function name prediction as a benchmark to evaluate open-source large language models (LLMs) for function-level semantic inference under realistic conditions. We systematically examine key factors affecting performance, including the pretraining domain, model scale, architecture and optimization settings, prompting, and contextual information. Experiments on a large-scale multi-project binary dataset reveal clear limitations in recovering function semantics from stripped binaries. Code-oriented LLMs consistently outperform general-purpose models, while increasing the model size alone does not yield monotonic gains. We further show that realistically recoverable contextual signals, especially structurally recoverable cross-function contexts, substantially mitigate semantic sparsity and improve prediction quality. These results delineate current capability boundaries of LLM-based binary semantic understanding and suggest future directions in contextual modeling and domain-adaptive techniques for practical reverse engineering.

面向二进制函数名预测的大语言模型性能评测

[1]Ahmad W, Chakraborty S, Ray B, et al., 2021. Unified pre-training for program understanding and generation. Proc Conf North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p.2655-2668.

[2]Allamanis M, Barr ET, Bird C, et al., 2015. Suggesting accurate method and class names. Proc 10^th Joint Meeting on Foundations of Software Engineering, p.38-49.

[3]Alon U, Brody S, Levy O, et al., 2019a. code2seq: generating sequences from structured representations of code. 7^th Int Conf on Learning Representations.

[4]Alon U, Zilberstein M, Levy O, et al., 2019b. code2vec: learning distributed representations of code. Proc ACM Programm Lang, 3:40.

[5]Bhattacharya P, Chakraborty M, Palepu KNSN, et al., 2023. Exploring large language models for code explanation.

[6]Boronat A, Mustafa J, 2025. MDRE-LLM: a tool for analyzing and applying LLMs in software reverse engineering. IEEE Int Conf on Software Analysis, Evolution and Reengineering, p.850-854.

[7]Chen M, Tworek J, Jun H, et al., 2021. Evaluating large language models trained on code.

[8]David Y, Alon U, Yahav E, 2020. Neural reverse engineering of stripped binaries using augmented control flow graphs. Proc ACM Programm Lang, 4(OOPSLA):225.

[9]Gao H, Cheng SY, Xue YX, et al., 2021. A lightweight framework for function name reassignment based on large-scale stripped binaries. Proc 30^th ACM SIGSOFT Int Symp on Software Testing and Analysis, p.607-619.

[10]GitHub, 2008. GitHub: a Platform for Hosting and Collaborating on Software Development. https://github.com/ [Accessed on Jan. 11, 2026].

[11]Guo DY, Ren S, Lu S, et al., 2020. GraphCodeBERT: pre-training code representations with data flow.

[12]Han K, Lim JH, Im EG, 2013. Malware analysis method using visualization of binary files. Proc Research in Adaptive and Convergent Systems, p.317-321.

[13]Harzevili NS, Belle AB, Wang JJ, et al., 2023. A survey on automated software vulnerability detection using machine learning and deep learning. https://arxiv.org/abs/2306.11673

[14]Høst EW, Østvold BM, 2009. Debugging method names. 23^rd European Conf on Object-Oriented Programming, p.294-317.

[15]Hu XY, Fu ZW, Xie SC, et al., 2025. SoK: potentials and challenges of large language models for reverse engineering.

[16]Hugging Face, 2016. Hugging Face. https://huggingface.co/ [Accessed on Jan. 15, 2026].

[17]Jaffal NO, Alkhanafseh M, Mohaisen D, 2025. Large language models in cybersecurity: a survey of applications, vulnerabilities, and defense techniques. AI, 6(9):216.

[18]Jiang LX, Jin X, Lin ZQ, 2025. Beyond classification: inferring function names in stripped binaries via domain adapted LLMs. Proc ACM SIGSAC Conf on Computer and Communications Security.

[19]Jin X, Pei KX, Won JY, et al., 2022. SymLM: predicting function names in stripped binaries via context-sensitive execution-aware code embeddings. Proc ACM SIGSAC Conf on Computer and Communications Security, p.1631-1645.

[20]Kaplan J, McCandlish S, Henighan T, et al., 2020. Scaling laws for neural language models. https://arxiv.org/abs/2001.08361

[21]Kim H, Bak J, Cho K, et al., 2023. A Transformer-based function symbol name inference model from an assembly language for binary reversing. Proc ACM Asia Conf on Computer and Communications Security, p.951-965.

[22]Lawrie D, Morrell C, Feild H, et al., 2006. What’s in a name? A study of identifiers. 14^th IEEE Int Conf on Program Comprehension, p.3-12.

[23]Meng XZ, Miller BP, 2016. Binary code is not easy. Proc 25^th Int Symp on Software Testing and Analysis, p.24-35.

[24]OpenAI, 2023. OpenAI API Documentation. https://developer.neureality.ai/docs/user-guide/openai-api-doc.html [Accessed on Jan. 11, 2026].

[25]Patrick-Evans J, Cavallaro L, Kinder J, 2020. Probabilistic naming of functions in stripped binaries. Proc 36^th Annual Computer Security Applications Conf, p.373-385.

[26]Sha ZH, Wang H, Gao ZY, et al., 2025. llasm: naming functions in binaries by fusing encoder-only and decoder-only LLMs. ACM Trans Softw Eng Methodol, 34(4):93.

[27]Shang XW, Cheng SY, Chen GQ, et al., 2024. How far have we gone in binary code understanding using large language models. IEEE Int Conf on Software Maintenance and Evolution, p.1-12.

[28]Shin ECR, Song D, Moazzezi R, 2015. Recognizing functions in binaries with neural networks. 24^th USENIX Conf on Security Symp, p.611-626.

[29]Song YF, Zhang DD, Wang J, et al., 2025. Application of deep learning in malware detection: a review. J Big Data, 12(1):99.

[30]Tan HZ, Luo Q, Li J, et al., 2024. LLM4Decompile: decompiling binary code with large language models. Proc Conf on Empirical Methods in Natural Language Processing, p.3473-3487.

[31]Tian JF, Xing WJ, Li Z, 2020. BVDetector: a program slice-based binary code vulnerability intelligent detection system. Inform Softw Technol, 123:106289.

[32]vLLM Project, 2023. vLLM: a high-throughput and memory-efficient inference and serving engine for LLMs. https://github.com/vllm-project/vllm [Accessed on Jan. 13, 2026].

[33]Wang Y, Wang WS, Joty S, et al., 2021. CodeT5: identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. Proc Conf on Empirical Methods in Natural Language Processing, p.8696-8708.

[34]Wong WK, Wu D, Wang H, et al., 2025. DecLLM: LLM-augmented recompilable decompilation for enabling programmatic use of decompiled code. Proc ACM Softw Eng, 2:ISSTA081.

[35]Xia B, Pang JM, Wang J, et al., 2021. Study on binary code evolution with concrete semantic analysis. 7^th Int Conf of Pioneering Computer Scientists, Engineers and Educators, p.30-43.

[36]Xia B, Ge YX, Yang RN, et al., 2023. BContext2Name: naming functions in stripped binaries with multi-label learning and neural networks. IEEE 10^th Int Conf on Cyber Security and Cloud Computing and IEEE 9^th Int Conf on Edge Computing and Scalable Cloud, p.167-172.

[37]Xu HX, Wang SN, Li NK, et al., 2025. Large language models for cyber security: a systematic literature review. ACM Trans Softw Eng Methodol, in press.

[38]Yao YF, Duan JH, Xu KD, et al., 2024. A survey on large language model (LLM) security and privacy: the good, the bad, and the ugly. High-Confid Comput, 4(2):100211.

[39]Zou MQ, Cai HY, Wu HW, et al., 2025. D-LiFT: improving LLM-based decompiler backend via code quality-driven fine-tuning.

Similar articles