Full Text:   <408>

Summary:  <9>

Suppl. Mater.: 

CLC number: TP311

On-line Access: 2022-05-19

Received: 2021-09-30

Revision Accepted: 2022-05-19

Crosschecked: 2022-01-11

Cited: 0

Clicked: 386

Citations:  Bibtex RefMan EndNote GB/T7714


Fanyi MENG


Hai YU


-   Go to

Article info.
Open peer comments

Frontiers of Information Technology & Electronic Engineering  2022 Vol.23 No.5 P.692-714


Devising optimal integration test orders using cost–benefit analysis

Author(s):  Fanyi MENG, Ying WANG, Hai YU, Zhiliang ZHU

Affiliation(s):  Software College, Northeastern University, Shenyang 110169, China; more

Corresponding email(s):   yuhai@mail.neu.edu.cn

Key Words:  Integration test order, Cost–, benefit analysis, Probabilistic risk analysis, Complex network

Fanyi MENG, Ying WANG, Hai YU, Zhiliang ZHU. Devising optimal integration test orders using cost–benefit analysis[J]. Frontiers of Information Technology & Electronic Engineering, 2022, 23(5): 692-714.

@article{title="Devising optimal integration test orders using cost–benefit analysis",
author="Fanyi MENG, Ying WANG, Hai YU, Zhiliang ZHU",
journal="Frontiers of Information Technology & Electronic Engineering",
publisher="Zhejiang University Press & Springer",

%0 Journal Article
%T Devising optimal integration test orders using cost–benefit analysis
%A Fanyi MENG
%A Ying WANG
%A Hai YU
%A Zhiliang ZHU
%J Frontiers of Information Technology & Electronic Engineering
%V 23
%N 5
%P 692-714
%@ 2095-9184
%D 2022
%I Zhejiang University Press & Springer
%DOI 10.1631/FITEE.2100466

T1 - Devising optimal integration test orders using cost–benefit analysis
A1 - Fanyi MENG
A1 - Ying WANG
A1 - Hai YU
A1 - Zhiliang ZHU
J0 - Frontiers of Information Technology & Electronic Engineering
VL - 23
IS - 5
SP - 692
EP - 714
%@ 2095-9184
Y1 - 2022
PB - Zhejiang University Press & Springer
ER -
DOI - 10.1631/FITEE.2100466

Integration testing is an integral part of software testing. Prior studies have focused on reducing test cost in integration test order generation. However, there are no studies concerning the testing priorities of critical classes when generating integration test orders. Such priorities greatly affect testing efficiency. In this study, we propose an effective strategy that considers both test cost and efficiency when generating test orders. According to a series of dynamic execution scenarios, the software is mapped into a multi-layer dynamic execution network (MDEN) model. By analyzing the dynamic structural complexity, an evaluation scheme is proposed to quantify the class testing priority with the defined class risk index. cost–;benefit analysis is used to perform cycle-breaking operations, satisfying two principles: assigning higher priorities to higher-risk classes and minimizing the total complexity of test stubs. We also present a strategy to evaluate the effectiveness of integration test order algorithms by calculating the reduction of software risk during their testing process. Experiment results show that our approach performs better across software of different scales, in comparison with the existing algorithms that aim only to minimize test cost. Finally, we implement a tool, ITOsolution, to help practitioners automatically generate test orders.


摘要:集成测试是面向对象软件测试的重要组成部分。传统的类级集成测试顺序研究策略大多围绕如何降低测试成本开展工作,并未考虑赋予可靠性风险较大的节点较高测试优先级,从而影响软件测试效率。本文提出一种兼顾测试成本与测试效率的方法生成集成测试序列。根据软件在不同场景下的运行状态,将其映射成多层动态执行网络(multi-layer dynamic execution network,MDEN)。借助该网络模型与概率风险评估方法为软件中每一个类赋予风险权重。利用成本收益分析方法,在生成测试用例的过程中保证两条原则:为高风险的类赋予较高权重,同时最小化测试桩复杂度。在此基础上,分析测试序列对软件系统总体运行风险的影响,从而提出评估测试序列优劣的度量方案。通过与现有算法的实验对比分析,证明所提算法生成的类级集成测试序列能有效降低测试代价。最后,将所提算法实现为自动生成集成测试序列的开源工具ITOsolution。


Darkslateblue:Affiliate; Royal Blue:Author; Turquoise:Article


[1]Abdurazik A, Offutt J, 2006. Coupling-based class integration and test order. Proc Int Workshop on Automation of Software Test, p.50-56.

[2]Al Mannai WI, Lewis TG, 2008. A general defender-attacker risk model for networks. J Risk Finance, 9(3):244-261.

[3]Amland S, 2000. Risk-based testing: risk analysis fundamentals and metrics for software testing including a financial application case study. J Syst Softw, 53(3):287-295.

[4]Assunção WKG, Colanzi TE, Vergilio SR, et al., 2014. A multi-objective optimization approach for the integration and test order problem. Inform Sci, 267:119-139.

[5]Bang L, Aydin A, Bultan T, 2015. Automatically computing path complexity of programs. Proc 10th Joint Meeting on Foundations of Software Engineering, p.61-72.

[6]Bansal P, Sabharwal S, Sidhu P, 2009. An investigation of strategies for finding test order during integration testing of object oriented applications. Proc Int Conf on Methods and Models in Computer Science, p.1-8.

[7]Binder RV, 1996. Testing object-oriented software: a survey. Softw Test Verif Reliab, 6(3-4):125-252.

[8]Bowring JF, Rehg JM, Harrold MJ, 2004. Active learning for automatic classification of software behavior. ACM SIGSOFT Softw Eng Notes, 29(4):195-205.

[9]Briand LC, Jie F, Labiche Y, 2002. Experimenting with genetic algorithms to devise optimal integration test orders. Technical Report, No. SCE-02-03, Carleton University, Ottawa, Canada.

[10]Briand LC, Labiche Y, Wang YH, 2003. An investigation of graph-based class integration test order strategies. IEEE Trans Softw Eng, 29(7):594-607.

[11]Cai KY, Yin BB, 2009. Software execution processes as an evolving complex network. Inform Sci, 179(12):1903-1928.

[12]da Veiga Cabral R, Pozo A, Vergilio SR, 2010. A Pareto ant colony algorithm applied to the class integration and test order problem. Proc 22nd IFIP WG 6.1 Int Conf on Testing Software and Systems, p.16-29.

[13]Deb K, Pratap A, Agarwal S, et al., 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans Evol Comput, 6(2):182-197.

[14]Floyd RW, 1962. Algorithm 97: shortest path. Commun ACM, 5(6):345.

[15]Frankl PG, Weyuker EJ, 2000. Testing software to detect and reduce risk. J Syst Softw, 53(3):275-286.

[16]Goseva-Popstojanova K, Hassan A, Guedem A, et al., 2003. Architectural-level risk analysis using UML. IEEE Trans Softw Eng, 29(10):946-960.

[17]Hao D, Zhang L, Zang L, et al., 2016. To be optimal or not in test-case prioritization. IEEE Trans Softw Eng, 42(5):490-505.

[18]Henry S, Kafura D, 1981. Software structure metrics based on information flow. IEEE Trans Softw Eng, 7(5):510-518.

[19]Huang CY, Lyu MR, 2005. Optimal testing resource allocation, and sensitivity analysis in software development. IEEE Trans Reliab, 54(4):592-603.

[20]Jiang SJ, Zhang YM, Li HY, et al., 2011. An approach for inter-class integration test order determination based on coupling measures. Chin J Comput, 34(6):1062-1074.

[21]Jiang SJ, Zhang M, Zhang YM, et al., 2021. An integration test order strategy to consider control coupling. IEEE Trans Softw Eng, 47(7):1350-1367.

[22]Jorgensen PC, Erickson C, 1994. Object-oriented integration testing. Commun ACM, 37(9):30-38.

[23]Knowles JD, Corne DW, 2000. Approximating the nondominated front using the Pareto archived evolution strategy. Evol Comput, 8(2):149-172.

[24]Kung D, Gao J, Hsia P, et al., 1995. A test strategy for object-oriented programs. Proc 19th Annual Int Computer Software and Applications Conf, p.239-244.

[25]Le Traon Y, Jeron T, Jezequel JM, et al., 2000. Efficient object-oriented integration and regression testing. IEEE Trans Reliab, 49(1):12-25.

[26]Lipow M, 1982. Number of faults per line of code. IEEE Trans Softw Eng, 8(4):437-439.

[27]McCabe TJ, 1976. A complexity measure. IEEE Trans Softw Eng, 2(4):308-320.

[28]Monden A, Hayashi T, Shinoda S, et al., 2013. Assessing the cost effectiveness of fault prediction in acceptance testing. IEEE Trans Reliab, 39(10):1345-1357.

[29]Myers CR, 2003. Software systems as complex networks: structure, function, and evolvability of software collaboration graphs. Phys Rev E, 68(2):046116.

[30]NASA, 1999. Pyroshock Test Crite. Technical Report, No. NASA-STD-7003A. NASA, Washington, USA.

[31]Sharma C, Sibal R, 2013. Application of different metaheuristic techniques for finding optimal test order during integration testing of object oriented systems and their comparative study. Int J Soft Comput Eng, 3(12):1-19.

[32]Tai KC, Daniels JF, 1999. Interclass test order for object-oriented software. J Obj-Orient Progr, 12(4):18-25.

[33]Vergilio SR, Pozo A, Árias JCG, et al., 2012. Multi-objective optimization algorithms applied to the class integration and test order problem. Int J Softw Tools Technol Transf, 14(4):461-475.

[34]Walters C, Ludwig D, 1994. Calculation of Bayes posterior probability distributions for key population parameters. Can J Fish Aquat Sci, 51(3):713-722.

[35]Wang Y, Zhu ZL, Yang B, et al., 2018a. Using reliability risk analysis to prioritize test cases. J Syst Softw, 139:14-31.

[36]Wang Y, Zhu ZL, Yu H, et al., 2018b. Risk analysis on multi-granular flow network for software integration testing. IEEE Trans Circ Syst II Expr Briefs, 65(8):1059-1063.

[37]Wang ZS, Li BX, Wang LL, et al., 2011. A brief survey on automatic integration test order generation. Proc 23rd Int Conf on Software Engineering & Knowledge Engineering, p.254-257.

[38]Weyuker EJ, 1988. Evaluating software complexity measures. IEEE Trans Softw Eng, 14(9):1357-1365.

[39]Xu C, Qin Y, Yu P, et al., 2020. Theories and techniques for growing software: paradigm and beyond. Sci Sin Inform, 50(11):1595-1611.

Open peer comments: Debate/Discuss/Question/Opinion


Please provide your name, email address and a comment

Journal of Zhejiang University-SCIENCE, 38 Zheda Road, Hangzhou 310027, China
Tel: +86-571-87952783; E-mail: cjzhang@zju.edu.cn
Copyright © 2000 - 2022 Journal of Zhejiang University-SCIENCE