Applying Cluster-based Approach to Improve the Effectiveness of Test Suite Reduction

doi:10.23940/ijpe.22.01.p1.110

Abstract

Abstract: Regression testing is an activity that ensures software quality as new features develop and needs to be conducted during software development. Practically, regression test suites have been growing with the development of software products. Test suite reduction is a classic technique that speeds up the regression test process by removing redundant test cases in the regression test suite. Thus, the execution time of the regression tests and software development costs can be effectively reduced. However, the method of test suite reduction may be time-consuming with large software or test suites and may lose some essential test cases while using only one testing criteria. The aim of this paper is to reduce test suite size with the same or higher fault detection capability by applying cluster-based test suite reduction (CB-TSR) methods with two testing criteria during regression testing. First, we applied three cluster algorithms to cluster test cases based on the similarity of function coverage and considered the silhouette coefficient to find the best cluster result. Second, two test suite reduction algorithms were applied with statement coverage against each cluster of the test case. Experiments based on real subject programs were evaluated using fault detection effectiveness (FDE) loss and other comparison criteria. Experimental results indicate that our proposed CB-TSR methods provide a lower processing cost and improves effectiveness with the same or better fault detection capability of the reduced test suite.

Key words: test suite reduction, regression test, cluster algorithm, testing criteria, code coverage

Chen-Hua Lee and Chin-Yu Huang. Applying Cluster-based Approach to Improve the Effectiveness of Test Suite Reduction [J]. Int J Performability Eng, 2022, 18(1): 1-10.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

References

1. Wong W.E., Horgan J.R., London S., andAgrawal H.A Study of Effective Regression Testing in Practice. In PROCEEDINGS The Eighth International Symposium On Software Reliability Engineering, IEEE, pp. 264-274, 1997.
2. Wong W.E., Li X., andLaplante P.A.Be More Familiar with Our Enemies and Pave the Way Forward: a Review of the Roles Bugs Played in Software Failures. Journal of Systems and Software, vol. 133, pp. 68-94, 2017.
3. Wong W.E., Debroy V., Surampudi A., Kim H., andSiok, M.F. Recent Catastrophic Accidents: Investigating How Software Was Responsible. In2010 Fourth International Conference on Secure Software Integration and Reliability Improvement, IEEE, pp. 14-22, 2010.
4. Bueno P.M., Jino M., andWong W.E.Diversity Oriented Test Data Generation Using Metaheuristic Search Techniques. Information Sciences, vol. 259, pp. 490-509, 2014.
5. Rothermel G., Untch R.H., Chu C., andHarrold M.J.Test Case Prioritization: An Empirical Study. In Proceedings IEEE International Conference on Software Maintenance-1999 (ICSM'99).'Software Maintenance for Business Change'(Cat. No. 99CB36360), pp. 179-188, IEEE, 1999.
6. Gao R., Eo J.S., Wong W.E., Gao X., andLee S.Y.An Empirical Study of Requirements-based Test Generation on an Automobile Control System. In Proceedings of the 29th Annual ACM Symposium on Applied Computing, pp. 1094-1099, 2014.
7. Wong W.E., Horgan J.R., London S., andMathur A.P.Effect of Test Set Minimization on Fault Detection Effectiveness. Software: Practice and Experience, vol. 28, no. 4, pp. 347-369, 1998.
8. Chiang C.L., Huang C.Y., Chiu C.Y., Chen K.W., andLee C.H.Analysis and Assessment of Weighted Combinatorial Criterion for Test Suite Reduction. Quality and Reliability Engineering International, 2021.
9. Chvatal V.A Greedy Heuristic for the Set-covering Problem. Mathematics of operations research, vol. 4, no. 3, pp. 233-235, 1979
10. Harrold, M.J., Gupta, R. and Soffa, M.L.A Methodology for Controlling the Size of a Test Suite. ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 2, no. 3, pp. 270-285, 1993.
11. Chen, T.Y. and Lau, M.F.A New Heuristic for Test Suite Reduction. Information and Software Technology, vol.40, no. 5-6, pp. 347-354, 1998.
12. Chen, T.Y. and Lau, M.F.A Simulation Study on Some Heuristics for Test Suite Reduction. Information and Software Technology, vol. 40, no. 13, pp. 777-787, 1998.
13. Yoo S., Harman M., Tonella P., andSusi A.Clustering Test Cases to Achieve Effective and Scalable Prioritisation Incorporating Expert Knowledge. In Proceedings of the eighteenth international symposium on Software testing and analysis, pp. 201-212, 2019.
14. Kanungo T., Mount D.M., Netanyahu N.S., Piatko C.D., Silverman R., andWu A.Y.A Local Search Approximation Algorithm for K-means Clustering. Computational Geometry, vol.28, no. 2-3, pp. 89-112, 2004.
15. Steinbach M., Karypis G., andKumar V. A Comparison of Document Clustering Techniques, 2000.
16. Yadav, D.K. and Dutta, S.K.Test Case Prioritization Using Clustering Approach for Object Oriented Software. International Journal of Information System Modeling and Design (IJISMD), vol. 10, no. 3, pp. 92-109, 2019.
17. Arafeen, M.J. and Do, H. Test Case Prioritization Using Requirements-based Clustering. In2013 IEEE sixth international conference on software testing, verification and validation, IEEE, pp. 312-321, 2013.
18. Elbaum S., Rothermel G., Kanduri S. and Malishevsky A.G.Selecting a Cost-effective Test Case Prioritization Technique. Software Quality Journal, vol. 12, no. 3, pp. 185-210, 2004.
19. Huang Z.Extensions to the K-means Algorithm for Clustering Large Data Sets with Categorical Values. Data mining and knowledge discovery, vol. 2, no. 3, pp. 283-304, 1998.
20. Madhuri R., Murty M.R., Murthy J.V.R., Reddy, P.P., and Satapathy, S.C. Cluster Analysis on Different Data Sets Using K-modes and K-prototype Algorithms. In ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India-Vol II. Springer, Cham, pp. 137-144, 2014.
21. He, Z., Deng, S. and Xu, X.Improving K-modes Algorithm Considering Frequencies of Attribute Values in Mode. In International Conference on Computational and Information Science, Springer, Berlin, Heidelberg, pp. 157-162, 2005.
22. Sangam, R.S. and Om, H.The K-modes Algorithm with Entropy Based Similarity Coefficient. Procedia Computer Science, vol. 50, pp. 93-98, 2015.
23. Mahmoodi M., Tavassoli S.H., Takayama O., Sukham J., Malureanu R., andLavrinenko A.V.Existence Conditions of High‐k Modes in Finite Hyperbolic Metamaterials. Laser & Photonics Reviews, vol. 13, no. 3, pp. 1800253, 2019.
24. Xiao Y., Huang C., Huang J., Kaku I. and Xu Y.Optimal Mathematical Programming and Variable Neighborhood Search for K-modes Categorical Data Clustering. Pattern Recognition, vol. 90, pp. 183-195, 2019.
25. Schütze H., Manning C.D., andRaghavan P.Introduction to information retrieval, Cambridge: Cambridge University Press, vol. 39, pp. 234-265, 2008
26. Gowda, K.C. and Krishna, G.Agglomerative Clustering Using the Concept of Mutual Nearest Neighbourhood. Pattern recognition, vol. 10, no. 2, pp. 105-112, 1978.
27. Franti, P., Virmajoki, O. and Hautamaki, V.Fast Agglomerative Clustering Using a K-nearest Neighbor Graph. IEEE transactions on pattern analysis and machine intelligence, vol. 28, no. 11, pp. 1875-1881, 2006.
28. Zhou, H.B. and Gao, J.T.Automatic Method for Determining Cluster Number Based on Silhouette Coefficient. In Advanced Materials Research, Trans Tech Publications Ltd, vol.951, pp. 227-230. 2014.
29. Dinh, D.T., Fujinami, T. and Huynh, V.N.Estimating the Optimal Number of Clusters in Categorical Data Clustering by Silhouette Coefficient. In International Symposium on Knowledge and Systems Sciences, Springer, Singapore, pp. 1-17, 2019.
30. Aranganayagi, S. and Thangavel, K. Clustering categorical data using silhouette coefficient as a relocating measure. In International conference on computational intelligence and multimedia applications (ICCIMA2007), IEEE, vol. 2, pp. 13-17, 2007.
31. Cruciani E., Miranda B., Verdecchia R. and Bertolino A. Scalable Approaches for Test Suite Reduction. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), IEEE, pp. 419-429, 2019.
32. Hutchins M., Foster H., Goradia T. and Ostrand T.Experiments on the Effectiveness of Dataflow-and Control-flow-based Test Adequacy Criteria. In Proceedings of 16th International conference on Software engineering, IEEE, pp. 191-200, 1994.
33. Just, R., Jalali, D. and Ernst, M.D. Defects4j: a Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. In Proceedings of the2014 International Symposium on Software Testing and Analysis, pp. 437-440, 2014.
34. Software-artifact Infrastructure Repository: Home. http://sir.csc.ncsu.edu/portal/ (accessed Dec. 11, 2020.
35. R. Just, rjust/defects4j. https://github.com/rjust/defects4j (accessed Mar. 06, 2021).
36. Gcov (Using the GNU Compiler Collection (GCC)). https://gcc.gnu.org/onlinedocs/gcc/Gcov.html (accessed Dec. 11, 2020).
37. PIT Mutation Testing. https://pitest.org/ (accessed Mar. 24, 2021).
38. Shi A., Yung T., Gyori A. and Marinov D. Comparing and Combining Test-suite Reduction and Regression Test Selection. In Proceedings of the2015 10th joint meeting on foundations of software engineering, pp. 237-247, 2015.
39. Zhang L., Marinov D., Zhang L., andKhurshid, S. An Empirical Study of Junit Test-suite Reduction. In2011 IEEE 22nd International Symposium on Software Reliability Engineering, IEEE, pp. 170-179, 2011.
40. Li Y., Wong W.E., Lee S.Y. and Wotawa F.Using Tri-relation Networks for Effective Software Fault-proneness Prediction. IEEE Access, vol. 7, pp. 63066-63080, 2019.
41. Lin, J.W. and Huang, C.Y.Analysis of test suite reduction with enhanced tie-breaking techniques. Information and Software Technology, vol. 51, no. 4, pp. 679-690, 2009
42. Rothermel G., Harrold M.J., Von Ronne, J., and Hong, C. Empirical Studies of Test‐suite Reduction. Software Testing, Verification and Reliability, vol. 12, no. 4, pp. 219-249, 2002.
43. Jeffrey, D. and Gupta, N.Improving Fault Detection Capability by Selectively Retaining Test Cases During Test Suite Reduction. IEEE Transactions on software Engineering, vol. 33, no. 2, pp. 108-123, 2007.

[1]	Chia-En Lai and Chin-Yu Huang. Developing a Modified Fuzzy-GE Algorithm for Enhanced Test Suite Reduction Effectiveness [J]. Int J Performability Eng, 2023, 19(4): 223-233.
[2]	Tingting Huo, Yan Zhang, Chunyan Xia, Zijiang Yang, Weisong Sun. Large-Scale Test Case Prioritization using Viterbi Algorithm [J]. Int J Performability Eng, 2020, 16(12): 1921-1932.