An Evaluation of Test Suite Minimization Techniques

Year	Authors	Publisher	DOI
2020	Noemmer, Raphael Haas, Roman	Springer	10.1007/978-3-030-35510-4_4

Year

Authors

Publisher

DOI

2020

Noemmer, Raphael Haas, Roman

Springer

10.1007/978-3-030-35510-4_4

Techniques	Applicability
TSR	Industry Author

Techniques

Applicability

TSR

Industry Author

Experiment subject(s)	Industrial Partner	Programming Language
Open-source Java projects (up to 14770 TCs) Open-source, very large scale	CQSE (Germany)	Java

Experiment subject(s)

Industrial Partner

Programming Language

Open-source Java projects (up to 14770 TCs) Open-source, very large scale

CQSE (Germany)

Java

Effectiveness Metrics	Efficiency Metrics	Other Metrics
Selection/reduction count/percentage, Testing time, Fault Detection Loss

Effectiveness Metrics

Efficiency Metrics

Other Metrics

Selection/reduction count/percentage, Testing time, Fault Detection Loss

Information Approach	Algorithm Approach	Open Challenges
Coverage-based	Greedy	Investigate multiple objective-based algorithms; increase variety of study subjects; evaluate TSR with historical data rather than mutation testing.

Information Approach

Algorithm Approach

Open Challenges

Coverage-based

Greedy

Investigate multiple objective-based algorithms; increase variety of study subjects; evaluate TSR with historical data rather than mutation testing.

Abstract

As a software project evolves over time, the associated test suite usually grows with it. If test suites are not carefully maintained, this can easily result in massive test execution duration, reducing the benefits of regression testing because faults are found later in development or even after release. Test suite minimization aims to combat long running test suites by removing redundant test cases. Previous work mainly evaluates test suite minimization techniques based on comparably small projects, which are less practically relevant. In this paper, we compare four test suite minimization techniques by applying them to several open source software projects and evaluate the results. We find that the size and execution time of all the test suites can be reduced by over 70% on average. However, there is a substantial loss in fault detection capability of, on average, around 12.5%, restricting the applicability of this form of test suite minimization. © Springer Nature Switzerland AG 2020.