Techniques |
Applicability |
TCS
|
Industry Motivation
Industry Evaluation
Industry Author
|
Experiment subject(s) |
Industrial Partner |
Programming Language |
Open-source C# projects (up to 183 TCs) plus one Microsoft project (37 "test modules")
Open-source, small scale
Industrial proprietary, unclear scale |
Microsoft (USA) |
C# |
Effectiveness Metrics |
Efficiency Metrics |
Other Metrics |
Selection/reduction count/percentage
|
Execution time, Total/End-to-end time
|
|
Information Approach |
Algorithm Approach |
Open Challenges |
Change-based
|
|
n/a
|
Abstract
Regression testing is used to check the correctness of evolving software. With the adoption of Agile development methodology, the number of tests and software revisions has dramatically increased, and hence has the cost of regression testing. Researchers proposed regression test selection (RTS) techniques that optimize regression testing by skipping tests that are not impacted by recent program changes. Ekstazi is one such state-of-the art technique; Ekstazi is implemented for the Java programming language and has been adopted by several companies and open-source projects.
We report on our experience implementing and evaluating Ekstazi#, an Ekstazi-like tool for .NET. We describe the key challenges of bringing the Ekstazi idea to the .NET platform. We evaluate Ekstazi# on 11 open-source projects, as well as an internal Microsoft project substantially larger than each of the open-source projects. Finally, we compare Ekstazi# to an incremental build system (also developed at Microsoft), which, out of the box, provides module-level dependency tracking and skipping tasks (including test execution) whenever dependencies of a task do not change between the current and the last successful build. Ekstazi# on average reduced regression testing time by 43.70% for the open-source projects and by 65.26% for the Microsoft project (the latter is in addition to the savings provided by incremental builds).