Techniques |
Applicability |
TCP
|
Industry Motivation
Industry Evaluation
|
Experiment subject(s) |
Industrial Partner |
Programming Language |
GSDTSR (up to 2.4k requests per minute), data scraped from the Chromium project (up to 149 requests per minute)
Industrial open-source, large scale |
|
Unclear |
Effectiveness Metrics |
Efficiency Metrics |
Other Metrics |
Time/tests To First Failure
|
|
|
Information Approach |
Algorithm Approach |
Open Challenges |
Fault-based
|
|
Evaluate on other projects; continue to examine failure distributions; explore use of other prioritization techniques.
|
Abstract
New changes are constantly and concurrently being made to large software systems. In modern continuous integration and deployment environments, each change requires a set of tests to be run. This volume of tests leads to multiple test requests being made simultaneously, which warrant prioritization of such requests. Previous work on test prioritization schedules queued tests at set time intervals. However, after a test has been scheduled it will never be reprioritized even if new higher risk tests arrive. Furthermore, as each test finishes, new information is available which could be used to reprioritize tests. In this work, we use the conditional failure probability among tests to reprioritize tests after each test run. This means that tests can be reprioritized hundreds of times as they wait to be run. Our approach is scalable because we do not depend on static analysis or coverage measures and simply prioritize tests based on their co-failure probability distributions. We named this approach CODYNAQ and in particular, we propose three prioritization variants called CODYNAQSINGLE, CODYNAQDOUBLE and CODYNAQFLEXI. We evaluate our approach on two data sets, CHROME and Google testing data. We find that our co-failure dynamic re-prioritization approach, CODYNAQ, outperforms the default order, FIFOBASELINE, finding the first failure and all failures for a change request by 31% and 62% faster, respectively. CODYNAQ also outperforms GOOGLETCP by finding the first failure 27% faster and all failures 62% faster. © 2018 IEEE.