Resource Constrained Test Case Prioritization with Simulated Annealing in an Industrial Context
2024 (English)In: 39TH ANNUAL ACM SYMPOSIUM ON APPLIED COMPUTING, SAC 2024, ASSOC COMPUTING MACHINERY , 2024, p. 1694-1701Conference paper, Published paper (Refereed)
Abstract [en]
We need to find an effective prioritization of regression test cases due to their growing number. This may happen on parallel test systems and software branches. We compared regression test prioritization approaches against several goals of importance in an industrial context. We experimentally compared different simulated annealing approaches, hypothetical ideal and worst prioritizations, as well as reference prioritizations such as random, historical failure rate, age, etc. These were evaluated against a heuristic metric that combines several factors, as well as reference metrics such as failure count, days since last execution, etc. By simulating resource starvation in terms of available time, we found that some approaches rapidly degraded, e.g., by only prioritizing recently failed tests, the average number of nights since last execution was about five times as bad as for a random selection. The simulated annealing approach with large search space and many iterations came out best for many metrics. Interestingly, the poorest prioritization was achieved by aiming at diversity, and the coverage-based prioritization was poor at finding failures.
Place, publisher, year, edition, pages
ASSOC COMPUTING MACHINERY , 2024. p. 1694-1701
Keywords [en]
software testing; test case prioritization
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:liu:diva-206950DOI: 10.1145/3605098.3635971ISI: 001236958200242ISBN: 9798400702433 (print)OAI: oai:DiVA.org:liu-206950DiVA, id: diva2:1892559
Conference
39th Annual ACM Symposium on Applied Computing (SAC), Univ Salamanca, Avila, SPAIN, apr 08-12, 2024
Note
Funding Agencies|Swedish Knowledge Foundation [20150277]; AIDOaRt project, an ECSEL Joint Undertaking (JU) [101007350]; Westermo
2024-08-272024-08-272024-11-22