liu.seSearch for publications in DiVA
Change search
ReferencesLink to record
Permanent link

Direct link
An Evaluation of Combination Strategies for Test Case Selection
Linköping University, Department of Computer and Information Science. Linköping University, The Institute of Technology.
School of Humanities and Informatics, University of Skövde, Skövde, Sweden.
Department of Information and Software Engineering, George Mason University, Fairfax, USA.
School of Humanities and Informatics, University of Skövde, Skövde, Sweden.
2006 (English)In: Journal of Empirical Software Engineering, ISSN 1382-3256, Vol. 11, no 4, 583-611 p.Article in journal (Refereed) Published
Abstract [en]

 This paper presents results from a comparative evaluation of five combination strategies. Combination strategies are test case selection methods that combine “interesting” values of the input parameters of a test subject to form test cases. This research comparatively evaluated five combination strategies; the All Combination strategy (AC), the Each Choice strategy (EC), the Base Choice strategy (BC), Orthogonal Arrays (OA) and the algorithm from the Automatic Efficient Test Generator (AETG). AC satisfies n-wise coverage, EC and BC satisfy 1-wise coverage, and OA and AETG satisfy pair-wise coverage. The All Combinations strategy was used as a “gold standard” strategy; it subsumes the others but is usually too expensive for practical use. The others were used in an experiment that used five programs seeded with 128 faults. The combination strategies were evaluated with respect to the number of test cases, the number of faults found, failure size, and number of decisions covered. The strategy that requires the least number of tests, Each Choice, found the smallest number of faults. Although the Base Choice strategy requires fewer test cases than Orthogonal Arrays and AETG, it found as many faults. Analysis also shows some properties of the combination strategies that appear significant. The two most important results are that the Each Choice strategy is unpredictable in terms of which faults will be revealed, possibly indicating that faults are found by chance, and that the Base Choice and the pair-wise combination strategies to some extent target different types of faults.

Place, publisher, year, edition, pages
2006. Vol. 11, no 4, 583-611 p.
Keyword [en]
Combination strategies, Orthogonal arrays, AETG, Test case selection, Testing experiment
National Category
Engineering and Technology
URN: urn:nbn:se:liu:diva-12971DOI: 10.1007/s10664-006-9024-2OAI: diva2:17564
Available from: 2008-02-29 Created: 2008-02-29
In thesis
1. Handling Combinatorial Explosion in Software Testing
Open this publication in new window or tab >>Handling Combinatorial Explosion in Software Testing
2007 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

In this thesis, the overall conclusion is that combination strategies, (i.e., test case selection methods that manage the combinatorial explosion of possible things to test), can improve the software testing in most organizations. The research underlying this thesis emphasizes relevance by working in close relationship with industry.

Input parameter models of test objects play a crucial role for combination strategies. These models consist of parameters with corresponding parameter values and represent the input space and possibly other properties, such as state, of the test object. Test case selection is then defined as the selection of combinations of parameter values from these models.

This research describes a complete test process, adapted to combination strategies. Guidelines and step-by-step descriptions of the activities in process are included in the presentation. In particular, selection of suitable combination strategies, input parameter modeling and handling of conflicts in the input parameter models are addressed. It is also shown that several of the steps in the test process can be automated.

The test process is validated through a set of experiments and case studies involving industrial testers as well as actual test problems as they occur in industry. In conjunction with the validation of the test process, aspects of applicability of the combination strategy test process (e.g., usability, scalability and performance) are studied. Identification and discussion of barriers for the introduction of the combination strategy test process in industrial projects are also included.

This research also presents a comprehensive survey of existing combination strategies, complete with classifications and descriptions of their different properties. Further, this thesis contains a survey of the testing maturity of twelve software-producing organizations. The data indicate low test maturity in most of the investigated organizations. Test managers are often aware of this but have trouble improving. Combination strategies are suitable improvement enablers, due to their low introduction costs.

Place, publisher, year, edition, pages
Linköping University, Department of Computer and Information Science, 2007
, Linköping studies in science and technology. Dissertations, ISSN 0345-7524 ; 1073
Input parameter models, software development, software testing, test process
National Category
Software Engineering
Research subject
urn:nbn:se:his:diva-1994 (URN)978-91-87515-74-9 (ISBN)
Public defence
Available from: 2008-04-21 Created: 2008-04-21 Last updated: 2013-03-26

Open Access in DiVA

No full text

Other links

Publisher's full textLink to Ph.D. thesis

Search in DiVA

By author/editor
Grindal, MatsLindström, BirgittaAndler, Sten F.
By organisation
Department of Computer and Information ScienceThe Institute of Technology
In the same journal
Journal of Empirical Software Engineering
Engineering and Technology

Search outside of DiVA

GoogleGoogle Scholar
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Altmetric score

Total: 73 hits
ReferencesLink to record
Permanent link

Direct link