Test Case Prioritization

Given a test suite T, the set of permutations of T, PT, and a function from PT to real numbers f: PT → R. Our program is to find $T^{\prime }\in PT$ such that $(\forall T^{\prime \prime })(T^{\prime \prime }\in PT)(T^{\prime \prime }\ne T^{\prime })[f(T^{\prime })\ge f(T^{\prime \prime })]$

This concerns ordering test cases for the early maximization of some desirable properties like the rate of fault detection. So this is concerned with the order in which tests run. It seems like the general idea is that we want to start with tests that reveal the most faults.

Code Coverage is usually a metric used as a prioritization criteria because the idea is that by maximizing structural coverage, we will also maximize fault detection. Therefore, while the goal of prioritization remains that of achieving a higher fault detection rate, it is actually to maximize early coverage. Different algorithms include:

• Random
• Hillclimbing
• Genetic Algorithms
• Greedy Algorithms

Historical Failure Probability

When tests fail, adjust the failure probability of queued tests. Is a test suite fails, how does it compare to other tests failing (Co-Failure)? Run the most likely test to fail.

Like other priority systems, we can have starvation where some tests are never executed. We have a dispatch queue and a waiting queue, if the dispatch queue is empty the tests in the beginning of the waiting queue will go into the dispatch queue in the original order. If the dispatch queue is smaller than a threshold, tests in the head of the waiting queue will be picked and put in the dispatch queue in a random order.