Test selection techniques are modification aware. Test cases are selected because they are relevant to the changed part of software which typically involves a white-box static analysis of some sort.
What to select
To reduce the time involved in re-testing, we only want to select tests from T that will produce different output when run on P’. These are called modification revealing tests.
Modification Revealing and Traversing
Summary
A test is modification revealing if it produces different outputs in P and P’
Although we cannot directly know if a test is modification revealing, we can define tests that execute modified code as modification traversing if it executes a new or modified statement in P’ or misses a statement in P’ that is executed in P.
You can identify the fault-revealing test cases for P’ by finding the modification revealing test cases for P and P’.
- P -Correct-for-T Assumption: It is assumed that for each test case t in T, when P was tested with t, P halted and produced the correct output
- Obsolete-Test-Identification Assumption: It is assumed that there exists a way to find test cases t that are obsolete for P’. From these assumptions, it is clear that every test case in T terminates and produces correct output for P while also producing the same output for P’. Therefore, a modification revealing test is also fault revealing. A weaker criterion here is using modification-traversing tests.
A third assumption here is the Controlled-Regression-Testing Assumption which states that when P’ is tested with t, all factors that might influence the output of P’ are held constant. Given that this holds, a non-obsolete test t can only be modification-revealing iff it is also modification-traversing.
Therefore we have this relationship if the P-Correct-for-T assumption also holds:
- , the key here is to find MT, or MR
Evaluation Criteria
- Suppose T contains n modification-revealing tests and S selects m of those tests. The inclusiveness of S is calculated as m/n
- If a method S always selects all modification-revealing tests, we call S safe
- Measure the extent to which a selective strategy omits tests that are non-modification-revealing
- Suppose T contains n non-modification-revealing tests and S omits m of these tests. The precision of S is m/n
Based on Dataflow
- Select test cases that exercise def-use paths from where a variable is defined and where its used without being redefined in between.
Based on Code
- The point here is to figure out what existing test cases need to be rerun after you fix bugs. You need to rerun tests that execute changed code.
Based on Slicing
This is a smart approach to regression test selection. We want to figure out if a changed line actually affects the output for a given test. We can compute a program slice which is a subset of statements that can influence the output for a given test case. This gives us a smaller subset of tests that need to be run.
Slicing
A program slice consists of all statements including conditions in the program that might affect the value of variable V and point P.
A backwards slice refers to statement fragments that contribute to the value of v at statement n. This is basically working backwards to figure out what the value of something is at some point in time.
A forward slice refers to all the program statements that are affected by the value of v and statement n. Refers to the predicate uses and computation uses of the variable v. Here, we pick a statement and see what statements changing it could affect.
Dynamic slicing can be used to assist debugging by simplifying the program. The slices constructed by slicing can be very large. A dynamic slice is constructed with respect to the traditional static slicing criterion together with dynamic information.
- Variables to be slices
- The point of interest within the program
- Sequence of input values for which the program is executed Unlike static slicing which considers all possible executions, dynamic slicing considers only what actually happens for a specific input. These are more precise which makes them better for debugging a specific failing test case but you need a concrete execution trace since they tell us about a particular run.
Note
There are two main properties desirable in slicing algorithms which we call soundness and completeness.
In order to be sound, an algorithm must never delete a statement from the original program which could have an effect upon the slicing criterion. This property allows us to analyze a slice in total confidence that it contains all relevant statements.
Soundness on its own is not enough, to be complete, a slicing algorithm must remove all statements which cannot affect the slicing criterion. In general, completeness is unachievable. Therefore, the goal of slicing algorithms is to delete as many statements as possible without giving up soundness. We can approach completeness but we can’t achieve it. A dynamic slice constructed for some variables at a certain point will always be at least as thin as the static slice.