How often should you launch your test suites in order to find bugs fast at little cost? What is the optimal launch schedule?

1. Optimal Usage of the
Computer Farm
for Regression Testing
Daniel Hansson, Verifyter
Patrik Granath, Verifyter

2. Background
We need some science!
3/2/2016 Daniel Hansson, Verifyter 2
Why does the need for computer
farm resources increase so fast?
We run more stuff
...and my jobs are still
queued. Buy more!

3. Optimal
Setup
3/2/2016 Daniel Hansson, Verifyter 3
A B C D
8 AM
sanity sanity sanity 9
nightly 10
11
Report Launch 12 PM
Each 8h Each 8h 1
2
3
sanity 4
sanity 5
6
sanity 7
nightly 8
nightly 9
10
11
nightly nightly 0 AM
1
2
3
4
5
6
7
Which regression
test setup is the
most efficient?
At the end of the
presentation you
will know

4. Defining Metrics (1/2)
3/2/2016 Daniel Hansson, Verifyter 4
Metric Definition
Cost Total CPU test time
Bug Identification
Time
Time from a bug is committed to the
revision control system until a failure
is reported
Bug Fix Time Bug Identification Time + debug time
+ committing fix
Test Fail Ratio
(Quality)
The number of failing tests / total
number of tests run, for a given period

5. Defining Metrics (2/2)
3/2/2016 Daniel Hansson, Verifyter 5

6. 3/2/2016 Daniel Hansson, Verifyter 6
Quality
• Better Test Fail Ratio
(Quality) means
Earlier Release
• Shorter BIT and
Shorter Bug Fix Time
=> Earlier Release
• What is the optimal
setup?

7. Coverage (1/2)
Functional Coverage (%) vs Test Suite Run Time (h)
3/2/2016 Daniel Hansson, Verifyter 7
Doubling the cost (run time)
does not double the chance
of finding a bug (coverage)

8. Coverage (2/2)
3/2/2016 Daniel Hansson, Verifyter 8
Longer test suite means
longer bug identification time
at a higher cost, but only gives
some better coverage

9. Launch Frequency
3/2/2016 Daniel Hansson, Verifyter 9
Running twice
instead of once per
day matters a lot!
For higher launch
frequencies it is a case
of diminishing returns

10. Equations for 1 Test Suite
3/2/2016 Daniel Hansson, Verifyter 10
CPUs Average #CPU’s
L Launches
TSL Test Suite Length

11. Conclusions for 1 test suite
3/2/2016 Daniel Hansson, Verifyter 11
• The BIT looks similar for different cases
• The reason: the BIT depends mainly on #launches
• Conclusion: 2-5 launches/cycle has a good cost-
benefit ratio
Good cost-benefit ratioGood cost-benefit ratio

12. 2 Test Suites
3/2/2016 Daniel Hansson, Verifyter 12
• Assumption: Functional coverage of
sanity is a sub-set of the nightly coverage

13. 2 Test Suites
3/2/2016 Daniel Hansson, Verifyter 13
• There are 3 test suite sequences
– sanity => sanity
– sanity => nightly
– nightly => sanity
• Probability for a bug to be
committed during a test sequence
• BIT for each test sequence
2

14. 2 Test Suites
3/2/2016 Daniel Hansson, Verifyter 14
Probability for a bug
to appear in this part
of the cycle
Average BIT for
each of the 3
sequences

15. 3/2/2016 Daniel Hansson, Verifyter 15
Optimal
Scheduling
• The optimal launch schedule is
achieved when all max BIT’s are
equal
• This keeps the max BIT’s too a min
Optimal free time between two sanity runs
Optimal free time between one sanity run and
one nightly run

16. 3/2/2016 Daniel Hansson, Verifyter 16
Optimal
Scheduling (2/2)
• If the nightly test suite length is
equal or larger than max
BITStoS then nightly will never
first find a bug in the sanity
coverage tranche
• In this case it can be
scheduled independently of the
sanity runs

17. Interleaving
• Interleaving test suites
reduces the overall BIT
• This is because the nightly
runs will be better spread
out (lower BITnightly)
3/2/2016 Daniel Hansson, Verifyter 17
• BITsanity is not order dependent if
it is optimally scheduled
A B
• With 3 test suites, run the shortest
at least every second run

18. 3/2/2016 Daniel Hansson, Verifyter 18
1 Nightly Run, X Sanity Runs
• 1 Nightly and 2-4 Sanity: good cost-benefit ratio
• Better than running 2 Nightly runs

19. 3 Test Suites
3/2/2016 Daniel Hansson, Verifyter 19
Same pattern as
for 2 Test Suites

20. Summary
• Detect Bugs Fast, Fix Bugs Fast => Earlier release
• Launch Frequencies with good cost-benefit ratios:
– 1 Test Suite: Run the test suite 2-5x per cycle
– Multiple Test Suites: Run the largest test suite once and
the smaller test suites 2-4x for each larger run
• Optimal Scheduling:
– All max BIT’s should be equal for lowest coverage tranche
– If a test suite is longer than the max BIT of a shorter test suite
then you can schedule the test suites independently
– Interleave test suites
– With 3 test suites, run the shortest at least every second run
3/2/2016 Daniel Hansson, Verifyter 20
To find your optimal
setup you need to use
the equations

21. Optimal
Setup
3/2/2016 Daniel Hansson, Verifyter 21
A B C D
8 AM
sanity sanity sanity 9
nightly 10
11
Report Launch 12 PM
Each 8h Each 8h 1
2
3
sanity 4
sanity 5
6
sanity 7
nightly 8
nightly 9
10
11
nightly nightly 0 AM
1
2
3
4
5
6
7
Which regression
test setup is the
most efficient?
Answer: A