The document discusses various aspects of automating software testing. It suggests automating the detection of flaky tests, determining the severity of test failures, converting tests to more isolated unit tests, and using usage data to determine what to test next. It emphasizes that while automation can improve testing efficiency, human oversight is still needed, and code reviews serve as the link between automated and manual processes.

1. Let’s assume
we had to pay for testing
Kim Herzig (Software engineer & Researcher)
kimh@microsoft.com
www.research.microsoft.com/people/kimh
www.kim-herzig.de
Visual Studio
Team Services
Tools for
Software Engineers
1

2. Visual Studio
Team Services
Tools for
Software Engineers
3

3. When I joined Microsoft
4

4. Let’s assume
we had to pay for testing
Kim Herzig (Software engineer & Researcher)
kimh@microsoft.com
www.research.microsoft.com/people/kimh
www.kim-herzig.de
Visual Studio
Team Services
Tools for
Software Engineers
5

5. Money (machine)
Effort (human)
Let’s assume
we had to pay for testing
6

6. How much quality assurance (testing) did you do for
your latest (research) project?
Could be more?
Because you had to ship your paper on a firm deadline.
But you could have automated it, right?
7

7. AST WORKSHOP WEBSITE
“In software development practice, testing accounts
for as much as 50% of the total development effort.
It is therefore imperative to reduce the cost and
improve the effectiveness of software testing
by automating the testing process.”
8

8. INNER & OUTER DEVELOPMENT LOOP
Pre-check-in Integration process
Fully automated, including test selection & code coverage
10
Check-in

9. What is the goal of your automation?
And how do you know when you reached it?
11

10. RETURN OF INVESTMENT
When do product teams buy our tools?
Your tool
X Y
<
12

11. Example
Let’s make an
14

12. EXAMPLE
Test selection based on code coverage
Your tool
CODE
COVERAGE
RUNTIME
REDUCTION
15
Slows down test
execution by 30%
Saving 50% of tests!
?

13. Product X:
 10,000 unit test cases
 0.12 sec per test case
LET’S DO SOME BASIC MATH
𝐶𝑜𝑠𝑡 𝑇𝑒𝑠𝑡 = 𝑇𝑒𝑠𝑡𝐶𝑎𝑠𝑒𝑠 ∗ 𝑅𝑢𝑛𝑡𝑖𝑚𝑒 𝑇𝑒𝑠𝑡𝑐𝑎𝑠𝑒 ∗ 𝐶𝑜𝑠𝑡 𝑀𝑎𝑐ℎ𝑖𝑛𝑒
= 10,000 ∗ 0.12 sec ∗ 5.14𝑒−4 ൗ$
𝑠𝑒𝑐 = $𝟎. 𝟔𝟐
This excludes human effort!
 ~2500 builds per day
 Azure machine: $1.85/hr
Assumption: you achieve 50% test reduction
per dayYou saved: $0.62 ∗ 2500 ∗ 0.5 = $722
per build
16

14. per build
𝐶𝑜𝑠𝑡 𝐶𝑜𝑑𝑒𝐶𝑜𝑣𝑒𝑟𝑎𝑔𝑒 = 𝐶𝑜𝑠𝑡 𝑇𝑒𝑠𝑡𝑖𝑛𝑔 ∗ 𝑅𝑢𝑛𝑡𝑖𝑚𝑒 𝑂𝑣𝑒𝑟ℎ𝑒𝑎𝑑 + 𝐶𝐶𝑆𝑖𝑧𝑒 ∗ 𝐶𝑜𝑠𝑡 𝐷𝑖𝑠𝑘
+ 𝐶𝐶𝑆𝑖𝑧𝑒 ∗ 𝑆𝑝𝑒𝑒𝑑𝐼/𝑂 ∗ 𝐶𝑜𝑠𝑡 𝑀𝑎𝑐ℎ𝑖𝑛𝑒
BUT HANG-ON …
You used code coverage, right?
= $0.62 ∗ 0.3 + 3.3𝐺𝐵 ∗ 0.35 ൗ$
𝐺𝐵
+ 2 Τ𝑠𝑒𝑐
𝐺𝐵 ∗ 3.3𝐺𝐵 ∗ 5.14𝑒−4 Τ$
𝑠𝑒𝑐= $1.34
Facts:
 Block coverage: 3.3 GB per build
 0.35 $/GB SDD cost
 SSD speed 500MB/sec
 Azure machine: $1.853/hr
You spend: $1.34 ∗ 2500 = $3,350 per day
17

15. $3,350 $722
Slows down test
execution by 30%
Saving 50% of tests!
And this excludes many aspects: e.g. your analysis time!
It might still be worth the effort
But it’s as good as you might think, and we have more pressing issues.
18

16. Time is money
19
If we are talking about testing as part of build,
we can measure cost in terms of time. It’s all about speed.

17. IT’S EVEN MORE COMPLICATED
Nobody runs tests after build (anymore) in the inner loop
Legend:
Source module
Test module
Build
Test execution
buddy
build
Rolling
build
CloudBuild
A
1. Build Graph Analysis
X C
B
D
E Z
A
X C
B
D
E ZY Y
2. Build/Test Execution
B1
B2
B3
B C
D
Y
Z Z
E
3. Results
Success or
Failure (log files)
Y
CloudBuild uses module-level test selection
when any dependencies of a module change, all tests inside that module are executed.
Thus, you can only reduce / re-order tests within the same target!
Max possible saving of ~8% of build time.
20

18. 21
System
performing test
selection based
on code
coverage
Mergingcoverage
files…
Version
control

19. HIGH EFFORT/COST
MOST RESEARCH 22
Check-in

20. Solve Problems Collaboratively
23

21. HIGH EFFORT/COST
MOST RESEARCH 25
Check-in

22. How do we system & integration
test a system like Windows?
Where
When
Who
How frequent
26

23. SYSTEM AND INTEGRATION TESTING
Software testing is expensive
• 10k+ gates executed, 1M+ test cases
• Different branches, architectures, languages, devices, platforms, …
• Aims to find code issues as early as possible
• Slows down product development 27

24. IT DOES NOT NEED ROCKET-SCIENCE
Cost function decides when to skip tests.
𝐶𝑜𝑠𝑡 𝐸𝑥𝑒𝑐𝑢𝑡𝑖𝑜𝑛 > 𝐶𝑜𝑠𝑡 𝑆𝑘𝑖𝑝 ? suspend ∶ execute test
𝐶𝑜𝑠𝑡 𝐸𝑥𝑒𝑐𝑢𝑡𝑖𝑜𝑛 = 𝐶𝑜𝑠𝑡 𝑀𝑎𝑐ℎ𝑖𝑛𝑒/𝑇𝑖𝑚𝑒 ∗ 𝑇𝑖𝑚𝑒 𝐸𝑥𝑒𝑐𝑢𝑡𝑖𝑜𝑛 + "Cost of potential false alarm"
= 𝐶𝑜𝑠𝑡 𝑀𝑎𝑐ℎ𝑖𝑛𝑒/𝑇𝑖𝑚𝑒 ∗ 𝑇𝑖𝑚𝑒 𝐸𝑥𝑒𝑐𝑢𝑡𝑖𝑜𝑛 + (𝑃𝑟𝑜𝑏 𝐹𝑃 ∗ 𝐶𝑜𝑠𝑡 𝐷𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟/𝑇𝑖𝑚𝑒 ∗𝑇𝑖𝑚𝑒 𝑇𝑟𝑖𝑎𝑔𝑒 )
𝐶𝑜𝑠𝑡 𝑆𝑘𝑖𝑝 = "Potential cost of elapsing a bug to next higher branch level"
= 𝑃𝑟𝑜𝑏 𝑇𝑃 ∗ 𝐶𝑜𝑠𝑡 𝐷𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟/𝑇𝑖𝑚𝑒 ∗ 𝑇𝑖𝑚𝑒 𝐹𝑟𝑒𝑒𝑧𝑒 𝑏𝑟𝑎𝑛𝑐ℎ ∗ #𝐷𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟𝑠 𝐵𝑟𝑎𝑛𝑐ℎ
time
Quality
Gate
Build Build
?
Build
28

25. Simulated on
Windows 8.1
development
period
(BVT only)
29

26. THINGS THAT DO NEED AUTOMATION (@ INDUSTRY)
Short story: everything that improves code velocity (integrate faster)
30

27. AUTOMATION GOAL
EffectivenessReliability
Cannot trust
result
Why are we
running this
test?
Ok
Rarely finds a
defect
high
lowhigh
low
Cannot trust
result
Rarely finds a
defect
high
lowhigh
low
31

28. AUTOMATION CONSTRAINTS
Do not sacrifice code quality
Avoid any runtime overhead
Present results to engineers (they need to understand)
32

29. 1 Flaky tests
Automated detection of
33

30. FLAKY TESTS ARE EXPENSIVE!
Every test failure breaks the build
• We need to re-run the builds.
• Slows down code velocity.
Fixing a flaky test is not for free either
• How do you make yourself independent from network?
• Pure unit tests are rare.
• Mocking is one thing, maintaining the mock another.
• Maybe it’s not even worth to fix the test.
Wasting human effort / time
• In many cases failure triage requires human supervision
34

31. WHAT TO AUTOMATE?
Identify flaky tests
• We use heisenbug queues to run ~1000 executions on a fixed code-base.
Is it a flaky passing or a flaky failing test?
• Maybe the test should never have passed.
How many (re-)tries do we need to make a definite call?
• Is this feasible to keep the test for a while or drop it completely?
Why is it flaky? / How to fix?
• I’m NOT talking about auto-fix
• Hints and strategies on making the test better.
35

32. 2 Test failure severity
Automated detection of
36

33. TEST FAILURE SEVERITY
DON’T CARE CAN WAIT BLOCKER
• Test issue
• Not my code
• Will not fix
• Low priority bug
• Before release
• Refactoring
• Fix it now
• I need to know this
37

34. 3 Convert tests to unit(ish) tests
Automatically
38

35. Unit testing
Build
Verification
Code ReviewStatic Analysis System testing
Runtime / cost
Fast but
limited scope
Essential but no
quality
indication
Way too many
false positives
Good for
maintainability
but / finds few
code defects
Full scope but
slow and flaky
Relations are not accurate but only to illustrate the problem.
39

36. 4 What to test next
Automated suggestions of
40

37. CATEGORIZATION OF LIVE SITE INCIDENTS FOR 17
DIERENT COMPANY PRODUCTS.
DATA FROM : MARIA CHRISTAKIS, CHRISTIAN BIRD
41

38. DIFFS ON (USAGE) SCENARIOS!
Engineer
Test
42

39. In collaboration with
Christopher Theisen, Dr. Laurie Williams
& Brendan Murphy (Microsoft Research)
Legend
Verification
Customer
Overlap
Legacy code
43

40. • No test failures on new code (unlikely)
• Is our test code base growing relatively to the code base?
• What’s the average age of tests?
Are we testing new code?
• No: we need to change tests or prevent misuse.
• Why are people using the feature differently?
Are features used as intended/tested?
• Tests need to adapt to current usage scenarios.
• What is the intended life-time of a test case?
How does usage change over time?
TESTING AS REALITY CHECK
44

41. 5 Code Reviews as the Glue
Every automation requires human supervision
45

42. 47