1. Test automation
The myth of ROI
Gilles Mantel
ALE2011

2. 2

3. What are we talking about ?
 Unit tests Only automated
 Integration tests Only automated
 Functional tests
▶ End to end
▶ GUI Cannot be
▶ Exploratory automated
Only automated
 Stress, performance, reliability etc
Cannot be
automated
 Usability, AB tests etc

4. What are we talking about ?
 Unit tests
 Integration tests
 Functional tests Automatable
with s/w robots
▶ End to end
▶ GUI
▶ Exploratory
 Stress, performance, reliability etc
 Usability, AB tests etc

5. ROI formula
 ROI = manual cost – automation cost
 Manual cost: execution
 Automation cost: software licenses, automation
scripts, results analysis, script maintenance

6. ROI calculators

7. Cumulated cost over time
3
2
1

8. Main points
 Complex / Cumbersome
 Many unknowns
▶ Make assumptions
 Results not encouraging
 What’s missing ?

9. Cost of a defect

10. Cost of a defect

11. Financial option
 Right to buy a financial product (underlying) at a given price
(strike) at a given date in the future (maturity date)
 Pay a prime to buy this right
 Ex: I pay 10$ the right to buy a Google share in December 2011 at
the price of 530$

12. Financial model
Benefits at
maturity
In the Money
Underlying
price at maturity
Out of Money

13. Financial model of test automation
ROI

14. Financial model of test automation
Saved cost of
identified defects

15. Modèle de gain
ROI
Cost of
existing
defects
Saved cost of
Investment in identified defects
automation

16. Investment in automation ?
 In a Financial Bank: the mean cost of a defect is
around 300 000 €
 At a web travel agency: 1mn website downtime
costs 20 000€in terms of lack of revenue
 Measure !

17. Investment in automation ?
 In your organisation, how much costs:
▶ A late delivery? Check with marketing dpt
▶ Maintenance teams fixing defects ? Check with the CIO
▶ Operational loss caused by defects ? Check with the financial dpt
▶ Negative perceptions of brand ? Check with the sales dpt
www.xebia.fr / blog.xebia.fr 17

18. Investment in automation ?

19. Investment in automation ?

20. Testing pyramid (Mike Cohn)

21. What ROI for an agile team ?

22. What ROI for an agile team ?

23. Where to invest ?
and last, a bit here
Then here
First here

24. Investment in automation ?

25. Waterfall Pyramid

26. What ROI for a legacy ?

27. What ROI for a legacy ?

28. Where to invest ?
First here
Then here
And last here

29. Conclusions
 The ROI is not directly linked with time
 The ROI is linked with non-quality
 Agile practices lead to reducing the
needed investment by iteratively
automating
 Measure how much you can invest based
on your non-quality predictions

30. Twitter: @gmantel
E-mail: gmantel@xebia.fr
MERCI