Test automation: the ROI myth

4,412 views

Published on

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,412
On SlideShare
0
From Embeds
0
Number of Embeds
378
Actions
Shares
0
Downloads
158
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Test automation: the ROI myth

    1. 1. Test automationThe myth of ROI Gilles Mantel ALE2011
    2. 2. 2
    3. 3. What are we talking about ? Unit tests Only
automated Integration tests Only
automated Functional tests ▶ End to end ▶ GUI Cannot
be
 ▶ Exploratory automated Only
automated Stress, performance, reliability etc Cannot
be
 automated Usability, AB tests etc
    4. 4. What are we talking about ? Unit tests Integration tests Functional tests Automatable
 with
s/w
robots ▶ End to end ▶ GUI ▶ Exploratory Stress, performance, reliability etc Usability, AB tests etc
    5. 5. ROI formula ROI = manual cost – automation cost Manual cost: execution Automation cost: software licenses, automation scripts, results analysis, script maintenance
    6. 6. ROI calculators
    7. 7. Cumulated cost over time 3 2 1
    8. 8. Main points Complex / Cumbersome Many unknowns ▶ Make assumptions Results not encouraging What’s missing ?
    9. 9. Cost of a defect
    10. 10. Cost of a defect
    11. 11. Financial option Right to buy a financial product (underlying) at a given price (strike) at a given date in the future (maturity date) Pay a prime to buy this right Ex: I pay 10$ the right to buy a Google share in December 2011 at the price of 530$
    12. 12. Financial model Benefits at maturity In the Money Underlying price at maturity Out of Money
    13. 13. Financial model of test automation ROI
    14. 14. Financial model of test automation Saved cost of identified defects
    15. 15. Modèle de gain ROI Cost of existing defects Saved cost of Investment in identified defects automation
    16. 16. Investment in automation ? In a Financial Bank: the mean cost of a defect is around 300 000 € At a web travel agency: 1mn website downtime costs 20 000€in terms of lack of revenue Measure !
    17. 17. Investment in automation ? In your organisation, how much costs: ▶ A late delivery? Check with marketing dpt ▶ Maintenance teams fixing defects ? Check with the CIO ▶ Operational loss caused by defects ? Check with the financial dpt ▶ Negative perceptions of brand ? Check with the sales dpt www.xebia.fr / blog.xebia.fr 17
    18. 18. Investment in automation ?
    19. 19. Investment in automation ?
    20. 20. Testing pyramid (Mike Cohn)
    21. 21. What ROI for an agile team ?
    22. 22. What ROI for an agile team ?
    23. 23. Where to invest ? and last, a bit here Then here First here
    24. 24. Investment in automation ?
    25. 25. Waterfall Pyramid
    26. 26. What ROI for a legacy ?
    27. 27. What ROI for a legacy ?
    28. 28. Where to invest ? First here Then here And last here
    29. 29. Conclusions  The ROI is not directly linked with time  The ROI is linked with non-quality  Agile practices lead to reducing the needed investment by iteratively automating  Measure how much you can invest based on your non-quality predictions
    30. 30. Twitter: @gmantelE-mail: gmantel@xebia.frMERCI

    ×