Test automation: the ROI myth

Test automationThe myth of ROI Gilles Mantel ALE2011

What are we talking about ? Unit tests Only automated Integration tests Only automated Functional tests ▶ End to end ▶ GUI Cannot be  ▶ Exploratory automated Only automated Stress, performance, reliability etc Cannot be  automated Usability, AB tests etc

What are we talking about ? Unit tests Integration tests Functional tests Automatable  with s/w robots ▶ End to end ▶ GUI ▶ Exploratory Stress, performance, reliability etc Usability, AB tests etc

ROI formula ROI = manual cost – automation cost Manual cost: execution Automation cost: software licenses, automation scripts, results analysis, script maintenance

ROI calculators

Cumulated cost over time 3 2 1

Main points Complex / Cumbersome Many unknowns ▶ Make assumptions Results not encouraging What’s missing ?

Cost of a defect

Financial option Right to buy a financial product (underlying) at a given price (strike) at a given date in the future (maturity date) Pay a prime to buy this right Ex: I pay 10$ the right to buy a Google share in December 2011 at the price of 530$

Financial model Benefits at maturity In the Money Underlying price at maturity Out of Money

Financial model of test automation ROI

Financial model of test automation Saved cost of identified defects

Modèle de gain ROI Cost of existing defects Saved cost of Investment in identified defects automation

Investment in automation ? In a Financial Bank: the mean cost of a defect is around 300 000 € At a web travel agency: 1mn website downtime costs 20 000€in terms of lack of revenue Measure !

Investment in automation ? In your organisation, how much costs: ▶ A late delivery? Check with marketing dpt ▶ Maintenance teams fixing defects ? Check with the CIO ▶ Operational loss caused by defects ? Check with the financial dpt ▶ Negative perceptions of brand ? Check with the sales dpt www.xebia.fr / blog.xebia.fr 17

Investment in automation ?

Testing pyramid (Mike Cohn)

What ROI for an agile team ?

Where to invest ? and last, a bit here Then here First here

Investment in automation ?

Waterfall Pyramid

What ROI for a legacy ?

Where to invest ? First here Then here And last here

Conclusions  The ROI is not directly linked with time  The ROI is linked with non-quality  Agile practices lead to reducing the needed investment by iteratively automating  Measure how much you can invest based on your non-quality predictions

Twitter: @gmantelE-mail: gmantel@xebia.frMERCI

http://agileinthemix.com	37
https://twitter.com	1
http://www.slideshare.net	1
http://paper.li	1
http://twitter.com	1

Test automation: the ROI myth

by Gilles Mantel on Sep 08, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

5 Embeds 41

Statistics

Test automation: the ROI myth — Presentation Transcript