Online Quality Assurance Day 2020 #2 РАМЕЛЛА БАСЕНКО «ROI of automation or how to sell your automation ideas to customers» telegram: wwww.t.me/goqameetup fb: www.fb.com/goqaevent fb: www.fb.com/qaday.org Сайт: www.qaday.org

1. ROI of Automation
or how to sell your automation ideas to customer
`

2. About Speaker
Speaker: Ramella Basenko
Role: Lead QA Engineer at AgileEngine, ISTQB certified
Test Manager
7 years of experience in QA area, 5 years of QA team
management experience.

3. Agenda
1. Common QA/AQA processes overview
2. What is ROI of automation and how to calculate it, how to get positive ROI rate for AQA
3. What are the key points which help to sell your automation ideas to customer
4. Example of a Success Story
5. Questions and answers

4. Common QA/AQA process overview
After just a few releases of comprehensive manual
regression tests, it becomes obvious that this is a big
bottleneck.
The business demands frequent releases, but there just
isn't the QA capacity to keep up. The team has two
choices:
- slim down its testing scope so that regression tests
can be completed more quickly (but of course this
would introduce a greater likelihood of regression
bugs)
- choose to release less frequently so not to repeatedly
incur the high cost of regression testing (but then
users would have to wait longer for new features, and
time-to-market opportunities would be missed

5. Common QA/AQA Process Overview
Regression testing often becomes a pain point
for an Agile teams, as it presents two major
challenges:
1. Increased time and efforts. Regression
testing may take up a lot of time, especially
in large projects. Sometimes, teams have
to spend an entire sprint on regression,
which is hardly acceptable. And increased
time and efforts mean increased costs.
2. Lack of test engineers’ concentration.
Running same test cases again and again
for a long time results in the lack of
concentration. This can lead to bugs
sneaking into production, which also
increases overall project costs.

6. Common AQA process overview
On the testing processes of medium and large-sized projects, there is a lot of repetitive work for regression tests
on each iteration of the development process. To reduce the expenses for QA processes, as a rule, companies
use testing automation.
Differences between manual and automated tests:
● Time when each kind of testing might be applied
● Speed of testing
● Error rate
What benefits we will get by using of test automation:
● Shorten Time to Market period
● Lower error rate
● Lower defect leakage to UAT team or customers

7. Common AQA process overview

8. ROI of automation and how to calculate it
Return on investment (ROI) is a financial ratio that illustrates business profit or loss margins.
Test Automation ROI = ((Savings-Investments)/ Investments)*100%
Investment = Automated Test Script Development Time
+ Automated Test Script Execution Time
+ Automated Test Script Analysis Time
+ Automated Test Maintenance Time
+ Manual Test Execution Time
Savings = Manual Test Execution or Analysis Time
* Total Number of Test Cases (Automated & Manual)
* Period of ROI
/ 8
**NOTE Period of ROI is the number of weeks for which ROI is to be calculated. Divided by 8 is
done wherever manual effort is needed. Divided by 18 is done wherever automation is done.

9. ROI of automation and how to calculate it
The cost function for manual testing is linear - it
costs a fixed X for each run of the test. The cost of
automated testing on the other hand is a step-wise
function: to run it once costs more than running it
manually (because you have to write the test), but
then every subsequent run costs nothing at all.
Overall return on investment is achieved when the two
lines intersect, and so the more often you run the
tests, the more you save.

10. Key Points to sell your automation ideas to customer
● You should always know what you are trying to accomplish with automation - define objectives
for your automation effort, define metrics for those objectives and define goals for each metric;
● Create comparison document with automation tools evaluation (include pros and cons of
mentioned tools and approaches, recurring and non-recurring costs, risks and opportunity
costs);
● Prepare Demo POC (if there are people with appropriate skills in a team) of selected tool and
show it to your customer;
● Perform a thorough ROI analysis and be able to show that the benefits of the tools exceed the
costs (according to release cycles on a project);
● Have a plan in place that manages the significant risks for the automation effort;
● Be ready to prove your point of view and answer any questions that customer may have
(technical or process questions);

11. Example of a Success Story
● Custom framework was built and demoed to customer with short list of tests in February 2019 (in three
weeks total);
● An additional AQA person was hired to the team in March 2019;
● In parallel the whole team with QA lead worked on Regression coverage document to have necessary base
for automation;
● In July 2019 we had first release where we used our automation suite for regression (around 700 test cases
which can be run against different browsers and databases) and regression time was reduced to 2.5-3
weeks;
● In February 2020 we had the whole regression suite automated + new features implemented between
release (more than 1200 test cases run in 40 minutes for one browser and one DB source) - regression
time is reduced to 1.5-2 weeks;
● Now - we can make regression in a single week and use our automation suite within Jenkins pipeline for
any build we need.
Precondition: Until we created automation suite and organized AQA process duration of our regression was 8-9
weeks of work (for team of 6 people) 👀
Goal of our customer was: To have more frequent releases and reduce regression time at least at least twice

12. Example of a Success Story
Manual effort before automation process started:
9 weeks => 9*5*8 = 360h a week for 1 QA person
6 persons => 2160h for one Regression cycle
Time cost for Automation process set up(non-recurring costs):
2 AQA persons * 9 months = 9m*4w*5d*8h*2p= 2880h
If we measure by automated regression runs per release and suppose that we run this suite twice
before release and we had 3 releases a 1 year, ROI in 1 year will be following :
ROI =( (3*2*2160 - 2880)/ 2880 )*100% = 350%

13. Q&A

14. Thanks for your
attention!