Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Automation roi microsoft e4 d event


Published on

Automation ROI Microsoft-E4D Event

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Automation roi microsoft e4 d event

  1. 1. Automation Testing ROI Presented by: Yaniv Cherkasky 054-7722568
  2. 2. Agenda• First Session – Automation – Why? When? When Not? – Test Automation Frameworks Models – Guidelines for first approaching Automation• Second Session – Automation ROI – Improving ROI – Case Study
  3. 3. Automation Testing DefinitionTest automation is the use of software tocontrol the execution of tests, Commonly, testautomation involves automating a manualprocess already in place that uses a formalizedtesting process. (Wikipedia)
  4. 4. When?• Repeatable Test Scenarios• Application Logic Level• Unit Test the code• Checking the Performance• Data/Event Driven Tests• Multiple configurations
  5. 5. When Not?• Test Cases that are complicated for coding (e.g. Image compression/Video quality etc.)• Test Cases that we run only a limited number of times• Premature parts in the Application (Constant Changes)
  6. 6. Why not Automate Everything?• Low ROI (Return On Investment)• Many times the Test Case is too complicated for a machine• Most of the time the Look and Feel needs to be done by Human Tester
  7. 7. Why do it?• Length of the testing phase is reduced• Higher efficiency (Save Time)• Improve Productivity (Increase bugs coverage)• Can reduce testing manpower (save money) TTM Resources Quality
  8. 8. Test Automation Types Load and Performance testing Installation and configuration testing Boundary testing Unit Testing Compliance testing Smoke and unit testing Localization testing Regression testing
  9. 9. Test Automation Framework common Models• Data Driven Automation Framework Model – Mass storage of data – Limited to the automation tool in use – Easy to implement, hard to maintain• Keyword Driven Automation Framework Model – Data tables using a keyword vocabulary – Hard to implement, easy to maintain
  10. 10. Test Automation Framework common Models• Test-driven development (TDD) —Focus on the task at hand —TDD leads to more modularized, flexible, and extensible code —Create unit tests before writing the code
  11. 11. Automation Entry criteria Frequentbuilds/patches/iterations Test cases that need verification with a high precision Test case functionality with high complexity
  12. 12. Automation Exit criteriaSuccessfulExecution Script has been reviewed for coding standards Review completed for variables, functions, test steps etc. Report Version Repository controlled
  13. 13. Guidelines for first approaching Automation1. Perform full analysis breakdown of system features & modules relationship2. Perform full analysis for manual test cases3. Mark the test cases which are repeatable which has long execute time4. Do a research of the environment (technology used, SW/HW limitations, platforms etc.)5. Make a check list for the required testing types (Functionality, GUI, Performance etc.)6. Pickup the appropriate test automation tool (MTM )
  14. 14. Guidelines for first approaching Automation7. Pick the suitable framework structure8. Conduct a breakdown of all framework methods9. Mark the common use of framework methods in all Test Cases – these methods will become your first building blocks!10. Calculate ROI11. Prepare a presentation with Automation Framework structure & ROI calculation12. Pick an example13. Run a POC
  15. 15. Tips & Tricks• Make sure your are considering future maintenance in your planning – maintenance cost money!• Don’t start with demanding resources, start with 1 resource for POC• Use also authentic factors for the ROI (items related to the unique product/environment etc.)• Automation is a development group, manage it as one!• When you approach your boss, in any given time, make sure you are translating all to budget terms :-)• Monitor Automation ROI on a yearly basis comparison
  16. 16. Questions?
  17. 17. Automation ROIIt is important to consider the following factors when evaluatingthe ROI of automation:• The cost of executing the automated tests. Consider the infrastructure required• The cost of maintaining the automation as the product evolves release by release.• The cost of any additional tasks necessitated by the automation• ROI will vary depending on your business environment and the application under test.
  18. 18. Importance of ROI• ROI is as important if not more important than technical automation topics• Test Automation is an investment
  19. 19. Return On Investment TTM Resources Quality
  20. 20. ROI Challenge UNREALISTIC EXPECTATIONS• 100% Automation• Immediate Payback• No Automation Analysis
  21. 21. ROI Dilemma• Continued automation justification• Difficult to meet unrealistic expectations True test automation benefits: Increased Cost Savings Increased Efficiency Increased Quality
  22. 22. Calculating ROI - Factors Automation Testing Factors – Tool CostsFixed • Software • License – Execution Machines – # of Tests to Automate/Effort EstimationVariable – Automated Development/Debugging Time (average per test) – Automated Test Execution Time (average per test) – Automated Test Analysis Time – Automated Test Maintenance Time
  23. 23. Calculating ROI – Factors Return on Investment Automation ROINumber of hourly employees? 500Average hourly rate of employee? 16.00Number of minutes to review/total each time card (APA 7)? 3Hourly rate of person reviewing time card? 16.00Percentage of inaccurate time cards (APA 1-8)? 2%Number of standard hours an employee worked in a pay period? 40.00Daily lost time in minutes for an employee (Calculation, Rouding) (APA 49)? 20 Card Audit SavingsTotal savings in minutes 1000Total minutes saved weekly 5000Total hours saved weekly 83.3Savings in dollars weekly $1,333.33Monthly savings $5,773.33Yearly Savings $69,333.33 Savings in Lost TimeDaily lost productivity in hours 0.33Average wages overpaid daily $5.28Average wages overpaid weekly $26.40Total wages recaptured weekly $13,200.00Total wages recaptured monthly $57,156.00Total wages recaptured yearly $686,400.00 Savings from Eliminating ErrorsTotal weekly payroll expense $320,000.00Weekly savings $6,400.00Monthly savings $27,712.00Yearly Savings $332,800.00 Total Savings SummaryWeekly audit savings $1,333.33Weekly error savings $6,400.00Weekly wages recaptured $13,200.00Weekly payroll savings $20,933.33Monthly payroll savings $90,641.33Yearly Payroll Savings $1,088,533.33
  24. 24. Calculating ROI for Automation Projects is an on-going task…Let:NPV(ATCF) = Net Present Value of After Tax Cash Flow generated by theinvestmentNPV(IC) = Net Present Value of Required InvestmentProfitability Index = 100*NPV(ATCF) NPV(IC) − NPV(IC)An outline of the equations underlying this expression is given in the Appendix.The elapsed time to positive cash flow, i.e. the time at which thediscounted cumulative cash flow crosses into the positive region is also important.This is sometimes called the discounted payback period.
  25. 25. Improving ROIImprovement Result
  26. 26. Improving ROIImprovement Result
  27. 27. Automation ROICase StudyOn-Line Shop Order Pages
  28. 28. Automation ROI – Case Study• Order pages test includes verifying the on-line buying of products• Effort Estimation for manual testing takes about 2 weeks for 1 person• The frequency of running the test is quite common since full regression is needed each time a change is made in results from a new marketing campaign or even a new discount• Manual test does not cover all buying aspects (boundary cases) – but only the positive testing required by the requirements• Currently, the test duration harms the project dynamic proceeding
  29. 29. Automation ROI – Case Study cost$ per month Manual Testing (today) Automatic testing Effort in days $ cost Effort in days $ costSickness 1 days 113.63 0 days 0Holiday 1.5 days 170.44 0 days 0Test Development time 7 days 795.41 19.5 days 2215.785Test Execution time 14 days 1590.82 1.5 days 170.44Bugs analysis time 5 days 568.15 1 days 113.63Report creation 3 days 340.89 1 days 113.63Test maintence time 1 days 113.63 2 days 227.26 Total: 3692.97 Total: 2840.745Legend: Month Total $Tester Hourly Rate 113.63 22 days 2500Automation Fixed costsTesting tool 3000
  30. 30. Automation ROI – Case Study Conclusions• Currently on “Order Pages” tests we save $ 852.225 per month (for 1 run) it’s $ 10226.7 per year by using automatic tests (numbers include lost business, refunds and testing the complex scenario with all it’s factors manually)• Currently in 1 year there are about 5 campaigns of to be tested…doing it with manual tests will take 120 days. If using automation for it, it will take 7.5 days. It means: 112.5 working days that are saved!!!
  31. 31. Automation ROI – Case Study Conclusions• Automation tests can run 10 times more in 1 month then using manual tests which means: more regression = more stable product• Automation 1 test run duration = 7 peoples work• The cost for the long run is actually decreases!• Most of the automation costs are fixed costs (one-time expenses) and are for the short run
  32. 32. Questions?
  33. 33. Thank You!!!