Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to Measure Success in Continuous Testing

204 views

Published on

In this webinar, Amir Rozenberg and Fernando Vidal will discuss the Sauce Labs Continuous Testing Benchmark Report, a new index based on insights gleaned from the more than 2 billion tests run on the Sauce Labs Continuous Testing platform. The index can be used by any organization to evaluate their progress towards continuous testing. Joining Amir and Fernando will be Phil Wells from the New York Times, a Sauce Labs customer that has achieved a nearly perfect 100 against the Continuous Testing Benchmark index.

Key takeaways:

- The components of the new continuous testing index
- What makes those components such strong maturity indicators
- How can organizations improve on each of the metrics
- What happens organizationally when those metrics are improved
- How the NY Times manages their continuous testing pipeline to deliver quality crosswords while driving constant improvements

Published in: Technology
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ http://1url.pw/lxAq0 ◀ ◀ ◀ ◀
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

How to Measure Success in Continuous Testing

  1. 1. HOW TO MEASURE SUCCESS IN CONTINUOUS TESTING MEASURE & OPTIMIZE YOUR WAY TO DELIVER FASTER, BETTER JULY 10, 2019
  2. 2. HELLO Amir Rozenberg Director, Product Fernando Vidal Sr. Data Scientist Phil Wells Software Engineer July 10, 2019 © Sauce Labs, Inc. 2
  3. 3. 3 AGENDA Introduction to CT Maturity Benchmark The Composition of the CT Maturity Benchmark From the Practitioner Perspective Q&A July 10, 2019 © Sauce Labs, Inc.
  4. 4. 4 DIGITAL TRANSFORMATION: LEADERSHIP OR EXTINCTION Where in the development cycles you encounter most delays? Source: GitLab 2018 Developer Survey July 10, 2019 © Sauce Labs, Inc.
  5. 5. 5 CONTINUOUS TESTING MATURITY MEASURING AND OPTIMIZING The Continuous Testing Journey LEGACY TESTING CONTINUOUS TESTING PROOF OF AUTOMATION VALUE July 10, 2019 © Sauce Labs, Inc.
  6. 6. 6 CONTINUOUS TESTING MATURITY MEASURING AND OPTIMIZING The Continuous Testing Journey LEGACY TESTING CONTINUOUS TESTING PROOF OF AUTOMATION VALUE July 10, 2019 © Sauce Labs, Inc.
  7. 7. 7 2 BILLION+ TESTS RUN TO DATE SAUCE LABS: THE ONLY VENDOR WITH THE DATA AND EXPERTISE TO CREATE THIS REPORT ● Millions of tests run every day on Sauce Labs ● Thousands of users run tests every day ● Unique position to use this data to understand what drives success July 10, 2019 © Sauce Labs, Inc.
  8. 8. 8 A SIMPLE 0-100 SCORE FOR AUTOMATED TESTING SUCCESS COMPONENTS OF THE SAUCE LABS CONTINUOUS TESTING BENCHMARK REPORT July 10, 2019 © Sauce Labs, Inc.
  9. 9. TEST QUALITY TEST SUITE MAINTENANCE AND BATTLING FLAKY TESTS July 10, 2019 © Sauce Labs, Inc. 9
  10. 10. TEST QUALITY: WHY IT MATTERS GOAL: Consider a build green if every test passes ● “Quarantine” tests that are flaky ● Test Fix Rate > Test Break Rate ● Quickly repair tests that break due to application changes High “ambient” levels of test failure due to flaky tests or an ever increasing number of quarantine tests decrease functional coverage and reduce confidence in the automated testing process TEST FAILURE SHOULD LEAD TO HIGH CONFIDENCE THAT APP IS BROKEN, NOT YOUR TEST July 10, 2019 © Sauce Labs, Inc. 10
  11. 11. TEST RUN TIME THE KEY TO TEST QUALITY AND MAINTAINABILITY July 10, 2019 © Sauce Labs, Inc. 11
  12. 12. TEST RUN TIME: WHY IT MATTERS LONG TESTS BOTTLENECK A TEST SUITE Long jobs slow everything down July 10, 2019 © Sauce Labs, Inc. 12
  13. 13. TEST RUN TIME: WHY IT MATTERS SHORT TESTS FAIL LESS OFTEN Long jobs slow everything down July 10, 2019 © Sauce Labs, Inc. 13
  14. 14. TEST PLATFORM COVERAGE QUALITY MEANS CONFIDENCE ACROSS MULTIPLE PLATFORMS July 10, 2019 © Sauce Labs, Inc. 14
  15. 15. TEST PLATFORM COVERAGE: WHY IT MATTERS THERE IS NO “STANDARD” FOR WEB OR MOBILE DESKTOP MOBILE BROWSERS July 10, 2019 © Sauce Labs, Inc. 15
  16. 16. TEST CONCURRENCY RUN AS MANY TESTS AS QUICKLY AS POSSIBLE FOR RAPID FEEDBACK July 10, 2019 © Sauce Labs, Inc. 16
  17. 17. TEST CONCURRENCY: WHY IT MATTERS ACCELERATE THE FEEDBACK LOOP & KNOW QUICKLY IF YOU CAN RELEASE The Importance of Leveraging Concurrency a Sample Test Suite with 50 Tests 30 Minute Build 6 Minute Build July 10, 2019 © Sauce Labs, Inc. 17
  18. 18. CT BENCHMARK: AVERAGE OF 4 SCORES July 10, 2019 © Sauce Labs, Inc. 18
  19. 19. Why this is Important to Get Right Develop Apps Faster Improve Developer Productivity Delight Users 19 Continuous Testing Unlocks Your Ability to Achieve Digital Excellence
  20. 20. 20 THE NY TIMES SCORES A PERFECT 100 THOUGHTS AND OBSERVATIONS FROM PHIL WELLS July 10, 2019 © Sauce Labs, Inc.
  21. 21. Q&A

×