When executing test automation at scale and continuously the value tends to decline over time, the team should follow recommended practices to keep their tests with high value. Getting the value from your test automation is fundamental for fast feedback, risk reduction and return on investment from your testing activities. Once developing the test scenarios, teams cannot stop monitoring and ensuring that their tests continuously bring value, are not flaky, and can support the latest functionalities in your web and mobile apps. Teams often “forget” about their tests once they have been developed and integrated into the CI pipeline regardless of the value they bring. This webinar will cover: -How to make smart decisions regarding which test scenarios to automate? -What are the criteria for a test to get included in the CI and continuous testing pipeline? -How to continuously maintain the tests and optimize your suite so it continues to bring value? -See a live demo of smart reporting and analytics that can serve as a monitoring and test maintenance tool.

1. 1 | Keeping Your Continuous Testing Suites Continuously Valuable in DevOps perfecto.io
Keeping Your Continuous Test Automation Suites
Continuously Valuable in DevOps

2. 2 | Keeping Your Continuous Testing Suites Continuously Valuable in DevOps perfecto.io
• Chief Evangelist and author at Perfecto
• Blogger and speaker
• 19+ years in development & testing
• Author of
“The Digital Quality Handbook”
and “Continuous Testing for
DevOps Professionals”
• Twitter: @ek121268
• Blog: continuoustesting.blog
• Website: perfecto.io
Today’s Speaker:
Eran Kinsbruner

3. perfecto.io3 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Today’s Agenda
1
2
3
Current Market State & the Agile Testing Manifesto
How to Decide Which Tests to Automate
Test Case Certifications in Continuous Testing
4 Test Maintenance & Test Aging
5 Live Demo

4. perfecto.io4 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Digital Engagement Reality – More Relevant Than Ever

5. perfecto.io5 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Common Pain Points in Test Automation
Automation Skillset
Merging Tests Into Pipeline
Escaped Defects Due to Noise
Advanced Automation Scenarios
Challenge to Setup Test Environment Designed for
Testability
Time Spent to Analyze Reports
Testing Is Done Separately
Test Maintenance & Digital Platforms Coverage Lack of Time to Automate
It all boils down to people, processes, and/or technology.

6. perfecto.io6 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Global Mobile Market Share – Q1 2020
Android OS Market Share
Android 6.x (Marshmallow) Android 7.x (Nougat) Android 8.x (Oreo) Android 9 (Pie) Android 10 (Q)
20%
42%
11%
8%
16% In Dev Preview
Coming Soon

7. perfecto.io7 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Android Stock OS vs. Custom OS
Device Model Custom OS Implementation Versions Base Stock OS
Samsung Galaxy S20 Ultra One UI 2 Android 10
Samsung Galaxy S7 TouchWiz UI Android 8.0
Samsung S8 One UI Android 9.0
Samsung Note 9 One UI 2 Android 10.0
Google Pixel 4 XL STOCK Android 10
Huawei Mate 20 Pro EMUI 9.1 Android 9.0
LG G8 ThinQ LG UX 9.0 Android 10.0
Oppo Reno 5G Color OS 6 Android 10
Huawei Y9 Prime Magic UI 2.1 Android 10

8. perfecto.io8 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Mobile Platforms Calendar H1 2020
Jan Feb March April May June July Aug Sept
Devices Samsung Galaxy A71
Samsung Galaxy Note 10
Lite
LG Q51
Samsung Galaxy Z Flip
Xiaomi Mi 10 Pro
Xiaomi Poco X2
Sony Xperia 1 II
Sony Xperia L4
LG V60 ThinQ
Samsung Galaxy A11
Samsung Galaxy M31
Samsung Galaxy S20
Samsung Galaxy S20+
Samsung Galaxy S20 Ultra
Samsung Galaxy Tab S6
Xiaomi Redmi Note 9 Pro
Max
Xiaomi Black Shark 3 Pro
Huawei P40 Lite E
Huawei Mate XS
Motorola Moto G8
Motorola Moto E6s
ZTE Axon 10S Pro
Oppo Find X2 Pro
Oppo Reno 3
Vivo Nex 3S 5G
Nokia 8.2
Apple iPhone SE2
Huawei P40 Pro
Huawei P40
OnePlus 8
OnePlus 8 Pro
Google Pixel 4A
Motorola Edge Samsung
Galaxy
Note 20
Apple
iPhone 12
Series
Pixel 5
Pixel 5 XL
OS iOS 13.1 Release
Android 10.0
Android 11 Developers
Preview
iOS 13.4 GA Android 11 Beta iOS 14 Beta Android 11
GA
iOS 14 GA

9. perfecto.io9 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Global Web Market Share – Q1 2020

10. perfecto.io10 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Web Platforms Calendar H1 2020
Jan Feb Mar Apr May Jun July Aug Sept
Chrome 79 80 81 82 83 Beta 83 84 Beta 84 85
Firefox 72 73 74 75 76 77
78
79 80 81
Safari 13.0.2
Edge 80 81
Beta
81 82 Beta 82 83 Beta 83 84 Beta
Desktop OS
Releases
Mac OS Catalina
10.15.3
MacOS
Next (TBD)

11. perfecto.io11 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Network Reliability – August 2019
Source: Open Signal

12. perfecto.io12 | Perfecto by Perforce © 2020 Perforce Software, Inc.
T H E D E V O P S M A N I F E S T O
 Continuous testing over testing at the end.
 Embracing all testing activities over only automated functional testing.
 Testing what gives value over testing everything.
 Testing across the team over testing in siloed testing departments.
 Product coverage over code coverage.
The Agile Testing Manifesto

13. perfecto.io13 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Are You Measuring Your DRE (Shift Left)?
DRE (Defect Removal Efficiency) =
Defects removed during the development phase
Defects detected later in the cycle (UAT, Production)
Coverage, lack of and late automation testing, designed
for testability, unit testing, outdated environments/platforms.

14. perfecto.io
Optimize Based on Data —
Recurring Patterns for Inefficiencies
Licenses In-use Disconnected
Orchestration issues
Network Availability Data
Backend issues
Objects Coding Time Other
Scripts & FW issues
• 10% of devices causing 80% of lab issues
Networking Stability Config Lock
Lab issues
Backend
30%
Lab
15%Orchestration
15%
Scripts & FW
40%
What’s wrong
with my scripts?
What’s wrong
with my
executions?
What’s wrong
with my lab?
What’s wrong with
my backend?

15. perfecto.io
Noise Reduction – Continuous Testing Enabler
“About 10–20 defects per 1000 lines of code during in-
house testing, and 0.5 defect per KLOC in production”
(Sogeti Labs)
(~1 real Defect per developer per day)
Regression suite of 10,000 tests == ~20% failures
Out of 2000 failures, only 20 are actual bugs,
1980 are false negative, noise
Reporting noise is a risky bad practice
•Reporting crashes, errors in functionality, performance issues – dev friendly
•Lab Issue, object not found, popups, alerts, orchestration etc. – Should never be
taken to dev

16. perfecto.io
Certifying Test Automation

17. perfecto.io17 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Test Automation Certification
• Think continuous value.
• Think maintenance and complexity to create, execute, monitor, and maintain (ROI).
• Test reliability over time.
If { (Test Scenario) == Candidate to add Value add
{ and x– It detect defects}
{ and – It passes across multiple platforms with consistent result}
{ and – Its execution time is < 5 minutes }
Then
{ Include in Test Suite & & CI }
Else {
}
Run such certifications once in every
2-3 software iterations.

18. perfecto.io18 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Measuring Continuous Testing Success
1. Average test case creation time
Total test cases
Total creation time
2. Test update/create ratio per sprint
Test case updated
New test created
3. Average test execution time
Test case executed
Total execution time
4. Test executed %
Test executed
Total test cases
5. False positive %
False failure
Total test cases
6. Tests blocked %
Test blocked
Total test cases
7. Average test case creation time
• Environment & test data preparation
• Test case maintenance, creation &
execution
8. Code coverage %
Lines of code executed
Total lines of code
9. Requirements coverage %
# of requrements
Total requirements
10. Risk coverage %
(frequency, damage)
11. Defect %
Test case failed
Total test cases executed
12. Defect leakage %
Defects slipped
Total defects
13. Defect rejection %
Defects rejected
Total defects found
14. Mean time to detect (MTTD)
Defects found
Total execution time
15. Product downtime %
Product downtime
Total run time
16. Exploratory testing hrs vs planned
17. Average exploratory session time
# of sessions
Total exploratory time
18. Average defects per session
Defects found
# of sessions
19. Defect distribution by
• Priority
• Severity
• Functional area
• Tester type
• Cause (env, architecture, code, design,
requirements, user entry)
• Test type (review, walkthrough, test
execution, exploratory, etc)
x100
x100
x100
x100
x100
x100
x100
x100
20. Cost of hardware
21. Cost of software
22. Cost of maintenance
(hardware, software)
23. Cost of test preparation
24. Cost of test execution
25. Average cost per tester
Total tester cost
# of testers
26. Cost of test maintenance over time
27. Cost of defect by
• Severity
• Stage found (dev, QA,
UAT, production)
S P E E D Q U A L I T Y C O S T

19. 19 | Keeping Your Continuous Testing Suites Continuously Valuable in DevOps perfecto.io
Personas and Tools Balance

20. perfecto.io20 | Perfecto by Perforce © 2020 Perforce Software, Inc.
I N T E R A C T I V E
T E S T S
UI/UX manual tests
Balancing Test Creation for the Three Different
Personas With the Right Tools
O P T I M I Z E D M O D E L
Developers & SDETs
(Code-Based)
Business Testers
Ownership (Codeless)
Business Testers
Ownership

21. perfecto.io21 | Perfecto by Perforce © 2020 Perforce Software, Inc.
A Single Pane of Glass — Collaborative Approach

22. Tool/Framework
Selection Criteria:
Technical Fit & Skills
SDLC
process fit
(integration,
plug-ins, skills,
etc.)
Community
size, support,
and docs
Feedback loop
and reporting
Automation
coverage
Cloud and
automation at
scale
Automation
robustness and
maintainability

23. perfecto.io
Mixing and Matching Technologies
Comparison Case Study & Recommended Practice

24. perfecto.io24 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Cross-Browser Testing Tools (Open Source) — Download Trends Vary
Protractor
Puppeteer
WebDriver IO
Cypress IO
Selenium WebDriver
Source: npmtrends

25. perfecto.io25 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Additional Categorization of JavaScript Browser Testing Tools
• Tools that...
• Provide a testing structure (BDD/ATDD) (Mocha, Jasmine, Jest, Cucumber).
• Provide assertion functions (Chai, Jasmine, Jest, Unexpected).
• Generate, display, and watch test results (Mocha, Jasmine, Jest, Karma).
• Generate code coverage reports (Istanbul, Jest, Blanket).
• Provide a browser or browser-like environment with control of scenario execution, UI testing, and more
(Protractor, Nightwatch, Phantom, Casper, Selenium, WebDriver.IO, TestCafe).
• Provide mocks, spies, and stubs (Sinon, Jasmine, enzyme, Jest, testdouble).
Source: Medium.com

26. High-Level Comparison Between Tools, Personas and Practices
Test Framework Supported
Dev
Languages
Supported
Platforms
Supported
Test
Frameworks
Setup and
Execution
Integrations Breadth of
Testing
Options
Maturity,
Documentation,
Support
Required Skillset Cloud and Execution at
Scale
Selenium/Appium
WebDriver
Java, C#,
Java Script,
Python,
Ruby,
Objective-
C
Chrome,
Safari,
Firefox, Edge,
IE/iOS/Andro
id
Mocha JS, Jest,
other super
set on top of
Selenium
(Protractor,
WebDriverIO,
etc.)
Download
relevant
driver, set up a
grid, network
and location
impacts
execution
speed
Plenty of
integrations
(CI, CD,
reporting,
visual testing,
cloud vendors)
End-to-
end,
security,
unit,
Robust community,
multiple bindings, best
practices
Coding skills required
(SDET Oriented)
Perfecto fully supports
Selenium and its
WebDriver
configurations. Local
execution requires
setting up a Selenium
grid
XCUITest/Espresso
/Headless
Objective
C/Java/Java
Script
Chrome,
Electron
NA Embedded
into IDEs,
headless
bundles a
browser in the
FW
CI/CD UI/Unit Good documentation
and code samples
Dev Oriented Built-in Chrome/Firefox
browsers in headless,
Perfecto Cloud supports
scaling
Espresso/XCUITest
Codeless Irrelevant,
based on
record and
playback
All Proprietary UI
with
underlying
Selenium
WebDriver
APIs
Mostly
SaaS/browser
plugin
installation
Limited Functional
/UI
Growing, limited No coding skills
required (Business
Tester Oriented)
Perfecto supports
codeless in the cloud
BDD Java, Ruby,
JS, Kotlin
All Junit,
Selenium,
Appium
Open source,
Maven/Gradle
/TestNG
Plenty + APIs
(e.g. Rest
Assured)
Functional Robust community,
docs, adoption
Step-definition
development in code is
required/scenarios are
no-code (Mix of
Business Tester and
SDETs)
Perfecto Quantum is a
web/mobile BDD
framework

27. perfecto.io
How to Measure Test Value
Through Reporting and Analytics
(What to Look For)

28. perfecto.io28 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Error Classification and Things to Consider > Test Productivity

29. perfecto.io29 | Perfecto by Perforce © 2020 Perforce Software, Inc.
The Power of Test Data Analysis – Look Out for Red Flags!
1. Long history of failures
A. Test vs. device issue
B. Specific area failures
2. Script development practices
A. Popup handling
B. Element not found
3. Are the tests bring any
value?
4. Platform health

30. perfecto.io30 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Create Your Own RCA

31. perfecto.io31 | Perfecto by Perforce © 2020 Perforce Software, Inc.
How Do Tests Perform Within CI?

32. perfecto.io32 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Top 8 Cloud Testing Benefits
Keep Up With Platform
Upgrades (Mobile/Web)
Always Available, Always
On, Reliable, 24x7
Productivity Anywhere, Global
Coverage for Business Continuity
Enterprise-Grade, Secure,
Access to Backend Services
Disaster Recovery Built-In,
Audit Ready
Unlimited Test
Automation Scalability
Cost Effective
Zero Cost Maintenance
1 2 3 4
6 75
Cross-Team
Collaboration
8

33. perfecto.io33 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Perfecto’s 4 Key Pillars of Continuous Testing in DevOps
This is where money is spent (Time, Tools, Resources)
This is where value is being
realized, quality is improved
Unified solution serves DevOps with greater effectiveness.

34. perfecto.io34 | Perfecto by Perforce © 2020 Perforce Software, Inc.

35. 35 | How to Select the Right Selenium Tools to Boost Your Test Automation perfecto.io
Thank You
Visit perfecto.io or follow us @perfectomobile