When executing test automation at scale and continuously the value tends to decline over time, the team should follow recommended practices to keep their tests with high value. Getting the value from your test automation is fundamental for fast feedback, risk reduction and return on investment from your testing activities. Getting the value from your test automation is fundamental for fast feedback, risk reduction and return on investment from your testing activities. Once developing the test scenarios, teams cannot stop monitoring and ensuring that their tests continuously bring value, are not flaky, and can support the latest functionalities in your web and mobile apps. Teams often “forget” about their tests once they have been developed and integrated into the CI pipeline regardless of the value they bring. Key Takeaways: -How to make smart decisions regarding which test scenarios to automate? -What are the criteria for a test to get included in the CI and continuous testing pipeline? -How to continuously maintain the tests and optimize your suite so it continues to bring value? -See a live demo of smart reporting and analytics that can serve as a monitoring and test maintenance tool.

1. Perfecto by Perforce © 2020 Perforce Software, Inc.
Keeping Your Continuous Test Automation Continuously Valuable

2. 2 | Advanced Codeless Testing for Web Apps perfecto.io
Today’s Speaker:
Eran Kinsbruner
• Chief Evangelist and author at Perfecto
• Blogger and speaker
• 19+ years in development & testing
• Author of
“The Digital Quality Handbook”
and “Continuous Testing for
DevOps Professionals”
• Twitter: @ek121268
• Blog: continuoustesting.blog
• Website: perfecto.io

3. perfecto.io3 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Digital Innovation Highlights

4. perfecto.io
World Quality
Report 2019-2020

5. perfecto.io
World Quality
Report 2019-2020
Maturing DevOps and Agile
actually requires more testing!

6. perfectomobile.com
What Is Continuous Testing?
Continuous testing is the process of executing automated
high value tests as a part of the software delivery pipeline
in order to obtain feedback on business risks associated
with a software release upon every code change.

7. perfecto.io7 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Optimize Based on Data
Recurring Patterns for Inefficiencies
Licenses In-use Disconnected
Orchestration issues
Network Availability Data
Backend issues
Objects Coding Time Other
Scripts & FW issues
• 10% of devices causing 80% of lab issues
Networking Stability Config Lock
Lab issues
Backend
30%
Lab
15%Orchestration
15%
Scripts & FW
40%
What’s wrong
with my scripts?
What’s wrong
with my
executions?
What’s wrong
with my lab?
What’s wrong with
my backend?

8. perfecto.io8 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Noise Reduction – Continuous Testing Enabler
“About 10 – 20
defects per 1000 lines
of code during in-
house testing, and 0.5
defect per KLOC in
production”
(Sogeti Labs)
~1 real defect per
developer per day
Regression suite of
10,000 tests == ~20% failures
Out of 2000 failures,
only 20 are actual bugs.
1980 are false negatives!
Reporting noise is
a risky & bad practice.
•Reporting crashes, errors in functionality,
performance issues – Dev Friendly
•Lab issue, object not found, popups, alerts,
orchestration, etc. should never be taken to Dev

9. perfecto.io9 | Perfecto by Perforce © 2020 Perforce Software, Inc.
• Think continuous value.
• Think maintenance and complexity to create, execute, monitor, and maintain (ROI).
• Test reliability over time.
Test Automation Certification
If { (Test Scenario) == Candidate to add Value add
{ and x– It detect defects}
{ and – It passes across multiple platforms with consistent result}
{ and – Its execution time is < 5 minutes }
Then
{ Include in Test Suite & & CI }
Else {
}
Run such certification
once in every 2-3
software iterations.

10. perfecto.io10 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Measuring Test Automation ROI Value
Value of Test
Automation
•Cycle testing time per iteration
•Test automation types (functional, non-functional, unit)
•Error detection efficiencies
•Product coverage (key business scenarios)
Cross-Team
Collaboration
•Three personas sharing load (resource, smart utilization of platforms)
•Manual testing %
•Backlog of test automation between iterations (in-sprint automation)
•Test debugging and analysis time per cycle
Test
Execution
•Automation within CI success rate
•Execution across platforms success rate
•Test redundancy/duplication/waste
•Coverage across environments (test, production)

11. perfecto.io
High-Level Comparison Between Tools and Practices
Test Framework Supported
Dev
Languages
Supported
Platforms
Supported
Test
Frameworks
Setup and
Execution
Integrations Breadth of
Testing
Options
Maturity,
Documentation,
Support
Required Skillset Cloud and Execution at
Scale
Selenium/Appium
WebDriver
Java, C#,
Java Script,
Python,
Ruby,
Objective-
C
Chrome,
Safari,
Firefox,
Edge,
IE/iOS/Andr
oid
Mocha JS, Jest,
other super
set on top of
Selenium
(Protractor,
WebDriverIO,
etc.)
Download
relevant
driver, set up a
grid, network
and location
impacts
execution
speed
Plenty of
integrations
(CI, CD,
reporting,
visual testing,
cloud vendors)
End-to-
end,
security,
unit,
Robust community,
multiple bindings, best
practices
Coding skills required
(SDET Oriented)
Perfecto fully supports
Selenium and its
WebDriver
configurations. Local
execution requires
setting up a Selenium
grid
XCUITest/Espresso/
Headless/Cypress
Objective
C/Java/Jav
a Script
Chrome,
Electron
NA Embedded
into IDEs,
headless
bundles a
browser in the
FW
CI/CD UI/Unit Good documentation
and code samples
Dev Oriented Built-in Chrome/Firefox
browsers in headless,
Perfecto Cloud supports
scaling
Espresso/XCUITest
Codeless Irrelevant,
based on
record and
playback
All Proprietary UI
with
underlying
Selenium
WebDriver
APIs
Mostly
SaaS/browser
plugin
installation
Limited Functional
/UI
Growing, limited No coding skills
required (Business
Tester Oriented)
Perfecto supports
codeless in the cloud
BDD Java, Ruby,
JS, Kotlin
All Junit,
Selenium,
Appium
Open source,
Maven/Gradle
/TestNG
Plenty + APIs
(e.g. Rest
Assured)
Functional Robust community,
docs, adoption
Step-definition
development in code is
required/scenarios are
no-code (Mix of
Business Tester and
SDETs)
Perfecto Quantum is a
web/mobile BDD
framework

12. perfecto.io12 | Perfecto by Perforce © 2020 Perforce Software, Inc.
• Fast feedback
• > Coverage of platforms
• < Execution of time constraints
The Need for Scale

13. perfecto.io13 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Scaling Best Practices
What to Test
(Scenarios + Platforms)?
Why These (Value)? When in the Cycle,
How Often (CI/Outside)?

14. perfecto.io14 | Perfecto by Perforce © 2020 Perforce Software, Inc.
• Stick to time and coverage objectives
• Burst to lower overall execution time
• Include more platform permutations in the cycle
• Continuously monitor your tests
• Improve test suites based on history
• Retire obsolete test scenarios
• Data-driven decision making — release/tests/platforms
• Scale wisely inside and outside your CI
• In CI:
• Subset of functional and non-functional (security, performance, accessibility)
• Build acceptance, unit, API
• Outside CI:
• Niche tests | Flaky/inconsistent test scenarios | Unchanged areas based testing
Concepts for Scaling Test Automation

15. perfecto.io15 | Perfecto by Perforce © 2020 Perforce Software, Inc.
• History of failures
• Test vs. device issue
• Specific areas failures
• Script development practices
• Popup handling
• Element not found
• Are the tests brining any value?
• Platform health? Lab?
Test Data Analysis – Act Upon Early Warnings

16. perfecto.io16 | Perfecto by Perforce © 2020 Perforce Software, Inc.
• Testing with Selenium (Java) the Boston Globe RWD site at scale
• Smart analytics with CI dashboard and cross-platform comparison
• Technologies used:
• IntelliJ Idea
• Selenium
• TestNG/Maven
• Perfecto
Live Demo

17. perfecto.io17 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Create Your Own RCA?

18. perfecto.io18 | Perfecto by Perforce © 2020 Perforce Software, Inc.
The 4 Key Pillars of Continuous Testing in DevOps
This is where money is spent (Time, Tools, Resources)
This is where value is being
realized, quality is improved
Unified solution serves DevOps with greater effectiveness.

19. Thank You!
Visit perfecto.io or follow us @perfectomobile