As organizations struggle to adopt continuous deployment and mature DevOps, teams can face a number of roadblocks. DevOps maturity relies on culture, processes, and technology. Getting all of them right requires continuous alignment and communication. But even that is not enough. Join us in this live webinar led by Chief Evangelist, Eran Kinsbruner. During this webinar, Eran will cover the following topics: - How to reduce noise and waste within the DevOps pipeline (planning, handoffs, task switching, waiting time, over productivity, etc.). - How to decide what to automate. - How to automate as much of the CI/CD activities as possible. - How defects and rework can be avoided with a smart continuous testing process.

1. 1 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
4 Testing Methods to Scale and Automate Your
DevOps Pipeline

2. 2 | Getting Ready for iOS 13 Testing perfecto.io
• Chief Evangelist and author at Perfecto
• Blogger and speaker
• 19+ years in development & testing
• Author of
“The Digital Quality Handbook”
and “Continuous Testing for
DevOps Professionals”
• Twitter: @ek121268
• Blog: continuoustesting.blog
• Website: perfecto.io
Today’s Speaker:
Eran Kinsbruner

3. 3 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
Today’s Agenda
1
2
3
Matching a test automation framework to the right persona
Scaling requirements across the DevOps pipeline
Machine learning as a vehicle to optimize testing activities for dev and testers
4 Importance of dashboards and proper test analytics for a healthy pipeline
5 Live demos and Q&A

4. 4 | 4 Key Pillars of Smart Continuous Testing in DevOps perfecto.io
What Is Continuous Testing?
Continuous testing is the process of executing automated
high value tests as a part of the software delivery pipeline
in order to obtain feedback on business risks associated
with a software release.

5. 5 | 4 Key Pillars of Smart Continuous Testing in DevOps perfecto.io
Continuous Testing Reasoning
Source: Enterprise Continuous Testing, Wolfgang Platz

6. 6 | How to Guarantee Continuous Value from your Test Automation perfecto.io
Personas and Tools Balance

7. 7 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
Continuous Testing — Personas Involved
Testing is led by three main personas.
B U S I N E S S T E S T E R
S D E T ( S O F T W A R E D E V E L O P E R I N T E S T I N G )
S O F T W A R E D E V E L O P E R
Performs exploratory and structured manual testing.
Creates functional and nonfunctional code-based test
automation scripts (via Selenium, Appium, etc.).
Creates unit and build acceptance tests.

8. 8 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
I N T E R A C T I V E
T E S T S
UI/UX manual tests
Balancing Test Creation for the Three Different
Personas With the Right Tools
O P T I M I Z E D M O D E L
Developers & SDETs
(Code-Based)
Business Testers
Ownership
Business Testers
Ownership

9. 9 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
High-Level Comparison Between Tools and Practices
Test Framework Supported
Dev
Languages
Supported
Platforms
Supported
Test
Frameworks
Setup and
Execution
Integrations Breadth of
Testing
Options
Maturity,
Documentation,
Support
Required Skillset Cloud and Execution at
Scale
Selenium/Appium
WebDriver
Java, C#,
Java Script,
Python,
Ruby,
Objective-
C
Chrome,
Safari,
Firefox, Edge,
IE/iOS/Andro
id
Mocha JS, Jest,
other super
set on top of
Selenium
(Protractor,
WebDriverIO,
etc.)
Download
relevant
driver, set up a
grid, network
and location
impacts
execution
speed
Plenty of
integrations
(CI, CD,
reporting,
visual testing,
cloud vendors)
End-to-
end,
security,
unit,
Robust community,
multiple bindings, best
practices
Coding skills required
(SDET Oriented)
Perfecto fully supports
Selenium and its
WebDriver
configurations. Local
execution requires
setting up a Selenium
grid
XCUITest/Espresso
/Headless
Objective
C/Java/Java
Script
Chrome,
Electron
NA Embedded
into IDEs,
headless
bundles a
browser in the
FW
CI/CD UI/Unit Good documentation
and code samples
Dev Oriented Built-in Chrome/Firefox
browsers in headless,
Perfecto Cloud supports
scaling
Espresso/XCUITest
Codeless Irrelevant,
based on
record and
playback
All Proprietary UI
with
underlying
Selenium
WebDriver
APIs
Mostly
SaaS/browser
plugin
installation
Limited Functional
/UI
Growing, limited No coding skills
required (Business
Tester Oriented)
Perfecto supports
codeless in the cloud
BDD Java, Ruby,
JS, Kotlin
All Junit,
Selenium,
Appium
Open source,
Maven/Gradle
/TestNG
Plenty + APIs
(e.g. Rest
Assured)
Functional Robust community,
docs, adoption
Step-definition
development in code is
required/scenarios are
no-code (Mix of
Business Tester and
SDETs)
Perfecto Quantum is a
web/mobile BDD
framework

10. 10 | Quest 2019 perfecto.io
Tool/Framework
Selection Criteria:
Technical Fit &
Skills
SDLC
process fit
(integration,
plug-ins, skills,
etc.).
Community
size, support,
and docs.
Feedback loop
and reporting.
Automation
coverage.
Cloud and
automation at
scale.
Automation
robustness and
maintainability.

11. 11 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
LIVE DEMO
• BDD across platforms running in parallel
L I V E D E M O

12. 12 | How to Guarantee Continuous Value from your Test Automation perfecto.io
Scaling Automation Across the Pipeline

13. 13 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
Scaling Test
Automation
Throughout
the Pipeline
Platform scalability
Test suite scoping
Execution time objectives

14. 14 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
Poll Question
Test cycle analysis
Test environment set up
Manual testing
Reducing noise from test flakiness
Other
What’s the most time-consuming testing activity within a test cycle?

15. 15 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
PLATFORM SCALABILITY
Acknowledge Your Pipeline Testing Requirements
P L AT F O R M C O V E R A G E & T E S T S U I T E S C O P I N G T H R O U G H O U T
T H E D E V O P S P I P E L I N E

16. 16 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
TEST SUITE SCOPING & TIMING
Gather Testing Productivity Metrics
• Measurable Metrics:
• Test suite size — Number of unique test cases (Unit, Regression, Non-Functional, etc.).
• Average time per test — Time in minutes (2-3 minutes is a best practice).
• Test execution — Number of hours a cycle should run (Nightly, per build).
• Soft Metrics:
• Platform specific — Defects history, unique features support, etc.
• Analytics data
• Tests specific — Test flakiness and inconsistency.

17. 17 | Getting Ready for iOS 13 Testing perfecto.io
Machine Learning and Pipeline Optimization

18. 18 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
Key Use Cases for Machine-Learning/AI
Recognize objects — reliable
automation
Make quality-related decisions based
on data — pipeline visibility
Identify trends and/or patterns —
team productivity
Low/no code development

19. 19 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
Optimize Based on Data —
Recurring Patterns for Inefficiencies
Licenses In-use Disconnected
Orchestration issues
Network Availability Data
Backend issues
Objects Coding Time Other
Scripts & FW issues
• 10% of devices causing 80% of lab issues
Networking Stability Config Lock
Lab issues
Backend
30%
Lab
15%Orchestration
15%
Scripts & FW
40%
What’s wrong
with my scripts?
What’s wrong
with my
executions?
What’s wrong
with my lab?
What’s wrong with
my backend?

20. 20 | 4 Key Pillars of Smart Continuous Testing in DevOps perfecto.io
Noise Reduction – Continuous Testing Enabler
“About 10–20 defects per 1000 lines of code during in-
house testing, and 0.5 defect per KLOC in production”
(Sogeti Labs)
(~1 real defect per developer per day)
Regression suite of 10,000 tests = ~20% failures
Out of 2000 failures, only 20 are actual bugs,
1980 are false negative noise
Reporting noise is a risky bad practice
•Reporting crashes, errors in functionality, performance issues – Dev Friendly
•Lab issues, object not found, popups, alerts, orchestration, etc. – Should never be
taken to dev

21. 21 | How to Guarantee Continuous Value from your Test Automation perfecto.io
Test Data Analytics and Visibility Role

22. 22 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
The Power of Test Data Analysis
Ask yourself:
1. Is there a long history of
failures:
A. Test flakiness
B. Exclude and debug
2. Are we following script
development practices to
avoid:
A. Popup/security issues
B. Element not found
C. Platform-specific issues
3. Do the tests bring any new
value?
4. Platform/CI health?

23. 23 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
A Single Pane of Glass — Collaborative Approach

24. 24 | How to Clean Up Your Continuous Testing Suites perfecto.io
How Do Tests Perform Within CI?

25. 25 | Getting Ready for iOS 13 Testing perfecto.io
Perfecto’s Four Key Pillars for Continuous Testing

26. 26 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
The Four Key Pillars of Continuous Testing in DevOps
ANALYSISCREATION EXECUTION LAB
Unified solution serves DevOps with greater efficiency.
Lab for continuous testing
• Backbone for all testing
activities
• Hosting of devices,
management, & logistics
• Availability & scaling
• Supporting functions
Test execution within the
pipeline
• Scale
• Orchestration
• Frameworks
Test creation and
maintenance
• Authoring tool
• Validations
Test analysis
• Fast feedback
• Root cause analysis

27. 27 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
The 4 Key Pillars of Continuous Testing in DevOps
This is where money is spent (Time, Tools, Resources) This is where value is being realized, quality is improved

28. 28 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io
Perfecto’s Smart Continuous Testing Solution
Smart Execution
Fast and parallel test execution with
multi-team orchestration abilities and
management.
Smart Analytics
"Single pane of glass" provides
visibility and scales to support
millions of test results.
Smart Creation
Automation creation that matches your
team’s skillset (Appium, Espresso,
XCUITest, Quantum BDD, Codeless).
Smart Lab
Always on and stable. Always up
to date. Supports all mobile OS
and platforms.
The Perfecto
human factor
increases your
chances to succeed.
Black Belt
Testing Experts
Training
Dedicated
Success Manager
24/7 VIP
Support

29. 29 | 4 Testing Methods to Scale and Automate your DevOps Pipeline perfecto.io

30. 30 | Getting Ready for iOS 13 Testing perfecto.io
Thank You
Visit perfecto.io or follow us @perfectomobile