Getting value from your test automation is fundamental for fast feedback, risk reduction, and return on investment from your testing activities. Once developing the test scenarios, teams cannot stop monitoring and ensuring that their tests continuously bring value, are not flaky, and can support the latest functionalities in your web and mobile apps. Teams often “forget” about their tests once they have been developed and integrated into the CI pipeline — regardless of the value they bring. In this session, learn how to optimize your Appium and Selenium test suites so you can get more value from them.

1. 1 | 4 Key Pillars of Smart Continuous Testing in DevOps perfecto.io
How to Guarantee Value from your Test Automation

2. 2 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
• Chief Evangelist and author at Perfecto
• Blogger and speaker
• 19+ years in development & testing
• Author of
“The Digital Quality Handbook”
and “Continuous Testing for
DevOps Professionals”
• Twitter: @ek121268
• Blog: continuoustesting.blog
• Website: perfecto.io
Today’s Speaker:
Eran Kinsbruner

3. 3 | How to Guarantee Continuous Value from your Test Automation perfecto.io
Today’s Agenda
1
2
3
Agile Testing Manifesto
How to Decide Which Tests to Automate
Test Case Certifications in Continuous Testing
4 Test Maintenance and Test Aging
5 Live Demo

4. 4 | How to Guarantee Continuous Value from your Test Automation perfecto.io
The Agile Testing Manifesto
T H E D E V O P S M A N I F E S T O
 Continuous testing over testing at the end.
 Embracing all testing activities over only automated functional testing.
 Testing what gives value over testing everything.
 Testing across the team over testing in siloed testing departments.
 Product coverage over code coverage.

5. 5 | How to Guarantee Continuous Value from your Test Automation perfecto.io
Shift Left: Manual vs. Automated Testing Impact

6. 6 | How to Guarantee Continuous Value from your Test Automation perfecto.io
Are You Measuring Your DRE?
DRE (Defect Removal Efficiency) =
Defects removed during the development phase
Defects detected later in the cycle (UAT, Production)
Coverage, lack of and late automation testing, designed
for testability, unit testing, outdated environments/platforms.

7. 7 | How to Guarantee Continuous Value from your Test Automation perfecto.io
1. What’s the test engineer’s gut feeling
😊
2. Risk calculated as probability to occur
and impact to customers.
3. Value — Does the test provide new
information and, if failed, how much
time to fix?
4. Cost efficiency to develop — How
long does it take to develop and how
easy is it to script?
5. History of test — Volume of historical
failures in related areas and
frequency of breaks.
Source: Angie Jones
Test Automation – What to Automate?
S C O R E S : 67-100 = Automate | 34-66 = Possibly Automate | 0-33 = Don’t Automate

8. 8 | How to Guarantee Continuous Value from your Test Automation perfecto.io
Certifying Test Automation

9. 9 | How to Guarantee Continuous Value from your Test Automation perfecto.io
Test Automation Certification
• Think continuous value.
• Think maintenance and complexity to create, execute, monitor, and maintain (ROI).
• Test reliability over time.
If { (Test Scenario) == Candidate to add Value
{ and x– It detect defects}
{ and – It passes across multiple platforms with consistent result}
{ and – Its execution time is < 5 minutes }
Then
{ Include in Test Suite & & CI }
Else {
}
Periodic Certification
(Per 2-3 software
iterations)

10. 10 | How to Guarantee Continuous Value from your Test Automation perfecto.io
Getting Started With Continuous Testing (Decision Making Process)
• Test Objectives
• What should it cover? Not overly complicated things.
• Should it be part of each build-acceptance/CI job? (Keep the pipeline GREEN)
• Reliability when running across multiple platforms/OS versions?
• Time to develop the test and maintain it?
• Coverage area should be top of mind.
• Environments (Test, Staging, Production)
• Target Platforms (Real devices vs Simulators, Browsers vs. Headless)
• Test development early warnings (mark tests as “flaky”).

11. 11 | Automated Testing in DevOps perfecto.io
ANALYSIS
Minimum capabilities needed:
”Single pane of glass" for visibility and fast triaging
CREATION EXECUTION LAB ANALYSIS
Scripts & FW Issues (40%)
Backend Issues (30%)
Lab Issues (15%)
Orchestration Issues (15%)
OBJECTS (40%) | CODING (20%) | TIME (20%) | OTHER (20%)
AVAILABILITY (40%) | NETWORK (20%) | DATA (30%) | OTHER (10%)
NETWORK (30%) | STABILITY (30%) | CONFIGURATION (20%) | LOCK (10%) | OTHER (10%)
IN-USE (40%) | DISCONNECTED (30%) | LICENSES (20%) | OTHER (10%)
40%30%
15%15%
What’s wrong with
my scripts?
What’s wrong with
my backend?
What’s wrong with
my lab?
What’s wrong with
my executions?
III. Perfecto’s Smart Continuous Testing Solution

12. 12 | Continuous Testing Strategy in DevOps perfecto.io
Look Into Test Data As an Ongoing Practice

13. 13 | Continuous Testing Strategy in DevOps perfecto.io
Look Into Test Data As an Ongoing Practice
1. Long history of failures
1. Test vs. device issue
2. Exclude and debug
2. Script development practices
1. Popup handling
2. Element not found
3. Are the 187 tests bringing
any value??
4. Real devices vs. Emulators

14. 14 | How to Guarantee Continuous Value from your Test Automation perfecto.io
Personas and Tools Balance

15. 15 | How to Guarantee Continuous Value from your Test Automation perfecto.io
I N T E R A C T I V E
T E S T S
UI/UX manual tests
Balancing Test Creation for the Three Different
Personas With the Right Tools
O P T I M I Z E D M O D E L
Developers & SDETs
(Code-Based)
Business Testers
Ownership (Codeless)
Business Testers
Ownership

16. 16 | How to Guarantee Continuous Value from your Test Automation perfecto.io
Collaborative Approach — Codeless & Code-Based

17. 17 | Quest 2019 perfecto.io
Tool/Framework
Selection Criteria:
Technical Fit &
Skills
SDLC
process fit
(integration,
plug-ins, skills,
etc.)
Community
size, support,
and docs
Feedback loop
and reporting
Automation
coverage
Cloud and
automation at
scale
Automation
robustness and
maintainability

18. 18 | How to Guarantee Continuous Value from your Test Automation perfecto.io
How to Measure Test Value
Through Reporting and Analytics

19. 19 | How to Clean Up Your Continuous Testing Suites perfecto.io
Error Classification and Test Productivity

20. 20 | How to Guarantee Continuous Value from your Test Automation perfecto.io
The Power of Test Data Analysis
1. Long history of failures
A. Test vs. device issue
B. Exclude and debug
2. Script development practices
A. Popup handling
B. Element not found
3. Are the tests bring any
value?
4. Platform Health?

21. 21 | How to Guarantee Continuous Value from your Test Automation perfecto.io
Slicing and Dicing Test Data

22. 22 | How to Guarantee Continuous Value from your Test Automation perfecto.io
How Do Tests Perform Within CI?

23. 23 | How to Guarantee Continuous Value from your Test Automation perfecto.io
Object Identification Weights Based on History

24. 24 | 4 Key Pillars of Smart Continuous Testing in DevOps perfecto.io
The 4 Key Pillars of Continuous Testing in DevOps
ANALYSISCREATION EXECUTION LAB
Unified solution serves DevOps with greater efficiency.
Lab for continuous testing
• Backbone for all testing
activities
• Hosting of devices,
management, & logistics
• Availability & scaling
• Supporting functions
Test execution within the
pipeline
• Scale
• Orchestration
• Frameworks
Test creation and
maintenance
• Authoring tool
• Validations
Test analysis
• Fast feedback
• Root cause analysis

25. 25 | How to Guarantee Continuous Value from your Test Automation perfecto.io
LIVE DEMO
• Execution of BDD test code in parallel.
• Execution of codeless Selenium in parallel.
• ML/AI in action (object locators & test analytics).
L I V E D E M O

26. 26 | How to Guarantee Continuous Value from your Test Automation perfecto.io
Measuring Continuous Testing Success in DevOps

27. 27 | How to Guarantee Continuous Value from your Test Automation perfecto.io
Some Recommended CT Measurements (KPIs)
• How fast are testing activities moving, and
what is slowing down these activities?
• Test flakiness.
• Test duration.
• Percentage of automated vs. manual tests.
• Application quality measurements.
• Number of escaped defects and in which areas.
• MTTD — mean time to detection of defect.
• Build quality.
• Pipeline efficiency measurements.
• Number of user stories implemented per iteration.
• Test automation as part of DoD across iterations.
• Broken builds with categories.
• CI length trending.
• Lab availability and utilization.
• Quality cost measurements.
• Operational costs, lab availability issues.
• Cost of hardware/software.
• Costs of defects by severity and stage

28. 28 | 4 Key Pillars of Smart Continuous Testing in DevOps perfecto.io
Perfecto’s Smart Continuous Testing Solution
Smart Execution
Fast and parallel test execution with
multi-team orchestration abilities and
management.
Smart Analytics
"Single pane of glass" provides
visibility and scales to support
millions of test results.
Smart Creation
Automation creation that matches your
team’s skillset (Appium, Espresso,
XCUITest, Quantum BDD, Codeless).
Smart Lab
Always on and stable. Always up
to date. Supports all mobile OS
and platforms.
The Perfecto
human factor
increases your
chances to succeed.
Black Belt
Testing Experts
Training
Dedicated
Success Manager
24/7 VIP
Support

29. 29 | 4 Key Pillars of Smart Continuous Testing in DevOps perfecto.io

30. 30 | How to Guarantee Continuous Value from your Test Automation perfecto.io
Thank You