No matter how good their test automation framework is, QA managers and engineers still get blamed whenever defects escape to production. This begs the question, why are critical defects — whether functional or non functional — escaping to production? In this virtual session, Eran Kinsbruner, Chief Evangelist at Perfecto by Perforce, will uncover the root causes behind escaped defects in traditional testing cycles. Eran will provide a prescriptive and proven approach to building a solid test automation coverage strategy that reduces such risks.

1. Perfecto by Perforce © 2020 Perforce Software, Inc.
How to Eliminate Escaped Defects With a Proven Test Automation
Coverage Strategy

2. 2 | Advanced Codeless Testing for Web Apps
z
Today’s Speaker:
Eran Kinsbruner
• Chief Evangelist and author at Perfecto
• Blogger and speaker
• 20+ years in development & testing
• Author of “The Digital Quality Handbook”
and “Continuous Testing for DevOps
Professionals”
• @ek121268

3. perfecto.io3 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Today’s Agenda
1
2
3
Connecting Quality to Digital, Agile, and DevOps Reality
Common Test Automation Pains
4
Test Automation Coverage Strategy
Ensuring Continuous Testing Suites Value Over Time
5 Q&A

4. Digital Engagement Reality — More Relevant Than Ever

5. perfecto.io5 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Valuable Test Automation to Optimize the Delivery Pipeline
vs
OPTIMIZED
DELIVERY PIPELINE Innovation
Throughout
Quality of
Output
Time Cost

6. perfecto.io6 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Common Pain Points in Test Automation
Automation Skillset
Merging Tests Into Pipeline
Escaped Defects Due to Noise
Advanced Automation Scenarios
Challenge to Setup Test Environment Designed for
Testability
Time Spent to Analyze Reports
Testing Is Done Separately
Test Maintenance & Digital Platforms Coverage Lack of Time to Automate
It all boils down to people, processes, and/or technology.

7. perfecto.io7 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Balancing Test Automation Coverage
TEST SCENARIO ONE
Test Case – 1 Test Case – 2 Test Case – 3 Test Case – N
2 sides of the same coin

8. Digital Platforms Test Coverage

9. perfecto.io9 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Digital Platform Coverage Strategy

10. perfecto.io10 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Digital Platform Coverage Strategy
Android OS Market Share
Android 6.x (Marshmallow) Android 7.x (Nougat) Android 8.x (Oreo) Android 9 (Pie) Android 10 (Q)

11. perfecto.io11 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Digital Platform Coverage Strategy
Device Model Custom OS Implementation Versions Base Stock OS
Samsung Galaxy S20 Ultra One UI 2 Android 10
Samsung Galaxy S7 TouchWiz UI Android 8.0
Samsung S8 One UI Android 9.0
Samsung Note 9 One UI 2 Android 10.0
Google Pixel 4 XL STOCK Android 10
Huawei Mate 20 Pro EMUI 9.1 Android 9.0
LG G8 ThinQ LG UX 9.0 Android 10.0
Oppo Reno 5G Color OS 6 Android 10
Huawei Y9 Prime Magic UI 2.1 Android 10

12. perfecto.io12 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Digital Platform Coverage Strategy

13. perfecto.io13 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Digital Platform Coverage Strategy —
Continuously Plan Your Lab Configuration

14. perfecto.io14 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Digital Platform Coverage Strategy
Jan Feb Mar Apr May Jun July Aug Sept
Chrome 79 80 81 82 83 Beta 83 84 Beta 84 85
Firefox 72 73 74 75 76 77
78
79 80 81
Safari 13.0.2
Edge 80 83
Beta
83 84 Beta 83 84 Beta
Desktop OS
Releases
Mac OS Catalina
10.15.3
MacOS
Next (TBD)

15. perfecto.io15 | Perfecto by Perforce © 2020 Perforce Software, Inc.
U N I T T E S T S
M A N UA L
T E S T S
Testing isolated units of code.
I N T E G R AT I O N T E S T S
Testing across integrations of more than one system
U I T E S T S
Testing through a user interface.
Per code commit/ Post commit
Build Acceptance/Regression Testing
E2E Testing
Exploratory/Manual Testing
Simulator Emulator
Dev
Triggered
Mix Real Devices & Emu./Simu.
CI
Triggered+
Real Devices Only
CI
Triggered
Real Devices OnlyProd. Monitoring
Non-Functional Testing:
Performance (UX)
Testing/Security
Perfecto’s Mobile Testing Pyramid ComboTesting Pyramid within Your DevOps Pipeline Requirements
Real Devices Only

16. Test Automation Scenarios Coverage Considerations

17. perfecto.io17 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Agile/DevOps Mandates Proper Mapping of Test Automation
Source: More Agile Testing: Learning Journeys
for the Whole Team” (Janet Gregory, Lisa Crispin)

18. perfecto.io18 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Implement a Relevant Test Automation Scenarios Strategy
Strate
gy
Technica
l
Skillset
Scope
Objectiv
es
ApproachSchedule
Environme
nt
Framewor
k and
Tools
Source: Thomas Haver

19. perfecto.io19 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Cost of Maintenance vs. Value to the Business

20. perfecto.io20 | Perfecto by Perforce © 2020 Perforce Software, Inc.
• Is the test or task going to be repeated? How often?
• How much time could be saved?
• Are the requirements, the test, or task low risk,
stable, or unlikely to change often?
• Is the test or task subject to human
error?
• Is the test or task time consuming?
Does it contain significant downtime
between test steps?
• Is the test or task repetitive?
What Should We Automate?
Source: Thomas Haver

21. perfecto.io21 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Combining Platform & Test Automation Scenarios – Golden Rules
Leverage market data and keep your
lab up to date and aligned with your
own customer usage data.
Continuously maintain, validate, and
modify your test automation suites.
Spread your test automation
executions throughout the pipeline
based on…
• Use calendars
• Follow market events
• Use web traffic analysis
• Match coverage to historical defects
• Keep your suite lean and valuable
• Never repeat failing tests without debugging
• Ensure your CI BAT and regression suites are
constantly updated
• Software iteration scope
• Regression needs
• Target features and platforms
• Phase in the CI/CD pipeline (use Test Pyramid)
• Persona, skillset, and tools

22. perfecto.io22 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Pipeline Testing Coverage Example

23. perfecto.io23 | Perfecto by Perforce © 2020 Perforce Software, Inc.

24. Thank You!
Visit perfecto.io or follow us @perfectomobile