The document discusses continuous testing in the context of DevOps, defining it as the automation of tests within the software delivery pipeline to achieve rapid feedback. It addresses critical questions on the distribution of testing responsibilities across different roles, the appropriate tools for each persona, and the benefits of correctly implemented continuous testing. The key pillars include stable automation, increased coverage, and the integration of various tools and practices to support effective DevOps processes.

1. 1 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Solving the 3 Biggest Questions
in Continuous Testing

2. 2 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
2© 2018, Perfecto Mobile Ltd. All Rights Reserved.
Housekeeping
1 2
Ask us questions in
the Q&A panel
Join our live poll
question
3
We will send you the
recording and the slides

3. 3 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Today’s Speaker:
Eran Kinsbruner
• Chief Evangelist and author at Perfecto
• Blogger and speaker
• continuoustesting.blog
• infoworld.com/author/Eran-Kinsbruner/
• enterprisersproject.com/user/eran-kinsbruner
• 19+ years in development & testing
• Author of
“The Digital Quality Handbook” and
“Continuous Testing for DevOps
Professionals”
• Twitter: @ek121268

4. 4 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Today’s Agenda
1
2
3
The State and Definition of Continuous Testing
How Should Testing Types Be Dispersed Among
the Three Personas and Throughout the DevOps Pipeline?
Which Tools Should Each of These Three Personas Use
For the Creation and Execution of Tests?
4
What Are the Key Benefits to Continuous Testing
When Mapped Correctly?
5 Q & A

5. 5 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
What is Continuous Testing?
Continuous testing is the process of executing automated
tests as part of the software delivery pipeline in order to
obtain feedback as rapidly as possible.

6. 6 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Continuous Testing in DevOps Has Never Been More Complex
Digital Transformation is Happening Now.
How Does It Impact Your Business?

7. 7 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
What the DevOps Process Probably Looks Like
Process
Impact
Organizational
Impact
2-3 WEEKS 1-3 WEEKS
❌ Unstable
❌ Labor-intensive
❌ Cluttered
❌ Slow
• Manual testing.
• Unreliable and flaky
executions.
• Long time to analyze results.
• Increases time to release.
• Increases risk and reduces
flexibility during the cycle.
• Reduces innovation time
versus bug fixes time.
• QA isn’t part of the daily cycle
since they have limited
feedback to share.
• Testing holds back
innovation.
”End of
Cycle”
Testing

8. 8 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
CONTINUOUS
What the DevOps Process Should Look Like

9. 9 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
The Path to Continuous Testing
Continuous testing is the process of executing automated tests as
part of the software delivery pipeline in order to obtain feedback
as rapidly as possible.
Removes the testing bottleneck.
Significantly reduces risk with on-going quality visibility.
Allows fixes DURING the sprint rather than AFTER it.
Significantly reduces developers’ bug fixing time and increases
innovation time.
Turns testing into VALUE instead of LIABILITY
B E N E F I T S

10. 10 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
The Path to Continuous Testing
1 2 3 4 5
Stable automation Daily cycle Increase coverage Reach 95% Continuous testing
W H A T Y O U ’ L L G E T
W H A T Y O U ’ L L N E E D
• 99.9% availability lab
• Evidence collection
• Process integration
• CI
• Defect tracking
• Execution control
• Dashboarding
• Create scripts
• Maintain scripts
• Understand what’s
wrong
• Skillset matched
tool
• False negative
detection in reports
• Run scripts daily
• Get results in
minutes
• Valuable coverage
• Meaningful daily
feedback
• Advance validations
• Visual automation
• Basic orchestration
• Up-to-date lab
• Scaled lab
• Analysis grouping
• Automate all that
possible
• Advanced
orchestration
• Elasticity
• Sharding
• Environment
control
• Scaled reporting
• Nightly 
Continuously
• Cloud execution
• Very high scale
D E V O P S F R I E N D L Y Z O N E

11. 11 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
1. How should testing types be
dispersed among the three personas
and throughout the DevOps pipeline?

12. 12 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io

13. 13 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
A Plethora of Tools Support the Entire Pipeline

14. 14 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Let’s Focus on Testing
Testing Is Led By 3 Main Personas
B U S I N E S S T E S T E R
S D E T ( S O F T W A R E D E V E L O P E R I N T E S T I N G )
S O F T W A R E D E V E L O P E R
Performs exploratory and structured manual testing
Creates functional and nonfunctional code-based test
automation scripts (via Selenium, Appium, etc.)
Creates unit and build acceptance tests

15. 15 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Business Tester:
Manual? Exploratory?
BDD? Codeless?
How Should Testing Types Be Dispersed Among the
Three Personas and Through the DevOps Pipeline?
P L A N C O D E B U I L D T E S T D E P L O Y R U N
DevOps Continuous Feedback
Test Platform
Test FW
Trigger
Feedback
Time
90% Emulator/Local Device
10% Cloud Device
Unit
API
Espresso / XCUITest
Exploratory
Pre-commit
Manually – On-demand
Few Minutes
70% Emulator
30% Cloud Devices
API
Espresso / XCUITest
Commit / Few # day
Automatic
< 5-10 Min
100% Cloud Devices
Espresso/XCUITest
Appium
Nightly
Automatic
< 6 Hours
100% Real Devices (S)
Espresso/XCUITest
24/7
Automatic
< 5 Min
SDET
Functional/NF
Developer
Unit/BAT

16. 16 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
2. Which tools should each of these
three personas use for the creation and
execution of tests?

17. 17 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
2/28/2019 17© 2015, Perfecto Mobile Ltd. All Rights Reserved.
BDD/Codeless
Scripting
Flaky and
common test
failures and
complex manual
test cases
Interactive Tests
UI/UX manual tests
High Skilled
(Dev/SDET)
code-based
scripting
Developers and
SDETs
Business Testers
Ownership
Business Testers
Ownership
Balancing Test Creation for the Three Different
Personas with the Right Tools

18. 18 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
2/28/2019 18© 2015, Perfecto Mobile Ltd. All Rights Reserved.
Interactive Tests
UI/UX manual tests
Developers and
SDETs
Business Testers
Ownership
Business Testers
Ownership
Balancing Test Creation for the Three Different
Personas with the Right Tools

19. 19 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Test Authoring Workflow and
Skillset Required
 Define manual test scenarios, BDD & user
stories style scenarios
 Create scripts in Java/JS etc. from within IDEs
(IntelliJ, Eclipse)
 Define Objects using Object Spy tools, DOM
viewers etc.
 Page based test steps creation
 Insert Visual validations and Assertions
 Typically takes longer to develop, more complex
(~6 Hours per test)
 Medium-High code development skills required
Test Maintenance
 Test changes are required proactively, tests are more
error-prone to changes in production, objects
 Tests are managed and maintained in an SCM tool
(GIT, Perforce)
 Tests are recorded with no coding in most
cases
 Codeless tools UI used for creation
 Objects are “learned” and generated on-the-fly
 Test scenarios are often less structured and
more exploratory based (flow based)
 Time to author is shorter, can be ~1 hour per
test
 Test reusability is easy
 Business tester, lower to no-coding skills is
sufficient
 Tools support self-healing with auto test correction
 Local or cloud test versioning is used, no GIT
integration
SDET/Developers Business Testers (&Developers?)
Mindset and Workflow Changes in Test Automation

20. 20 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Tools Maturity
 Highly mature, includes samples, best
practices, documentation
 Integrations exists to ALM tools, Defect
management etc.
Testing Types and App Types
Supported
 Functional, API, Load, others
 Mobile Native (Appium e.g.) and Desktop
Web (Selenium e.g.)
 Emerging technology, less mature, no well-
defined guidelines and practices (e.g. guide
to shift from standard to ML?)
 Web is more mature than mobile codeless,
basic integrations only
 Mostly functional (E2E) and basic API
 Most support web, mobile is lagging behind
SDET/Developers Business Testers (&Developers?)
Test Execution
 Configured environment using tools like
TestNG data provider
 Execution done locally, via CI, cloud-based
 Execution management is built-in the
codeless tools
 Execution done locally, via CI, cloud-based
Mindset and Workflow Changes in Test Automation

21. 21 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
1. Identify manual and/or flaky test steps
2. Mark these as “Codeless” to disconnect them from the
code-based automation suite (avoid duplications)
3. Record the tests in a “SMART” manner:
- Flow based & Exploratory recording
- Reusable components
- Robust Object Locators
- Tagging for reporting and tracking
4. Connect codeless to CI and pipeline for continuous
testing
5. Identify maintenance and examination points for both
suites
5 Recommended Practices
for Codeless Testing

22. 22 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
3. What Are the Key Benefits to
Continuous Testing When Mapped
Correctly?

23. 23 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
CREATION
EXECUTION
LAB
ANALYSIS
SCALE
• Few executions/day
• 2 platforms
• 10 executions/day
• 2 platforms
• 1000 executions/day
• 10-20 platforms
• 25 concurrent
• 10K executions/day
• 25-50 platforms
• 25-150 concurrent
• 100K executions/day
• 25-50 platforms
• 1000 concurrent
1. Stable automation 2. Daily cycle 3. Increase coverage 4. Reach 95% 5. Continuous testing
Q A D E V
What Are the Key Benefits to Continuous Testing When Mapped Correctly?

24. 24 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
The Key Pillars of Continuous Testing in DevOps
ANALYSISCREATION EXECUTION LAB
Unified solution Serves DevOps with greater effectiveness.
Lab for continuous testing
that serves as the backbone
for all testing activities.
• Hosting of devices,
management & logistics
• Availability & scaling
• Supporting functions
Test execution within the
pipeline
• Scale
• Orchestration
• Frameworks
Test creation and
maintenance
• Authoring tool
• Validations
Test analysis
• Fast feedback
• Root cause analysis

25. 25 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Perfecto’s Smart Automation for Continuous Testing
SMART Test Execution
Fast and parallel test execution with
multi-team orchestration abilities and
management.
SMART ML-Driven Test
Reporting and Analysis
"Single pane of glass" provides
visibility and scales to support
millions of test results.
SMART Test Creation
Automated creation that matches your
team’s skillset (Appium, Espresso,
XCUITest, Quantum BDD).
SMART CLOUD Lab
Always on and stable. Always up
to date. Supports all mobile OS
and platforms.
The Perfecto
human factor
increases your
chances to succeed.
Black Belt
Testing Experts
Training
Dedicated
Success Manager
24/7 VIP
Support

26. 26 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
About Perfecto

27. 27 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Perfecto Is a Perforce Company
A G I L E
P L A N N I N G
C O D E M G M T. &
C O L L A B O R AT I O N
E M B E D D E D
C O M P O N E N T S
O P E R AT I O N S
T O O L S
A U T O M AT E D
T E S T I N G

28. 28 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io

29. 29 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Thank You