Software testing tools are evolving. More testing frameworks are emerging through the open source community and commercial vendors. In addition, we’re starting to see the rise of machine-learning (ML) and artificial intelligence (AI) in testing solutions.
Given this evolution, it is important to map the tools that match both the practitioners’ skills and their testing types. When referring to the testing practitioners, we mainly look at three different personas:
-The business tester
-The software developer in test (SDET)
-The software developer
These practitioners are tasked with creating, maintaining, and executing unit tests, build acceptance tests, integration, regression, and other nonfunctional tests.
In this webinar led by Perfecto’s Chief Evangelist, Eran Kinsbruner, you will learn the following:
-How should testing types be dispersed among the three personas and throughout the DevOps pipeline?
-What tools should each of these three personas use for the creation and execution of tests?
-What are the key benefits to continuous testing when mapped correctly?
3. 3 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Today’s Speaker:
Eran Kinsbruner
• Chief Evangelist and author at Perfecto
• Blogger and speaker
• continuoustesting.blog
• infoworld.com/author/Eran-Kinsbruner/
• enterprisersproject.com/user/eran-kinsbruner
• 19+ years in development & testing
• Author of
“The Digital Quality Handbook” and
“Continuous Testing for DevOps
Professionals”
• Twitter: @ek121268
4. 4 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Today’s Agenda
1
2
3
The State and Definition of Continuous Testing
How Should Testing Types Be Dispersed Among
the Three Personas and Throughout the DevOps Pipeline?
Which Tools Should Each of These Three Personas Use
For the Creation and Execution of Tests?
4
What Are the Key Benefits to Continuous Testing
When Mapped Correctly?
5 Q & A
5. 5 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
What is Continuous Testing?
Continuous testing is the process of executing automated
tests as part of the software delivery pipeline in order to
obtain feedback as rapidly as possible.
6. 6 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Continuous Testing in DevOps Has Never Been More Complex
Digital Transformation is Happening Now.
How Does It Impact Your Business?
7. 7 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
What the DevOps Process Probably Looks Like
Process
Impact
Organizational
Impact
2-3 WEEKS 1-3 WEEKS
❌ Unstable
❌ Labor-intensive
❌ Cluttered
❌ Slow
• Manual testing.
• Unreliable and flaky
executions.
• Long time to analyze results.
• Increases time to release.
• Increases risk and reduces
flexibility during the cycle.
• Reduces innovation time
versus bug fixes time.
• QA isn’t part of the daily cycle
since they have limited
feedback to share.
• Testing holds back
innovation.
”End of
Cycle”
Testing
8. 8 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
CONTINUOUS
What the DevOps Process Should Look Like
9. 9 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
The Path to Continuous Testing
Continuous testing is the process of executing automated tests as
part of the software delivery pipeline in order to obtain feedback
as rapidly as possible.
Removes the testing bottleneck.
Significantly reduces risk with on-going quality visibility.
Allows fixes DURING the sprint rather than AFTER it.
Significantly reduces developers’ bug fixing time and increases
innovation time.
Turns testing into VALUE instead of LIABILITY
B E N E F I T S
10. 10 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
The Path to Continuous Testing
1 2 3 4 5
Stable automation Daily cycle Increase coverage Reach 95% Continuous testing
W H A T Y O U ’ L L G E T
W H A T Y O U ’ L L N E E D
• 99.9% availability lab
• Evidence collection
• Process integration
• CI
• Defect tracking
• Execution control
• Dashboarding
• Create scripts
• Maintain scripts
• Understand what’s
wrong
• Skillset matched
tool
• False negative
detection in reports
• Run scripts daily
• Get results in
minutes
• Valuable coverage
• Meaningful daily
feedback
• Advance validations
• Visual automation
• Basic orchestration
• Up-to-date lab
• Scaled lab
• Analysis grouping
• Automate all that
possible
• Advanced
orchestration
• Elasticity
• Sharding
• Environment
control
• Scaled reporting
• Nightly
Continuously
• Cloud execution
• Very high scale
D E V O P S F R I E N D L Y Z O N E
11. 11 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
1. How should testing types be
dispersed among the three personas
and throughout the DevOps pipeline?
12. 12 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
13. 13 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
A Plethora of Tools Support the Entire Pipeline
14. 14 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Let’s Focus on Testing
Testing Is Led By 3 Main Personas
B U S I N E S S T E S T E R
S D E T ( S O F T W A R E D E V E L O P E R I N T E S T I N G )
S O F T W A R E D E V E L O P E R
Performs exploratory and structured manual testing
Creates functional and nonfunctional code-based test
automation scripts (via Selenium, Appium, etc.)
Creates unit and build acceptance tests
15. 15 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Business Tester:
Manual? Exploratory?
BDD? Codeless?
How Should Testing Types Be Dispersed Among the
Three Personas and Through the DevOps Pipeline?
P L A N C O D E B U I L D T E S T D E P L O Y R U N
DevOps Continuous Feedback
Test Platform
Test FW
Trigger
Feedback
Time
90% Emulator/Local Device
10% Cloud Device
Unit
API
Espresso / XCUITest
Exploratory
Pre-commit
Manually – On-demand
Few Minutes
70% Emulator
30% Cloud Devices
API
Espresso / XCUITest
Commit / Few # day
Automatic
< 5-10 Min
100% Cloud Devices
Espresso/XCUITest
Appium
Nightly
Automatic
< 6 Hours
100% Real Devices (S)
Espresso/XCUITest
24/7
Automatic
< 5 Min
SDET
Functional/NF
Developer
Unit/BAT
16. 16 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
2. Which tools should each of these
three personas use for the creation and
execution of tests?
19. 19 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Test Authoring Workflow and
Skillset Required
Define manual test scenarios, BDD & user
stories style scenarios
Create scripts in Java/JS etc. from within IDEs
(IntelliJ, Eclipse)
Define Objects using Object Spy tools, DOM
viewers etc.
Page based test steps creation
Insert Visual validations and Assertions
Typically takes longer to develop, more complex
(~6 Hours per test)
Medium-High code development skills required
Test Maintenance
Test changes are required proactively, tests are more
error-prone to changes in production, objects
Tests are managed and maintained in an SCM tool
(GIT, Perforce)
Tests are recorded with no coding in most
cases
Codeless tools UI used for creation
Objects are “learned” and generated on-the-fly
Test scenarios are often less structured and
more exploratory based (flow based)
Time to author is shorter, can be ~1 hour per
test
Test reusability is easy
Business tester, lower to no-coding skills is
sufficient
Tools support self-healing with auto test correction
Local or cloud test versioning is used, no GIT
integration
SDET/Developers Business Testers (&Developers?)
Mindset and Workflow Changes in Test Automation
20. 20 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Tools Maturity
Highly mature, includes samples, best
practices, documentation
Integrations exists to ALM tools, Defect
management etc.
Testing Types and App Types
Supported
Functional, API, Load, others
Mobile Native (Appium e.g.) and Desktop
Web (Selenium e.g.)
Emerging technology, less mature, no well-
defined guidelines and practices (e.g. guide
to shift from standard to ML?)
Web is more mature than mobile codeless,
basic integrations only
Mostly functional (E2E) and basic API
Most support web, mobile is lagging behind
SDET/Developers Business Testers (&Developers?)
Test Execution
Configured environment using tools like
TestNG data provider
Execution done locally, via CI, cloud-based
Execution management is built-in the
codeless tools
Execution done locally, via CI, cloud-based
Mindset and Workflow Changes in Test Automation
21. 21 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
1. Identify manual and/or flaky test steps
2. Mark these as “Codeless” to disconnect them from the
code-based automation suite (avoid duplications)
3. Record the tests in a “SMART” manner:
- Flow based & Exploratory recording
- Reusable components
- Robust Object Locators
- Tagging for reporting and tracking
4. Connect codeless to CI and pipeline for continuous
testing
5. Identify maintenance and examination points for both
suites
5 Recommended Practices
for Codeless Testing
22. 22 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
3. What Are the Key Benefits to
Continuous Testing When Mapped
Correctly?
23. 23 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
CREATION
EXECUTION
LAB
ANALYSIS
SCALE
• Few executions/day
• 2 platforms
• 10 executions/day
• 2 platforms
• 1000 executions/day
• 10-20 platforms
• 25 concurrent
• 10K executions/day
• 25-50 platforms
• 25-150 concurrent
• 100K executions/day
• 25-50 platforms
• 1000 concurrent
1. Stable automation 2. Daily cycle 3. Increase coverage 4. Reach 95% 5. Continuous testing
Q A D E V
What Are the Key Benefits to Continuous Testing When Mapped Correctly?
24. 24 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
The Key Pillars of Continuous Testing in DevOps
ANALYSISCREATION EXECUTION LAB
Unified solution Serves DevOps with greater effectiveness.
Lab for continuous testing
that serves as the backbone
for all testing activities.
• Hosting of devices,
management & logistics
• Availability & scaling
• Supporting functions
Test execution within the
pipeline
• Scale
• Orchestration
• Frameworks
Test creation and
maintenance
• Authoring tool
• Validations
Test analysis
• Fast feedback
• Root cause analysis
25. 25 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Perfecto’s Smart Automation for Continuous Testing
SMART Test Execution
Fast and parallel test execution with
multi-team orchestration abilities and
management.
SMART ML-Driven Test
Reporting and Analysis
"Single pane of glass" provides
visibility and scales to support
millions of test results.
SMART Test Creation
Automated creation that matches your
team’s skillset (Appium, Espresso,
XCUITest, Quantum BDD).
SMART CLOUD Lab
Always on and stable. Always up
to date. Supports all mobile OS
and platforms.
The Perfecto
human factor
increases your
chances to succeed.
Black Belt
Testing Experts
Training
Dedicated
Success Manager
24/7 VIP
Support
26. 26 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
About Perfecto
27. 27 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Perfecto Is a Perforce Company
A G I L E
P L A N N I N G
C O D E M G M T. &
C O L L A B O R AT I O N
E M B E D D E D
C O M P O N E N T S
O P E R AT I O N S
T O O L S
A U T O M AT E D
T E S T I N G
28. 28 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
29. 29 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Thank You