Enhancing your Test automation Scenario Coverage Using Selenium by Eran Kinsbrunber

1. Perfecto by Perforce © 2020 Perforce Software, Inc.
Enhancing your Test Automation Scenario Coverage with Selenium

2. 2 | Advanced Codeless Testing for Web Apps perfecto.io
Today’s Speakers:
Eran Kinsbruner
• Chief Evangelist and author at Perfecto
• Blogger and speaker
• 19+ years in development & testing
• Author of “The Digital Quality Handbook”
and “Continuous Testing for DevOps
Professionals”
• @ek121268

3. perfecto.io3 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Today’s Agenda
1
2
3
Continuous Testing & the Need for Scale
Cross Browser Testing & Mobile Landscape Overview
4
How to Scale Testing for Web & Mobile Apps
Demo and Q&A

4. Continuous Testing and Test Automation at Scale

5. perfectomobile.com
What Is Continuous Testing?
Continuous testing is the process of executing automated
high value tests as a part of the software delivery pipeline
in order to obtain feedback on business risks associated
with a software release upon every code change.

6. perfectomobile.com
Measuring Continuous Testing Success

7. Mature DevOps Organizations Are Aiming Higher
Source: Dora, https://codeclimate.com/blog/state-of-devops-2019/

8. perfecto.io
World Quality
Report 2019-2020

9. perfecto.io
World Quality
Report 2019-2020
Maturing DevOps and Agile
Actually Requires More Testing!

10. perfecto.io10 | Perfecto by Perforce © 2020 Perforce Software, Inc.
• Fast feedback
• > Coverage of platforms
• < Execution of time constraints
The Need for Scale

11. perfecto.io11 | Perfecto by Perforce © 2020 Perforce Software, Inc.
• Scaling Methods — Build vs. Buy
• DIY (aka Selenium Grid)
• Containerized Computing
• Cloud
• Firefox is switching to a monthly release cycle
The Need for Scale – Web Testing
W H A T ’ S Y O U R P R I M A R Y F O C U S ?

12. Market Overview

13. perfecto.io13 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Mobile Capabilities in Web Browser (Incomplete) — Ramping Up
CameraMic
Device Authentication
BT/BLE
Notification Network

14. perfecto.io14 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Mixing
Technologies
for Maximum
Test
Automation
Coverage
Selenium
Headless (Puppeteer)
Codeless (Perfecto, Selenium IDE)
BDD

15. perfecto.io15 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Codeless
Open
Source
Cross-Browser Testing Market Is Highly Fragmented

16. perfecto.io16 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Cross-Browser Testing Tools (Open-Source) — Download Trends Vary
Protractor
Puppeteer
WebDriver IO
Cypress IO
Selenium WebDriver
Source: npmtrends

17. perfecto.io17 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Tools that...
• Provide a testing structure (BDD/ATDD) (Mocha, Jasmine, Jest, Cucumber).
• Provide assertion functions (Chai, Jasmine, Jest, Unexpected).
• Generate, display, and watch test results (Mocha, Jasmine, Jest, Karma).
• Generate code coverage reports (Istanbul, Jest, Blanket).
• Provide a browser or browser-like environment with control of scenario execution, UI testing, and more
(Protractor, Nightwatch, Phantom, Casper, Selenium, WebDriver.IO, TestCafe).
• Provide mocks, spies, and stubs (Sinon, Jasmine, enzyme, Jest, testdouble).
Additional Categorization of JavaScript Browser Testing Tools
Source: Medium.com

18. perfecto.io18 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Motivation
• Receive fast feedback (unit and basic functional tests).
• Easy environment setup, no IT dependencies.
• Uses standard JS language.
• Supported and promoted by browser vendors (Google, Mozilla).
• Additional use cases to cover:
• HAR file
• Performance testing
• Basic UI
• Security
Headless Browsers Role

19. How to Scale Test Automation

20. perfecto.io20 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Continuous Testing — Personas Involved
Testing is led by three main personas.
B U S I N E S S T E S T E R
S D E T ( S O F T W A R E D E V E L O P E R I N T E S T I N G )
S O F T W A R E D E V E L O P E R
Performs exploratory and structured manual testing.
Creates functional and nonfunctional code-based test
automation scripts (via Selenium, Appium, etc.).
Creates unit and build acceptance tests.

21. 21 | Advanced Codeless Testing for Web Apps perfecto.io
Code-Based AutomationCodeless Automation

22. perfecto.io
High-Level Comparison Between Tools and Practices
Test Framework Supported
Dev
Languages
Supported
Platforms
Supported
Test
Frameworks
Setup and
Execution
Integrations Breadth of
Testing
Options
Maturity,
Documentation,
Support
Required Skillset Cloud and Execution at
Scale
Selenium/Appium
WebDriver
Java, C#,
Java Script,
Python,
Ruby,
Objective-
C
Chrome,
Safari,
Firefox,
Edge,
IE/iOS/Andr
oid
Mocha JS, Jest,
other super
set on top of
Selenium
(Protractor,
WebDriverIO,
etc.)
Download
relevant
driver, set up a
grid, network
and location
impacts
execution
speed
Plenty of
integrations
(CI, CD,
reporting,
visual testing,
cloud vendors)
End-to-
end,
security,
unit,
Robust community,
multiple bindings, best
practices
Coding skills required
(SDET Oriented)
Perfecto fully supports
Selenium and its
WebDriver
configurations. Local
execution requires
setting up a Selenium
grid
XCUITest/Espresso/
Headless/Cypress
Objective
C/Java/Jav
a Script
Chrome,
Electron
NA Embedded
into IDEs,
headless
bundles a
browser in the
FW
CI/CD UI/Unit Good documentation
and code samples
Dev Oriented Built-in Chrome/Firefox
browsers in headless,
Perfecto Cloud supports
scaling
Espresso/XCUITest
Codeless Irrelevant,
based on
record and
playback
All Proprietary UI
with
underlying
Selenium
WebDriver
APIs
Mostly
SaaS/browser
plugin
installation
Limited Functional
/UI
Growing, limited No coding skills
required (Business
Tester Oriented)
Perfecto supports
codeless in the cloud
BDD Java, Ruby,
JS, Kotlin
All Junit,
Selenium,
Appium
Open source,
Maven/Gradle
/TestNG
Plenty + APIs
(e.g. Rest
Assured)
Functional Robust community,
docs, adoption
Step-definition
development in code is
required/scenarios are
no-code (Mix of
Business Tester and
SDETs)
Perfecto Quantum is a
web/mobile BDD
framework

23. perfecto.io
Scaling Generates Large Test Data – Important Strategy Item

24. perfecto.io24 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Scaling Best Practices
What to Test
(Scenarios + Platforms)?
Why These (Value)? When in the Cycle,
How Often (CI/Outside)?

25. perfecto.io25 | Perfecto by Perforce © 2020 Perforce Software, Inc.
• Think continuous value.
• Think maintenance and complexity to create, execute, monitor, and maintain (ROI).
• Test reliability over time.
Test Automation Certification
If { (Test Scenario) == Candidate to add Value add
{ and x– It detect defects}
{ and – It passes across multiple platforms with consistent result}
{ and – Its execution time is < 5 minutes }
Then
{ Include in Test Suite & & CI }
Else {
}
Run such certification
once in every 2-3
software iterations

26. perfecto.io26 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Measuring Test Automation ROI Value
Value of test
automation
•Cycle testing time per iteration
•Test automation types (Functional, Non-Functional, Unit)
•Error detection efficiencies
•Product coverage (Key business Scenarios)
Cross-Team
Collaboration
•3 personas sharing load (resource, smart utilization of platforms)
•Manual testing %
•Backlog of test automation between iterations (in-sprint automation)
•Test debugging and analysis time per cycle
Test
Execution
•Automation within CI success rate
•Execution across platforms success rate
•Test redundancy/duplication/waste
•Coverage across environments (test, production)

27. LIVE DEMO

28. perfecto.io28 | Perfecto by Perforce © 2020 Perforce Software, Inc.
• Testing with Selenium (Java) the Boston Globe RWD site at scale
• Smart analytics with CI dashboard and cross-platform comparison
• Technologies Used:
• IntelliJ Idea
• Selenium
• TestNG/Maven
• Perfecto
Live Demo

29. Perfecto’s Four Key Pillars for Continuous Testing

30. perfecto.io30 | Perfecto by Perforce © 2020 Perforce Software, Inc.
The 4 Key Pillars of Continuous Testing in DevOps
This is where money is spent (Time, Tools, Resources)
This is where value is being
realized, quality is improved
Unified solution serves DevOps with greater effectiveness.

31. Thank You!
Visit perfecto.io or follow us @perfectomobile