Selenium Today vs. Selenium Tomorrow: Digital as the Convergence of Mobile & Web Programs
1. Selenium Today vs. Selenium Tomorrow:
Digital as the Convergence of Mobile & Web
Programs
Sveta Kostinsky, Director, Field Engineering
2. Overview
• Mobile changes test
planning
• Describe CI challenges
with mobile
• Describe a working
pattern
• Share some examples
Objectives
• Share best practices for
extending CI to mobile
projects
• Unleash the power of
Appium
3. Velocity – Without Compromising on Quality
2010
4 per year
2015
36 per year
Typical app releases per year
IDC
Proprietary & Confidential 3
Code Test Deploy Production
Defect Reporting
Defect < Incidents
±4-6 months delivery cycle
“Traditional Development
Practices Will Fail for Mobile
Apps”
[ Gartner April 2014 ]
±4 weeks delivery cycle
The Desired Mobile SDLC with Continuous Quality
4. 90 Minute Commute
• Check Facebook
• Skip that OS update
• Watch YouTube videos
• Clear easy emails
• Start expense reports
• Answer the phone
• Finish expense reports
• Order a gift on online
• Send several text
messages
• Accept LinkedIn invite
5. “Poor app quality is the primary reason behind the weak adoption, usage
and even failure of many mobile apps.” [ Gartner, Sep 2014 ]
5
Using the Citi Bank App
1 2
Working within a
network: Cellular, Wifi
3
Interacting with other
apps
4
- Specific Device
- Specific profile
5
Using your location
6
Interacting with other
Devices
7
Working with sensors
Peak time -
Millions of transactions
8
6. Easily Applied for Browsers …
Mobile Create Challenges …
UI
Integration
Unit Testing
7. UI
Integration
Unit Testing
Performance Security
Mobile Requires Shifting the Plan …
Target
Devices
Target
Env.OS
OS versions
Form factors
CPU & Memory
Manufacturers
Wearables
IOT
Networks
Orientation
Location
Notifications
Events
Device events
8. Manage the Time Budget …
Build
Frequency
Test
Duration
Test Scope
Device
Coverage
Unit System Smoke Regression Performance
Hourly 20-40 Min ±4
3-4 times a
day
30-60 Min ±6
Nightly 2-7 Hours ±10
Weekend 10-48 Hours ±20
10. Perfecto Mobile - Selenium Web Test Execution Flow
Execution Process
1. A Jenkins build job initiates and downloads the source code from GitHub
2. GitHub provides the source code and Maven downloads all the build dependencies
3. The Java code compiles and begins executing according to the Maven build objectives
4. The build job begins executing the TestNG tests
5. TestNG tests concurrently execute and call the Selenium libraries
6. The Selenium tests execute and begin to execute on Selenium Grid via Remote Web Driver, running on real devices
7. The Selenium tests complete and return the results back to Jenkins
1 2 3
7 6
5
4
Digital
LAB
11. Customer Challenges executing CI
1. Set up
2. Stability
3. Test scenario limitations
4. Reporting
11Proprietary & Confidential
12. CI Blockers: Adding Real DevicesWhat
• Struggling to
add real
devices to the
CI process
Why
• Uncertain
device
availability
• Test readiness
and
configuration
control of real
devices
How
• Managed lab
• Devices SLA
13. Device Lab that …
1. SLA assured devices availability
2. Parallel execution capability
3. Manage by attributes capability
18. CI Blockers: Excessive Execution FailuresWhat
• > 25%
execution
failures and
errors
Why
• Unexpected
notifications
• Environment
readiness
• Device
errors
How
• Separate test
code &
execution mgt
• Perform pre-test
readiness
checks
• Add Error /Event
handling
19. Best Practice: Graceful Test Recovery
System level control
Application control
Device control
Manage Mobile
Events to
Reduce False
Positives
20. CI Blockers: Poor Execution PerformanceWhat
• UI Automated
test execution
exceeds
available test
window Why
• All the tests wait
for the device
(resources issue )
• Lack of execution
management
• Hardcoded
“Waits”
How
• Apply
automation
best practices
• Parallel
execution
• Use sync
mechanism
22. Our Working Cookbook
Automation
Apply Real
User
Conditions
Manual
Where Must
Early
Non-Functional
Shared Lab
Test
Framework
Align QA tools to Dev
Continuous Integration
24. Unleash the power of Appium
Automation
Real User Env Capabilities
Enterprise Grade LAB
25. Unleash the power of Appium
Real User Env Capabilities
Enterprise Grade Service
System
Level
Control
Parallel
& Cross
Platform
OCR & Image
Object
Recognition
Rich
ReportsAutomation
26. Unleash the power of Appium
Real User Env Capabilities
Enterprise Grade Service
System
Level
Control
Parallel
& Cross
Platform
OCR & Image
Object
Recognition
Rich
ReportsAutomation
27. The power of visual analysis
Add Visual
analysis to your
test
28. Unleash the power of Appium
Automation
Enterprise Grade Service
Real user
environment
GPS
Injection
Network
Virtualization
Single User
Performance
Hybrid Cloud
Global
Coverage
29. Should I apply more packet loss?
Run the test with
variation of
network
conditions
30. iOS9 - Proactive Assistance - Siri Gets Smarter
• Enhanced voice commands
• Contextual notifications &
reminders
• Browse through Apps via voice
• New test cases
• springboard control
• Voice injection support
required for testing
Description Implications
31. Perfecto Mobile for Appium
Open
Source
iOS &
Android
Native
Object
Recognition
CI Support
Automation
Real user
environment
Enterprise
Grade
System
Level
Control
Parallel
& Cross
Platform
OCR & Image
Object
Recognition
Rich
Reports
GPS
Injection
Network
Virtualization
Single User
Performance
Hybrid Cloud
Global
Coverage
Enterprise
Security
24X7
Support
Scale &
Flexible
Roles &
Permissions