Presented by Sveta Kostinsky (Director, Field Engineering)

1. Selenium Today vs. Selenium Tomorrow:
Digital as the Convergence of Mobile & Web
Programs
Sveta Kostinsky, Director, Field Engineering

2. Overview
• Mobile changes test
planning
• Describe CI challenges
with mobile
• Describe a working
pattern
• Share some examples
Objectives
• Share best practices for
extending CI to mobile
projects
• Unleash the power of
Appium

3. Velocity – Without Compromising on Quality
2010
4 per year
2015
36 per year
Typical app releases per year
IDC
Proprietary & Confidential 3
Code Test Deploy Production
Defect Reporting
Defect < Incidents
±4-6 months delivery cycle
“Traditional Development
Practices Will Fail for Mobile
Apps”
[ Gartner April 2014 ]
±4 weeks delivery cycle
The Desired Mobile SDLC with Continuous Quality

4. 90 Minute Commute
• Check Facebook
• Skip that OS update
• Watch YouTube videos
• Clear easy emails
• Start expense reports
• Answer the phone
• Finish expense reports
• Order a gift on online
• Send several text
messages
• Accept LinkedIn invite

5. “Poor app quality is the primary reason behind the weak adoption, usage
and even failure of many mobile apps.” [ Gartner, Sep 2014 ]
5
Using the Citi Bank App
1 2
Working within a
network: Cellular, Wifi
3
Interacting with other
apps
4
- Specific Device
- Specific profile
5
Using your location
6
Interacting with other
Devices
7
Working with sensors
Peak time -
Millions of transactions
8

6. Easily Applied for Browsers …
Mobile Create Challenges …
UI
Integration
Unit Testing

7. UI
Integration
Unit Testing
Performance Security
Mobile Requires Shifting the Plan …
Target
Devices
Target
Env.OS
OS versions
Form factors
CPU & Memory
Manufacturers
Wearables
IOT
Networks
Orientation
Location
Notifications
Events
Device events

8. Manage the Time Budget …
Build
Frequency
Test
Duration
Test Scope
Device
Coverage
Unit System Smoke Regression Performance
Hourly 20-40 Min ±4
3-4 times a
day
30-60 Min ±6
Nightly 2-7 Hours ±10
Weekend 10-48 Hours ±20

9. Tuning the Process for Mobile
Proprietary & Confidential

10. Perfecto Mobile - Selenium Web Test Execution Flow
Execution Process
1. A Jenkins build job initiates and downloads the source code from GitHub
2. GitHub provides the source code and Maven downloads all the build dependencies
3. The Java code compiles and begins executing according to the Maven build objectives
4. The build job begins executing the TestNG tests
5. TestNG tests concurrently execute and call the Selenium libraries
6. The Selenium tests execute and begin to execute on Selenium Grid via Remote Web Driver, running on real devices
7. The Selenium tests complete and return the results back to Jenkins
1 2 3
7 6
5
4
Digital
LAB

11. Customer Challenges executing CI
1. Set up
2. Stability
3. Test scenario limitations
4. Reporting
11Proprietary & Confidential

12. CI Blockers: Adding Real DevicesWhat
• Struggling to
add real
devices to the
CI process
Why
• Uncertain
device
availability
• Test readiness
and
configuration
control of real
devices
How
• Managed lab
• Devices SLA

13. Device Lab that …
1. SLA assured devices availability
2. Parallel execution capability
3. Manage by attributes capability

14. Best Practice: Select Devices By Attribute
Select
targets by
attribute

15. CI Blockers: Inefficient Test DevelopmentWhat
• Developing UI
automation
takes too long
• Automation
requires
constant
maintenance
Why
• Scripting
per
platform
• Skillset -
shifting
from
scripting to
coding
How
• Easy object
identification
• Develop
framework
• Reuse predefined
code functions
• Write cross
platform test
code

16. Best Practice: Collaborate for Test Friendly Code
Catalog objects
Object Properties
Inherit Properties
Perform Actions

17. Best Practice: Cross Platform Test Code
Cross platform
execution

18. CI Blockers: Excessive Execution FailuresWhat
• > 25%
execution
failures and
errors
Why
• Unexpected
notifications
• Environment
readiness
• Device
errors
How
• Separate test
code &
execution mgt
• Perform pre-test
readiness
checks
• Add Error /Event
handling

19. Best Practice: Graceful Test Recovery
System level control
Application control
Device control
Manage Mobile
Events to
Reduce False
Positives

20. CI Blockers: Poor Execution PerformanceWhat
• UI Automated
test execution
exceeds
available test
window Why
• All the tests wait
for the device
(resources issue )
• Lack of execution
management
• Hardcoded
“Waits”
How
• Apply
automation
best practices
• Parallel
execution
• Use sync
mechanism

21. Error handling Remove CI/Automation Blockers
NativeDriver.manage().timeouts().implicitlyWait(15, TimeUnit.SECONDS);
NativeDriver.manage().timeouts().pageLoadTimeout(15, TimeUnit.SECONDS);
Implement
dynamic waits &
Parallel
Execution

22. Our Working Cookbook
Automation
Apply Real
User
Conditions
Manual
Where Must
Early
Non-Functional
Shared Lab
Test
Framework
Align QA tools to Dev
Continuous Integration

23. CI Recommendations
1. Share tools
2. Maintain test code guidelines
3. POM maintenance
4. Error & event handling
5. Readiness checks
6. Externalize Resource mgt
7. Optimize execution
8. Multi-threaded execution

24. Unleash the power of Appium
Automation
Real User Env Capabilities
Enterprise Grade LAB

25. Unleash the power of Appium
Real User Env Capabilities
Enterprise Grade Service
System
Level
Control
Parallel
& Cross
Platform
OCR & Image
Object
Recognition
Rich
ReportsAutomation

26. Unleash the power of Appium
Real User Env Capabilities
Enterprise Grade Service
System
Level
Control
Parallel
& Cross
Platform
OCR & Image
Object
Recognition
Rich
ReportsAutomation

27. The power of visual analysis
Add Visual
analysis to your
test

28. Unleash the power of Appium
Automation
Enterprise Grade Service
Real user
environment
GPS
Injection
Network
Virtualization
Single User
Performance
Hybrid Cloud
Global
Coverage

29. Should I apply more packet loss?
Run the test with
variation of
network
conditions

30. iOS9 - Proactive Assistance - Siri Gets Smarter
• Enhanced voice commands
• Contextual notifications &
reminders
• Browse through Apps via voice
• New test cases
• springboard control
• Voice injection support
required for testing
Description Implications

31. Perfecto Mobile for Appium
Open
Source
iOS &
Android
Native
Object
Recognition
CI Support
Automation
Real user
environment
Enterprise
Grade
System
Level
Control
Parallel
& Cross
Platform
OCR & Image
Object
Recognition
Rich
Reports
GPS
Injection
Network
Virtualization
Single User
Performance
Hybrid Cloud
Global
Coverage
Enterprise
Security
24X7
Support
Scale &
Flexible
Roles &
Permissions

32. Zero Code Change

33. Sveta Kostinsky
Dir, Field engineering
svetak@perfectomobile.com