1.
101 – How To Deliver Winning Mobile Apps
Perfecto Mobile
&
Toronto Association of System and Software

2.
Eran Kinsbruner is a director & mobile technical evangelist at
PerfectoMobile, one of the leading mobile cloud and automation companies.
Formerly CTO for mobile testing and Texas Instruments project manager at
Matrix, Eran has been in testing since 1999 with experience that includes
managing teams at Qulicke & Soffa, Sun Microsystems, General Electric, and
NeuStar. The co-inventor of a test exclusion automated mechanism for
mobile J2ME testing at Sun Microsystems, Eran has experience in the mobile
testing world. You can find Eran on Facebook, Twitter @ek121268, LinkedIn,
and his professional mobile testing blog at ek121268.wordpress.com.

3.
What To Do?
Great News – Your
first Mobile Project
has arrived!
Now What?

4.
Presentation Objectives
• Mobile Market State
• Mobile QA/Dev Objectives
• Sustaining a Mobile QA Plan Based on CQL
• Functional Testing
• Non Functional Testing
• Regression Testing/CI
• Pillars Of A Continuous Quality Lab
• Mobile QA Best Practices

5.
Mobile Market State
Digital experience drives
innovation
• IOT
• Complex use cases
Fragmentation becomes
more painful
• Devices
• OS’s
Tool mix as a key for agile
adoption
• Commercial, Open Source,
Proprietary

6.
Mobile QA & Dev Objectives
Source: OpenXcell
Quality
Velocity
Increase Quality
Increase Velocity
Mobile QA & Dev
Objectives

7.
QA & Dev Testers Vision (aspiration)
Increase mobile app
velocity without
compromising on quality
Keep up with mobile
market changes and
meet end user
requirements
Develop & Test
against the real end
user profiles
Leverage unattended
automation as part of
any nightly build
through CI workflow
Receive actionable
mobile app quality
insights per build

8.
RBC – QA Objectives
v
Improved Quality Optimized Cost Faster Time to Market
Operational
Effectiveness
Operational
Efficiency
Business
Value
• Increase Test Coverage
• Test Effectiveness – reduce test
duplication between QA & business
• Reduce defect leakage to production
• Reuse test automation during UAT
test cycles
• Decrease QA test cycles – via test
automation
• Near zero downtime/
production failures
• Applications perform as expected
• Test automation cost savings
• Manual and automation test
management savings 10% using BDD
Approach
• Onsite to offshore ratio 40:60
• QA Technical Solution Lead across all
projects
• Leverage Perfecto Mobile cloud solution
to save on devices and data plans
• Reduction in Cost of Testing due to
productivity improvements-
Maintenance and Projects
• Reduce regression test cycle by 5-10
days per project
• Perfecto Mobile UFT
• Continuous improvement in test
design & execution
• Maximize Test Automation coverage
• Increase re-use - BDD
• Enable business to respond to
market change
• Proactive feedback on market
changes to existing production
mobile applications

9.
SUSTAINING A MOBILE QA PLAN

10.
Sufficient Coverage
• Choose the right
devices and OS
platforms for your
project
• Get Perfecto
Mobile’s Q1 2015
Mobile Test
Coverage Index

11.
Q1-2015 Mobile Test Coverage Index (Preview)

12.
What’s Coming? Why It’s Important?

13.
Manual & Interoperability
• Starting point for Pre-Automation phase
(foundation)
• Candidates to keep manual:
– Incoming events
• Calls
• Text Messages
• Notification pop ups
– Complex use cases to automate

14.
Functional Automation – That Works
• Reusable, Repeatable,
Predictable Tests
– Unattended
– Continuous
– Easy to maintain
• Object Support On Any
Mobile App Style
– Dom, Visual, Native
– Instrumentation (Yes/No)
– System Level Control
• Flexible Frameworks
100% accurate native
object rec.
Much faster than OCR
Support all languages
Does not often detect
GUI defects
Imperative for end
user’s experience
Imperative for GUI
glitches
Implement the
WYSIWYG
Slower then object
level analysis
Dependent on 3rd party
Does not support all
languages
Object level analysis
Visual analysis (OCR)
May require
Instrumentation

15.
Levels of Test Automation
Level 1
Level 2
Level 3
Level 4
Level 5
Navigation flows
Content validation
Error Validation
Data validation
(if feasible)
UI validation
(if feasible)
Check automation feasibility
Identify scenarios for basic
navigation & successful function flow
Navigation flows are automated
Check automation feasibility of
screen content
Level 2 Scenarios are ready
Check automation feasibility of
error validation
Level 3 scenarios are ready
Check automation feasibility of
data validation
Level 4 scenarios are ready
Check automation feasibility of UI
validation and other complex features
Automated scenarios of basic
Navigation % happy path
are ready
Content validation scripts are ready
and plugged in with exiting automated
navigation BPT component
Error validation components are
ready and plugged in with
automated components of Level 2
Data validation scripts are written
and plugged in with existing
automated components of Level 3
Automated scripts for UI screen &
other complex scenarios are ready
and plugged with existing scripts
DataDrivenAutomationFramework

16.
Non Functional Testing
• Cover key business transactions
response time & general mobile
app availability
• Early actionable performance
feedback & insights
• Assures high end user experience
in various conditions (aka: Wind Tunnel)
– Various network profiles
– Various locations
– Various load conditions
– Incoming events
– Per app insights

17.
Mobile App Monitoring
• The only Efficient Way to Monitor Mobile Apps
– Browsers and Emulators - irrelevant for mobile
monitoring
• Capture Your True Mobile End User Experience
– Monitor mobile engagement scenarios, including
interoperability on real devices on real networks
• Early Warnings and Fast Correction
– Real-time alerting triggered when transaction
availability or response times are compromised
• Extend your existing quality assets for monitoring
– Automation test code
– Performance test code

18.
What Makes A True CQL
Quality Lab

19.
What Makes A Complete Continuous Quality Lab
Real Device Lab (Device as a Service)
Functional Manual Testing
Functional Automation
Performance Testing
Real environments & Profiles
Production Monitoring
Hybrid Cloud
(Local, Public, Private)
Operations, Security & Support
3rd Party API’s & Integrations

20.
Mobile QA Best Practices

21.
Mobile Continuous Integration Recommended Work Flow
Dev Team
Source Control
Code Commit
CI Server
ReportingUnit Tests
Launch Perfecto Job
(Release Package)
Support today:
PM Native & Selenium
Automated App Deployment
On Real Devices
Functional Testing
Integration Testing
Performance Testing
Continuous Feedback
Test Planning & Test Execution
Bug Reporting

22.
Build Your Mobile App Test Plan Properly
Functional
• Validation of Functionality
• Smoke / Regressions Testing
• Offline access testing
• Negative Testing
Non Functional
• Network Strength / Outage /
Recovery
• Different NW Types
• Peripheral Testing
• Services
Interoperability (IOP)
• Voice / SMS interrupts
• Notifications
• Battery /Cable Removal
Memory Leak
• Memory Usage
• Memory Leaks
• Garbage Collection
Installation Testing
• New App Install
• Uninstall and Reinstall
• Upgrade testing
Language Testing
• Validation for Locales (French &
English)
• Images and Text
• Currencies, time zones etc.
• Context
Performance Testing
• CPU Usage testing
• Network Usage
• Page Render time or activity
Render time
Security Testing
• OWASP Vulnerabilities
• Sensitive data (App/Device)
• Dynamic Testing
• Static Code Analysis
• Data Encryption
Usability Testing
• User Experience
• Competitive Analysis
• Expert Review

23.
RBC Mobile Quality Lab
Team Size –
45 (Onshore/Offshore Mix)
>100 devices - Blackberry, Android &
iOS platforms
Using Selenium
& UFT

24.
Continuous Automation Regression Model
• Executing automated test scripts 24*7
(consider offshore team)
– Private Perfecto Cloud – used for project
development
• Execute automated testing suite to reduce project
timelines
– Internal Perfecto Cloud –used for project and
maintenance testing on supported devices
– Perfecto Public Cloud – test non supported
devices (new market devices/ mimic customer
behaviour). Leverage Perfecto public metrics i.e.
what is currently in use by public (device and
configuration)
Hybrid Cloud

25.
Q&A