A presentation delivered in the Test Automation and Digital QA Summit 2018 held in Hong Kong. A shift of paradigm for software testing as a result of new technologies in mobile computing, cloud computing, big data, and machine learning is observed in recent years. The presentation discusses how understanding on mobile app performance can be improved, and subsequently diagnosed, and monitored using the new paradigm.

1. TESTINGMIND
www.testingmind.com
MANAGING MOBILE APP PERFORMANCE THROUGH
GEO-PERFORMANCE VISIBILITY
DR DANIEL CHAN
#TAS18
Hong Kong
November 7th , 2018

2. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
CONTENTS
• Performance as a critical success
factor for mobile apps
• Common practice in mobile app
performance assessment
• Importance of geo-performance
• Distributed performance
assessment platform with
centralised orchestration
• Issue identification & classification
through machine learning
• Enabling agile mobile evolution
2

3. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
PERFORMANCE AS A CRITICAL
SUCCESS FACTOR FOR MOBILE APPS
• Great mobile experiences require great performance
Engaging mobile experiences are half user interface, half performance
• Performance must be a focus throughout the development
cycle
Mobile consumers often do not allow developers multiple chances
• Complexity hurts performance
Simplicity of architecture is mandatory to deliver great mobile performance
• Offline support drives amazing performance
Engineering client-side caching solutions is challenging but rewarding
3

4. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
PERFORMANCE IS NOT A PRIORITY FOR
MOBILE DEVELOPMENT TEAMS
4
Source: Forrester’s Forrsights Developer Survey, Q1, 2013 based on North American and European
mobile and mobile web app developers

5. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
Emulators
• SDK or emulator softwares
used to develop and test
• Does not reproduce device
conditions
Tethers
• Run automated scripts on
devices connected to laptop
• Not scalable and a very
localized solution
Device Cloud
• Scale device testing for
distributed teams
• Does not reproduce
real world conditions
Experience Platform
• Real devices, real user
conditions, real cellular
• Endpoint performance
monitoring
TEST APPROACH EVOLUTION

6. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
CLOUD DEVICE FARMS
6

7. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
VOICE OF DEVELOPERS
7
We have zero visibility into the
network characteristics of our
app
NETWORK
Real localization testing is
very challenging
LOCALIZATION
Our app content resides
almost completely in
the cloud
CLOUD
Weak code is increasing
data plan costs and
network issues are
draining the battery
CODE QUALITY
We have bandwidth
issues in large parts
of the world
GEO-PERFORMANCE
Over 50% of our poor ratings in
the app store are due to
network issues
USER RATINGS

8. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
THE CHALLENGES
8
• Device diversity
• Operating system diversity
• Carrier networks
• Apps on the device
• App versions
• Background processes
• App clients
• Client-server communication
• Geo-performance visibility

9. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong 9
The permutations
of the possible
iOS versions and
device types
that users will
access an app is
fairly high.
IOS
FRAGMENTATION

10. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
ANDROID
FRAGMENTATION
How many mobile
devices are running
Android?
An OpenSignal
survey from 2015
puts the number of
devices at 24,093
under 1,294 brands.
How many to cover
in your app testing is
a really a question
about device
penetration in your
target markets.
10

11. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
REAL
DEVICES
To simulate real user
geo-performance
exeperiences
• Real devices,
particularly newly
released devices
• Real SIM cards
• Geographically
distributed in
different cities &
countries
11

12. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
REMOTE
CONTROL
To orchestrate the
remote devices
centrally from a
console
• Provides access to
devices connected
to mobile networks
around the world
right from the web
browser
• Allows app testing in
real world conditions
- pre and post
release
12

13. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
TEST AUTOMATION
13
Automation Frameworks

14. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
PERFORMANCE DIAGNOSTICS
14
To detect &
categorize issues
• Captures user
experience issues
across the CDN,
servers, app,
device or carriers
• Identifies, isolates,
and prioritizes
performance
fixes

15. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
CASE STUDY: TELSTRA CONDUCTING DRIVE
TESTS TO UNDERSTAND NETWORK
PERFORMANCE ON THE MOVE
15
Using mobile physical boxes allow Telstra to experience their network in a familiar real-
world use case — while driving. Accurate geo locations, correlated video & audio
feeds, and video & MOS quality scores allow Telstra to accurately identify areas in its
network that offer poor user experience.

16. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
CONTINUOUS
MONITORING
To monitor mobile app health
◦ Monitors critical user experience
KPIs
◦ Automatically provides alerts to
irregularities - not just at the time
of an incident but beforehand
◦ Assesses speech quality of
voice enabled app
◦ Integrates with bug tracking
services (e.g. JIRA, ServiceNow)
16

17. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
CASE STUDY: FACEBOOK ASSESSING
PERFORMANCE IMPACTS OF RELEASES
17
Slows down Post Image times
Slows down Search times
Version 139.0.0.26.93 141.0.0.31.91 143.0.0.32.90142.0.0.29.92
Slows down Search times Slows down Search times
Speeds up Post Image times
Speeds up Video Playback Time
Speeds up Search times

18. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
CASE STUDY: TELSTRA NETWORK
PERFORMANCE MONITORING
18
Telstra continuously monitors real-world network performance through
collecting KPI data for 25 popular apps every 10 minutes

19. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
ENABLING AGILE MOBILE EVOLUTION
• Fast feedback is a natural complement to continuous delivery
Continuous delivery & feedback from performance management tools &
analytics framework define the new app development & delivery cycle
• Focus on collecting 3 types pf mobile measures
Focus on metrics that measure your app’s technical fitness, level of user
engagement, and business results
• The faster you can act on feedback, the better
Feedback has a half-life: its value degrades with time, so the faster you
can act on it, the better
In the long run you should collect real-time user feedback on local devices
and use it to predict the next best action your app should take
19

20. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
REAL-TIME TECHNICAL & ENGAGEMENT
METRICS
20
Audience exposed
To ads of sponsors
Track
Converted
Audience
Mobile
Purchase
Measure
conversion
attributed
to exposed
audience

21. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
AWS ARCHITECTURE
21
Availability Zone
region
Administration VPC
EC2 instancesVPN
gateway
Security
group
Security
group
date center n
Ubuntu 16.04 MacOS 10.12
date center 1
Ubuntu 16.04 MacOS 10.12
……
VPN
connection
Route 53
user
Dedicated Customer VPC
VPN
gateway
VPN
connection
VPC subnet
VPC subnet
Security
group
Unified
Controller
EC2 instances RDB
VPN
connection
VPN
connection

22. TESTINGMIND CONSULTING Test Automation and Digital QA Summit | Hong Kong
Thank You
22