A case study of the 2015 Grand National Horse Race in England. Perfecto Mobile and Intechnica describe the results of their monitoring 34 million mobile bets during the 2015 Grand National and share best practices for app performance.

1. Grand National 2015
Mobile application performance insights
How to get monitoring right with millions of users

2. Perfecto Mobile
The world leader in mobile application continuous
quality helping the world’s biggest brands to deliver
better apps faster
“Perfecto Mobile currently has the
strongest 3rd party position in the
market” [ Thomas Murphy]
Intechnica
• Vendor/Technology Independent
• Enable Performance by Design
• Provide Performance Management Services
• Specialists in Agile Performance
• Enable and deliver Performance Best Practice

3. Your presenters:
Larry Haig
Senior Performance Consultant, Intechnica
larry.haig@intechnica.co.uk
+44 (0)845 054 2381
Contributing author:
The Art of Application Performance Testing
[Molyneaux, 2nd Ed 2014 Pub O’Reilly]
Amir Rozenberg
Director, Product Management, Perfecto Mobile
amirr@perfectomobile.com

4. Agenda
Grand National: Fast Facts
Our Findings
The Next Race
Best Practices
Demonstration
v
v
v
v
v

5. Grand National 2015 – fast facts
• OpenBet record 19,300,000 bets within 24 hours
– Peak 45,000 bets pm
– 54,000,000 account transactions [UK population c63m (2011)]
• + 24% y-o-y
– Average bet size is £8.00
– 63% of remote bets via mobile,
– [up from 54% 2014]
• UK Transaction volume +12.6% y-o-y
– (+83.6% vs typical April trading day)
• UK Transaction value + 20.3% y-o-y Refs: OpenBet; Worldpay

6. Survey & caveats
• Focus on Mobile perspectives, end user/client side issues
– Android and iOS
– Complementary testing:
• Mobile emulation / public carrier (4G) / webkit browser (m. sites)
• Real device testing (native apps)
• Raceday & 2 day buildup
• High vs low traffic
• Performance and Quality of Service
• Caveats:
– Snapshot testing – indicative, not absolute / continuous
– High level monitoring only
• No device metric correlation
– Real device test redundancy limited
• Native App availability not quoted
Example – race day errors log

7. Summary of key findings
• Overall performance improving y-o-y, BUT
• Still evidence of traffic related stress
• All m. sites (and some native apps) showed some evidence of Quality of Service
issues
– 85%+ likely to be revenue relevant
– Brand impact
• m.sites:
– iPad2 Tablet slower than Smartphone
– 3rd party content source of failure
– Lack of traffic resilience
– Transaction payload highly correlated with performance

8. Summary – traffic & payload:

9. Summary – traffic & payload:

10. NATIVE MOBILE APPLICATIONS

11. Native App Response Times by Device- The Good
4.23s 2.59s 2.91s 8.06s
5.16s 9.32s 5.67s 1.48s
Web Emulation = 1.15s

12. Poll
v
Impact of 5 second delay in a revenue transaction
• I would call customer support
• I would complain on social networks
• I would transition to competitor application
• I would try again

13. Poll Results
v
Impact of 5 second delay in a revenue transaction
I would call customer
support
4%
I would complain on
social networks
5%
I would transition to a
competitor application
56%
I would try again
35%

14. Native mobile applications – maximized revenue?
Mobile Advertising on app
launch takes user attention
Application behavior in absence of network-
opportunity to optimize experience

15. Native applications Panel performance – ‘Time to Bet’
Panel # Android iOS
10 5.0
7 8.9
8 5.1 3.4
9 5.9 1.5
13 10.2 5.0
< 5 sec
>5 - < 10 sec
> 10 sec
Raceday median (seconds)

16. Performance consistency (raceday):
0600hrs 11 April
Transactionresponse(seconds)

17. Example root cause analysis
0
50000
100000
150000
200000
250000
300000
350000
400000
0
10
20
30
40
50
60
70
80
90
100
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55
CPU User (%)#Device CPU Total (%)#Device
Bytes In - WLAN#Device

18. Rapid find & fix - root cause analysis - drilldown
• Network wait: Client side processing
• Large JavaScript D/L: Repeated downloads
• Significant advertising/tracking/3rd party activity

19. Brand impact [race day errors]:

20. PLANNING FOR THE NEXT PEAK EVENT

21. Digital Experience Unhappy PathNew Use Cases
What will the next race look like?

22. Quality of Experience- Adopt the user “glass”
Server Architecture Delivery Chain Devices & User
3rd party
Device Emulation UX Timer App-level
CPU Peak
iPhone 6 0.9s 1.15s 67%
iPhone 4S 0.9s 8.9s 85%
Typical Application Launch Time
Recommendation 1

23. Daily Weekly
Sensors
(NFC, BT)
Events
Incoming
(Call/SMS/Alert)
Location
Rotation
WiFi
On/Off
Airplane
mode
Dynamic Backend
Switching
Light/Heavy
Load
Device Stress
Restricted
Permissions
Environments
Network
Conditions
Sensor
allocation
UI Assets
OS & App-
level Vitals
Optimizatio
n Directions
Reporting
Rich Media QoE
KPIs
Network
Traffic
Continuous Quality Lab
Performance TestingAdvanced Functional Testing
Recommendation 2
Test, Optimize & Get Ready

24. Recommendation 3
Alerting Drives Early
Awareness (MTTK) &
correction (MTTR)Real Devices
Coverage
Geos/Scenarios/Carriers
Remain in control:
Know early, resolve early, and keep
winning!

25. Key suggestions for monitoring
1. Understand customer devices, locations, trends
2. Monitor appropriately and manage all digital delivery channels
– PC, m.
– Native mobile applications
– Other (in store kiosks, etc.)
3. Lean is fast – manage page weights in m. sites
4. Audit & optimise client side/application components
5. Explicitly monitor object rendering
6. Understand device specific effects
7. Manage 3rd party inclusions
– SLAs, monitoring
8. Plan ahead
– Understand traffic load effects ahead of time
– Baseline key metrics

26. www.intechnica.co.uk
www.perfectomobile.com
Thank you.