Watch a live presentation at http://offer.bitbar.com/measure-and-optimize-graphics-performance-of-android-games Good graphics performance is closely related to fantastic user experience. That’s why we’ve integrated the unique GameBench profiling tool into Testdroid Cloud. This is a powerful resource for any developer who wants to spot performance bottlenecks and optimize their games across an array of Android devices with different silicon, screen sizes, memory, and many other variances in software and hardware. Stay tuned and join our upcoming webinars at http://bitbar.com/testing/webinars/

1. 11 August 2015
Ville-Veikko Helppi
Technical Product Manager
ville-veikko.helppi@bitbar.com
How to Reliably Measure and
Optimize Graphics
Performance of Your Android
Games
W E B I N A R
Sharif Sakr
Director of Business Dev.
sharif@gamebench.net

2. Agenda
• Testdroid Statistics from Q2'2015
• Testing Mobile Games – Why and How?
• Optimize Graphics Performance with
Help of Gamebench - Introduction
• Demonstration
• Q&A

3. TESTDROID STATISTICS

4. Q2'2015 – Test Run Stats

5. Why Mobile Games Fail?

6. iOS 8.0
iOS 8.0.2
iOS 8.1
iOS 8.1.1
iOS 8.2
iOS 8 ALL
KitKat 4.4
KitKat 4.4.2
KitKat 4.4.3
KitKat 4.4.4
KitKat ALL
Lollipop 5.0
Lollipop 5.0.1
Lollipop 5.0.2
Lollipop ALL
Android 'M'
0 5 10 15 20 25 30 35
Failed test runs. Percentage (%).

7. Some Observations
1
2

8. Some Observations
3
4

9. Some Observations
5
6

10. Android Fragmentation 2015
by Open Signal report August 2015
Testdroid Cloud
>95%

11. Android Fragmentation 2015
by Open Signal report August 2015
Testdroid Cloud
+
Gamebench
>85%

12. How Many Devices is Enough?
~90%
market coverage can
be achieved with
128
devices
~20%
market coverage can
be achieved with
14
devices
US Market
30 Android devices
= ~2/3 market
Global Market
80 Android devices
= ~1/2 market

13. TESTING MOBILE GAMES

14. Correct behaviour across
platforms and browsers
Integration with web back-
ends
Typically need to
fully utilize HW
(CPU+GPU)
Resource (e.g.
battery)
consumption
OpenGL ES 2/3
Performance!
Functionality and
usability
Screen
orientations,
connectivity, user
profiles
Robustness
Robustness and security!
Brand
Compliances, verification
with back-ends and data
Different 'Verticals'

15. Manual Testing is not Agile

16. Test Automation is Agile

17. OS
versions
Chipsets
CPU +
GPU
Tens of
OEMs
Memory
Displays
(resolutions,
physical hw)
OEM
mods
Other
hardware
(connectivity
calibration)
Relation
to other
software
Where Test Automation Can Help

18. Why Real Devices Are Must-to-Have
• Emulators cannot help you to test...
• User Experience
• Usability
• Hardware
• Software
• Infrastructure
0 % = the percentage of your app users
that use emulator to run your app!

19. Why Testing Games Matter?
• Graphics performance
capability is the most
important thing for mobile
games when forming UX
• Too many games fail to work
properly with low-end/mid-
range devices due hardware
performance
• The real understanding of
game's performance
capability can be only figured
out by testing on real devices Understanding
How quickly gamer assimilates what the
game is about, how to fully play with it, what
are the functionalities.
Usability
The main User Interface, menus, design and
how usable those are
Playing Experience
Intuition to follow the game and go forward
with its instructions and game-play
Monetization
Are gamers willing to buy – invest – in game

20. Types of Game Testing
• Functionality of the game-play, featuresFunctional Testing
• Done always when new features /regressions
are includedRegression Testing
• How game runs on different configurationsCompatibility Testing
• Different languages, geo-focused materialsLocalization Testing
• Endurance test to determine if system can
handle the loadSoak Testing
• Measures the capacity of the systemStress Testing
• Simplest form of performance testing,
measures how system handles certain loadsLoad Testing
• Isolation of the environment (e.g. from
network) to see how game worksHermetic Testing
Feature
-based
testing
Performance
testing
End-user
testing

21. Gameflow - Performance
• Compared to regular app testing, mobile game testing
may seem to be more challenging
• e.g. OpenGL ES content not action-identifiable
• Graphics and UI – Identical look&feel across all
devices and variants
• Usability controls in the context of game
• Navigation/game-play, Progress, Delays between
"game-stages"
• Animations, Fonts, Other graphics assets
• Screen Resolution and Orientation

22. • Studies: Majority of users leave game
due sluggish, general slowness or
other bad performance behavior
• It's very important to understand how
game behaves on different devices,
under different circumstances
Performance Makes +/- UX

23. Performance Testing
• Performance, Load, Stress, Spike, Soak
• Measure the real
user experience
• Execute tests with
all possible devices
(SW&HW)
• Discover the impact
of server loads, app
performance
(CPU/GPU
utilization)

24. Components in Game Testing
App Middleware Platform Connectivity/Network Back-end
GAME
Game
Engine
Android
iOS
...
API
&
services
Server
Database

25. Starts at Design Level
App Middleware Platform Connectivity/Network Back-end
GAME
Game
Engine
Android
iOS
...
API
&
services
Server
Database
Game Design,
GFX Assets,
Files, Optimization

26. Performance Testing w/HW
App Middleware Platform Connectivity/Network Back-end
GAME
Game
Engine
Android
iOS
...
API
&
services
Server
Database
Real Devices and Platform

27. Compatibility Testing
App Middleware Platform Connectivity/Network Back-end
GAME
Game
Engine
Android
iOS
...
API
&
services
Server
Database
Variations

28. Reference Testing
App Middleware Platform Connectivity/Network Back-end
GAME
Game
Engine
Android
iOS
...
API
&
services
Server
Database

29. Back-End Performance
App Middleware Platform Connectivity/Network Back-end
GAME
Game
Engine
Android
iOS
...
API
&
services
Server
Database
Game with Back-End
Integration / Google
Play Services

30. Game Testing Process

31. Trends in Mobile Game Testing
• Image recognition
15 seconds

32. GAMEBENCH

33. A traffic light system for mobile apps and devices

34. 50%

35. People notice performance

39. Performance = promotion

40. OEM and
Chip
makers
Developers – Games, Apps, VR
Journalists
Testing /
QA
40k+ downloads in less
than 1 yr since launch,
>250 users any day
Market traction

41. App and Game
Developers
Chip makers
OEM
Consumers
Journalists
and
Enthusiasts
Used to certify
performance
before release
Used by retailers to match
consumers’ usage profiles
to mobile devices
Helps with device
and app performance
reviews

42. Copyright GameBench Limited
Q&A

43. Who are the people behind GameBench
•Sri Kannan Iyer: CEO & Founder
(London)
•Over 15 years experience in the mobile
•ARM, MediaTek, Analog Devices and Intel
•MSEE (UT Austin), MBA (Oxford)
•4 patents, 1 book chapter, speaker at
several conferences
•Karthik Hariharakrishnan: CTO &
Founder (Bristol)
•Over 10 years in mobile engineering
•ARM, Motorola, National Instruments
•MSEE (University of Illinois), 3 patents
Six employees
based in
Bristol &
London
Get in touch!

44. sharif@gamebench.net

45. Demonstration
Gamebench with Real Devices from Testdroid
Cloud

47. THANK YOU!
More information about frameworks, devices
and testing at www.testdroid.com