Best Practices in Mobile Game Testing

Learn the Best
Practices in Mobile
Game Testing
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved. 1
2 December 2014
Ville-Veikko Helppi
Technical Product Manager
ville-veikko.helppi@bitbar.com
W E B I N A R

• Different Types of Testing and How Game
Testing Differs from Regular App Testing?
• Frameworks, Tools and Technologies – What is
Important in Mobile Game Testing Today?
• Top Priorities for Mobile Game Testing
• Infrastructure for Successful Game Testing
• Hands-on Example (Image Recognition)
• Testdroid Update
• Q&A
Agenda

Let's Define 'Testing'...

Different Types of Testing?
• Black-box testing vs. White-box testing
• Methods used in Mobile Game Testing
– Functional
– Compatibility
– Performance
– Localization
– Regression
– Load
– Smoke
– Acceptance

Defining Some Ways to Test
Functional Testing • Functionality of the game-play, features
• Done always when new features
/regressions are included Regression Testing
Compatibility Testing • How game runs on different configurations
Localization Testing • Different languages, geo-focused materials
• Endurance test to determine if system can
handle the load Soak Testing
Stress Testing • Measures the capacity of the system
• Simplest form of performance testing,
measures how system handles certain loads Load Testing
• Isolation of the environment (e.g. from
network) to see how game works Hermetic Testing
Feature
-based
testing
End-user
testing
Performance
testing

The Old-Fashion Thinking
Deterministic systems
Identifiable objects
Tool support(s)
User-driven actions
Simple inputs/outputs
Virtualization
Random setups/results
No handle on objects
No standards
User reacts to input
Complex inputs/outputs
Real hardware
<>
<>
<>
<>
<>
<>
High-level of automation Low-level of automation

Test Automation as a Solution
Random setups/results -> Easy to generate random events
No handle on objects -> Variety of options (e.g. image
recognition)
No standards -> Variety of frameworks (cross-platform
etc.)
User reacts to input -> Test scripts can handle random
events
Complex inputs/outputs -> Test scripts can generate input/content
Test Automation brings real value
Real hardware -> Availability of real mobile hardware
with variety of options to mobile game testing

Challenges that Test Automation Tackles
• Proliferation of Devices, Brands, Operators,
Platforms
• Coupling the Testing and Development process
(Continuous Integration/Deployment/Delivery)
• Different Types of Testing (as automated)
– Stress, Load, Performance, Functionality etc.
• Test script & test asset usability
• Cons of Manual testing
– Tedious effort, Expensive human resources, Error-prone
(as done by humans) & Not reusable (done once)

Significant Benefits of Using Automation
OEM
mods
Relation
to other
software
OS
versions
Chipsets
CPU + GPU
Tens of
OEMs
Memory
Displays
(resolutions,
physical
hw)
Other
hardwar
e
(connectivity,
calibration)

Different 'Verticals' Require Different Practices
Typically need to
fully utilize HW
(CPU+GPU)
Resource (e.g.
battery)
consumption
OpenGL ES 2/3
Correct behaviour
across platforms and
browsers
Integration with web
back-ends
Functionality
and usability
Screen
orientations,
connectivity,
user profiles
Robustness
Robustness and
security!
Brand
Compliances,
verification with back-ends
and data

Importance of
Mobile Game Testing

Important Aspects in Game Testing
• User Interface and Functionality
– UI layouts/elements, Menus, Resolution,
Orientation, Overall visuality
• Graphics Performance
• Usability and User Experience
– Responsiveness, Real Occuring Events
• Multi-player/User
• Social Integrations
• Security and Liabilities

User Interface and Functional
• Compared to regular app testing, mobile game
testing may seem to be more challenging
– e.g. OpenGL ES content not action-identifiable
• Graphics and UI – Identical look&feel across all
devices and variants
• Usability controls in the context of game
• Navigation/game-play, Progress, Delays
between "game-stages"
• Animations, Fonts, Other graphics assets
• Screen Resolution and Orientation

Performance Testing
• Performance, Load, Stress, Spike, Soak
• Measure the real
user experience
• Execute tests with
all possible devices
(SW&HW)
• Discover the impact
of server loads, app
performance
(CPU/GPU
utilization)

Performance Makes +/- UX
• Studies: Majority of
users leave game due
sluggish, general
slowness or other bad
performance behavior
• It's very important to
understand how game
behaves on different
devices, under
different circumstances

Building the User Experience
Monetization
Are gamers willing to buy – invest – in game
Playing Experience
Intuition to follow the game and go forward
with its instructions and game-play
Usability
The main User Interface, menus, design and
how usable those are
Understanding
How quickly gamer assimilates what the
game is about, how to fully play with it, what
are the functionalities.
• Graphics performance
capability is the most
important thing for mobile
games when forming UX
• Too many games fail to
work properly with low-end/
mid-range devices due
hardware performance
• The real understanding of
game's performance
capability can be only
figured out by testing on
real devices

• End-to-end between tests, AI or
more simple implementations
• Optimize the data communication
between server and clients
• Test simultaneously on several
(different) devices
• Check what happens if game gets
interrupted (platform)
• Location-based features – how to
properly mock up those?
• Hermetic environment – what
happens?
Multi-Player/User Features

• Important part of the after-engagement
(how gamers
share their results)
• Many of OEM branded
devices provide certain
SoMe apps pre-installed
(version?)
• Connectivity, login and
data transferring are
tested in context of game
• Hermetic environment –
what happens?
Social Integrations

Security and Liabilities

4 Top Priorities in
Mobile Game Testing

Localization

Device Diversity
By OpenSignal, Inc. 2014 - http://opensignal.com/reports/2014/android-fragmentation/

Real Devices vs. Emulators
Why Real Devices are Must-to-Have?
• Emulators/simulators won’t help you to test…
– User Experience and Usability
– Hardware
– Software
– Infrastructure
0 % = the percentage of your app users
that use emulator to run your app!

Manual vs. Automation
Manual Automation
Smaller coverage, More
money burnt & time
wasted, Error-prone
Large coverage,
quickly
completed, Less
money & time
wasted, Exact
results.

Infrastructure for
Mobile Game Testing

High-Level Infrastructure

Architecture / Infrastructure
GAME
Game
Engine
Android
iOS
...
API
&
services
Server
Database
App Middleware Platform Connectivity/Network Back-end

Isolated, Hermetic Testing
GAME
Game
Engine
Android
iOS
...
API
&
services
Server
Database
Hermetic Environment

Compatibility & Porting Testing
GAME
Game
Engine
Android
iOS
...
API
&
services
Server
Database
Real Devices

Smoke/Reference Testing
GAME
Game
Engine
Android
iOS
...
API
&
services
Server
Database
Reference Environment

Back-End Performance Testing
Massive no. of users
and significant load
GAME
Game
Engine
Android
iOS
...
API
&
services
Server
Database

Continuous Integration

Game Testing Process

Hands-on Example:
Integrating Development with
Testing Environment

Setup
• Continuous Integration (e.g. Jenkins)
• Appium framework (for Android and iOS)
• No need for devices – we'll use devices on
Testdroid Cloud service
• Test scripts / test cases for game

Continuous Integration

Family Tree: Android Test
Automation Frameworks
Appium works for both, Android and iOS
Appium Intro

• Uses Selenium Webdriver (W3C standard) as a
2014
38
scripting framework
• Supports native Android, native iOS and mobile
web:
– Android via uiautomator (API level >=16) and
Selendroid (API level <16)
– iOS via UI Automation
– Mobile web as Selenium driver for Android and iOS
• You can write your Appium scripts on almost any
programming language
(Haskell/Go/Clojure/Java/Ruby)
Appium Basics

Desired Capabilities / Android
All rights reserved.
39
Test Script
Desired
Capabilities
Test Case
{
“device”: “Android”,
“app”: “/Users/user/ApiDemos.apk”
“app-package”: “com.example.android.apis”
“app-activity”: “.ApiDemos”
}
Example

40
Test Script
Desired
Capabilities
Test Case
{
“app”: “com.bitbar.testdroid.BitbarIOSSample”
}
Example
Desired Capabilities / iOS

Appium @ Localhost
Example
41
Test Script
WebDriver
Desired
Capabilities
Test Case
http://localhost_4723/wd/hub
4723
Appium Server
Device

From Localhost to Testdroid
Cloud
*Testdroid Caps
Test Script
WebDriver
Desired
Capabilities
Test Case
http://localhost_4723/wd/hub
http://appium.testdroid.com/wd/hub
Example

From Localhost to Testdroid
Cloud
WebDriver {
Test Script
Desired
Capabilities
Test Case
“testdroid_username”: “user@domain.com”,
“testdroid_password”: “p4s$w0rd”,
“testdroid_project”: “My First Project”,
“testdroid_testrun”: “Test 1”,
“testdroid_device”: “iPad Mini 7.0.4 A1432”,
“testdroid_app”: “http://domain.com/app_v1.ipa”
.
.
“app”: “com.bitbar.testdroid.BitbarIOSSample”
}
Example

Example
44

350+ Android & 20+ iOS Devices

Example
46

Example
47

Example
48

Example
49

Example
50

Example
51

Projects and Test Runs
Example

Behind the Scene
Appium
Broker
5 Appium Ready
sessionid
53
Test
Script
Device Cluster
Device
1
Device
2
Device
3
1 WebDriver Session Request
@ http://appium.testdroid.com/wd/hub/
Desired Caps, .apk / .ipa
2 Configure project
3
Wait for device to
become available
4 Start Appium
Session
Map
6 WebDriver Session response
sessionid
7 Test Run
sessionid Proxy
Appium
Device
1
Example

Image Recognition
The basic example of Image Recognition
in Mobile Game Testing: Template
Matching

Code Example

Example: Clash of Clans
• Test Script contains the following 'steps':
1. Identify the used platform (Android / iOS)
2. Search for 'Goldmine' graphical element
(using find_image to search for screen)
3. Script runs test to go to shop
4. Script runs test to buy a cannon
5. Script runs test to place the cannon (all
these three are defined in .png files)
6. Script runs test to start a battle!
7. Game is brought down and test ends.

Image Recognition with Real Android and
iOS Devices in Local Environment
VIDEO

Testdroid Update

New UI in Testdroid Cloud 2.0

Testdroid Cloud's Interactive

Testdroid
PrivateCloud -
Running user-selected
at any of our
hosting facilities
devices

NEW HOSTING CENTER!
Mountain View, CA
San Francisco
and
Mountain View
datacenter

Testdroid Deployment Options
Option 1:
Public Device Cloud
on-demand devices
(multitenant)
Mobile app testing on
over 400+ real
Android and iOS
devices hosted by
Bitbar
Option 2:
Private Device Cloud
reserved devices
Hosted by Bitbar in the
US and/or Europe
Devices chosen by and
reserved only for the
Customer
Option 3:
On-premise Device
Cloud
Automated mobile
app testing on real
Android and iOS de-vices
hosted by the
customer, usually
30-500 devices

THANK YOU!
W E B I N A R
Video recording & slides of this webinar will
be soon available at Vimeo and Slideshare!
Please visit at www.testdroid.com for more
information.

We operate the largest global device
cloud with over 400 devices
instantly available for developers
Devices from the US, Europe, China, Japan and Korea
– global devices for global customers

Best Practices in Mobile Game Testing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Best Practices in Mobile Game Testing

Similar to Best Practices in Mobile Game Testing (20)

More from Bitbar

More from Bitbar (15)

Recently uploaded

Recently uploaded (20)

Best Practices in Mobile Game Testing