Slides From QA and Localisation Conference in London June 2013 Topics: Impact of Lean Startup strategies on QA approach - From minimum viable product (‘build the right thing’) to a fully supported service (‘build the thing right’) - Agile Methodology: traps and benefits Test case design methodology. What can we learn from BDD / TDD? Data Analytics supports Test scope Recruitment and Career progression for Games QA teams in comparison to Software QA Tester The Test Automation challenge: Experiences from a mature industry The speaker: Thomas Hulvershorn Quality Assurance and Analytics Manager Outplay Entertainment Biography With 11 years of experience in developing and testing non game related software such as mathematical insurance systems and e-commerce websites Thomas has also worked on the release of over 50 games on Mobile / PC and Facebook during the last 7 years. Having worked in different areas of QA he has a unique perspective of current trends in Software QA and the challenges in Games QA. In his career he has built and led QA teams in Eastern and Western Europe and the USA. He works for Outplay Entertainment in Dundee as QA and Analytics Manager. Being a pragmatist, his favourite quote is ‘A fool with a tool is still a fool’. {thanks to picol.org for providing some of the icons in this presentations}

1. The Mobile Challenge
Testing Mobile Games in a
Start-up Environment
Thomas Hulvershorn
Outplay Entertainment

3. Established 2011 in Dundee
Outplay Entertainment is a fast
growing, venture-backed Developer
and Publisher of social and mobile
Games.

4. >Technical Sophistication <
>Lean Production<
>High Impact Test Team <
What is special about Mobile Games?

5. Architecture (simplified)
Game
Core Code
C++, Flash…
PortingEngine
(Marmalade,Cocos2d,…)
iOS
Client
Android
Client
Facebook
Client
API
Services
Server
Code
Node.js
Data
base
MongoDB
Online
Connection
Wi-Fi, 3G

6. Test Stages
Games Testing (System Test)
Game
Core Code
C++, Flash…
PortingEngine
(Marmalade,Cocos2d,…)
iOS
Client
Android
Client
Facebook
Client
API
Services
Server
Code
Node.js
Data
base
MongoDB
Online
Connection
Wi-Fi, 3G
Porting Test
Compatibility Test
Compatibility Test
Compatibility Test
Reference Test
Server Test

7. Good practice from TDD / BDD approaches
Scenarios
“Given”
“When”
“Then”

8. Test Plan Advanced (TPA)
TPA

9. Acer Nexus 7
1280*800
iPad 2
1024 * 768
iPad 3
2048 * 1536
iPad 1
1024 * 768
Xperia Ray
854 * 480
Galaxy Gio
480 * 320
Nexus S
800 * 480
iPhone 3GS
480 * 320
iPod 4G
960 * 640
iPad Mini
1024 * 768
Galaxy Note
1280 * 800
HTC Desire HD
800 * 480
Razr Maxx
960 * 540
Galaxy SIII
1280 * 720
HTC One
1920 * 1080
iPod 5G
1136 * 940

10. Outplay Build and Device DB: “The Vault”

11. Analytics support for Balancing Tests

12. Lean Production ‘Validated Learning’Impact

13. Test Automation
UI Test
Game
Core Code
C++, Flash…
PortingEngine
(Marmalade,Cocos2d,…)
iOS
Client
Android
Client
Facebook
Client
API
Services
Server
Code
Node.js
Data
base
MongoDB
Online
Connection
Wi-Fi, 3G
API Tests
Unit Tests
Unit Tests

14. Significant investment
Maintenance costs
Different Skillset
Careful ROI analysis
Server Side, API and Client Code
Bespoke tools as enablers
Game UI Automation
Test Automation Snake Oil

15. API / Unit Test Framework

16. .2 million simulated Battles

17. Deterministic Systems <> Random setups / results
Identifiable Objects <> No Handle on Objects
Tool Support <> No Standards
User drives Actions <> User reacts to Game
Simple Input <> Complex timed Inputs
Virtualisation <> Hardware dependant
UI Automation for Games is hard by default
High Level of Automation Very little Automation

18. Keep it simple, modular and close to real User experience
Random setups / results: Set Random seed srand(0)
No Handle on Objects: Use image recognition
No Standards: try Sikuli, AutoHotkey…
User reacts to Game: Depends on type of game
Complex timed Inputs: Depends on type of game
Hardware dependant: PC build, Emulator

19. > Agile vs classic Milestones
> Picture of team
Outplay High Impact Team

20. Software Engineer in Test
Experience in Automation
Hard to find, Expensive
Career Path
Testing as Career
Enthusiast Gamer
Manual Tester
Easy to recruit, Entry Level
Temporary career step
> Recruit Tester with
basic coding Skills
> Keep your team motivated
through Training
> Evolve from Games Testing
to Application Testing

21. > Outplay.com
> Twitter ‘DerHUL’
> Further Reading / Viewing
‘The Lean Startup’, Eric Ries
‘How Google Tests Software’, James A. Whittaker et.al.
‘How we test Software at Microsoft’, Alan Page. et. al.
Google GTAC 2013, YouTube
Test-automation-zombie-apocalypse.pdf
www.ministryoftesting.com/, Info hub
cuke4ninja.com, good start into BDD and automation
alarcosj.esi.uclm.es/CombTestWeb/combinatorial.jsp
What next?