Gaming is a multibillion-dollar industry, and good testing is critical to any game’s success. Game testing has traditionally been black-box through the client—a method clearly insufficient with increasingly more complex software incorporating 3D physics, thousands of linked and interacting assets, large databases, and client-server architecture. Automation is an obvious answer, but how do you automate when the user interface is an immersive virtual environment, the data is as vital a part of the software as the code (and actually more likely to create bugs), and the games themselves are often built specifically to prevent automation? Brett Roark describes how Blizzard Entertainment is meeting this challenge by using automation to tame complex asset pipelines and building custom tools that make each tester more efficient. Take away a deeper understanding of the unique complexities of modern game testing, see why they require fresh and creative solutions, and consider how these solutions might apply to non-game testing.

1. T6
Concurrent Class
10/3/2013 9:45:00 AM
"Test Automation Challenges in
the Gaming Industry"
Presented by:
Brett Roark
Blizzard Entertainment
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Brett Roark
Blizzard Entertainment
A gaming industry professional with more than fifteen years as a tester, developer, and
producer, Brett Roark has shipped multiple AAA titles as well as numerous casual games.
Before joining Blizzard Entertainment in 2010, Brett had spent more than a decade at Microsoft
as a software design engineer in test. While working on Windows 7, he also taught game design
and production at DigiPen.

3. 9/6/2013
Test Automation Challenges
In The Gaming Industry
©2013 Blizzard Entertainment, Inc. All rights reserved.
1

4. 9/6/2013
Brett Roark
• Senior Producer, Test
Automation
• 15 years game industry
experience
– Tester
– Developer
– Producer
Agenda
• Introduction
• Why testing games is
challenging
• Which test functions
should be automated
• How Blizzard built an
automation test team
2

5. 9/6/2013
Blizzard Product Lines
3

6. 9/6/2013
Core Values
Blizzard “polish” doesn’t just
refer to our gameplay
experiences, but to every
aspect of our jobs. Approach
each task carefully and
seriously. Seek honest feedback
from your peers and use it to
improve the quality of your
work. At the end of the day, no
one will remember if the game
was late – only that it was great.
4

7. 9/6/2013
Why Testing Games is
Challenging
Software Complexity
•
•
•
•
Large development teams
Long maintenance periods
Shared technologies
Computer science problems
– Distributed computing
– 32/64 bit conversions
– Complex math for physics and 3D
graphics
– Artificial intelligence
5

8. 9/6/2013
Software Complexity
•
•
•
•
3D environments
Audio and cinematics
Localization
Allowing user interface
customization
Multiplayer Interactions
• Adds complexity similar to multithreaded vs. single-threaded
programming
• Consistency of experience
• Latency sensitivity
6

9. 9/6/2013
Data
• Games often data heavy
• Designers solve problems by using
game systems in unexpected ways
• Data source control difficulties
Which Test Functions
Should Be Automated?
7

10. 9/6/2013
Benefits of Automation
• Increase coverage to
mitigate risk
• Improve process
efficiency
• Find bugs earlier
Content that’s Difficult to
Automate
•
•
•
•
•
•
Usability
Graphic style
Lore and story
Game balance
Difficulty
FUN!
8

11. 9/6/2013
Areas to Consider
•
•
•
•
Clients
Servers
Data
Tools
Client Automation
• Challenges
– Frequent user interface changes
– 3D environments
– Automating around anti-cheat code
• Useful targets
– Client performance
– Compatibility suites
– Regression for expansions
9

12. 9/6/2013
Server Testing
• Functionality and APIs
• Error injection
• Stress testing
– Performance under stress
– User experience under stress
– Fault tolerance and recovery
Data Validation
•
•
•
•
Content pipelines
Data diffs
Delivery and patching
Graphics systems
– Shaders
– Lighting
– Hardware driver updates
10

13. 9/6/2013
Tools
• Help make QA and/or development
processes more efficient
• Examples:
– Quest tracker
– Arena teams
– In-game bug reporting
How Blizzard Built an
Automation Team
11

14. 9/6/2013
Blizzard Structure
• Multiple development teams
• Shared platform and technologies
through Battle.net
• QA is a separate division that
supports all development teams
Automation Team Structure
Each automation team consists of:
– An engineering lead
– A producer
– Automation engineers
12

15. 9/6/2013
Building Momentum
• Hired talented engineers who are
a cultural fit
• Started with Battle.net
• Expanded to products in early
development
• Built tools to increase efficiency
Talent Development
• Internal training team
• Robust onboarding program
• Multi-tiered training programs
– Leadership
– Project Management
– Testing
• Professional certifications
– ISTQB Foundation and Advanced Levels
– Scrum Master/Product Owner
13

16. 9/6/2013
Summary
•
•
•
•
•
Games have become incredibly complex
Some content is difficult to automate
Identify key areas for automation
Build momentum with initial successes
Invest in your people
Questions?
14