We can all learn valuable lessons from game development where, in addition to functional performance, overall experiential quality—user experience (UX)—is of critical importance. Blockbuster game development presents particular challenges with regard to scale, rapid iteration, and fuzzy requirements. Learn from Tulay McNally and Alex Lucas how BioWare QA participates in development from concept through release, employs key methodologies like session-based and agile testing, and provides a path for Video Game Testing as a career. Additionally, discover how Tulay and Alex take quality engineering beyond test automation by eliminating broken builds, enhancing tester capacity and accuracy, employing machine learning, and developing industry-leading telemetry and data visualization solutions. Learn how to meet these challenges with an embedded model―one that partners QA with developers―and an aggressive QA technology roadmap. Take back new ideas and approaches for meeting consumer and customer demand for higher interactivity and deeper levels of engagement.

1. GAJAH ANNUAL REPORT 2015 | ‹#›
Testing Blockbuster
Games
Tulay McNally,
Alex Lucas

2. Challenge #1
Game platforms like the Microsoft Xbox and Sony
Playstation are inherently difficult to develop for.
Challenge #2
In-house engines, custom renderers, and 1st party
platforms mean that we can’t use off-the-shelf tools. We
must build our own (where possible).
Challenge #3
Nebulous requirements and frequent system overhauls.
Achieving ‘fun’ is an iterative process.
Challenge #4
Games are massive. Build times are long, binaries are
huge, incredible amounts of graphics, data, and sound
working (and needing testing) in concert.
Challenge #5
Game quality is a function of the interoperability of many
complex systems. The individual parts may ‘work’ while
the whole does not.
Technical
challenges in
game quality
assurance.
Games feature a level of scope and interconnectedness that
is not typically seen in other kinds of software. Our
customers do not merely use our software; they project
themselves into our software as participants in a simulacrum
of a real world.
In order to convey a convincing and fully realized world, our
developers create a multitude of systems and features that
incorporate truly massive amounts of supporting data. The end
result is a piece of software that is nearly impossible to verify
completely; a challenge further complicated by the need to also
assure its experiential quality.

3. Many of the technical hurdles of supporting QA on a title like
Dragon Age, while challenging, have been overcome with an
ambitious investment in QA technology.
Recognizing that a key contributor to overall game quality is
iteration opportunity, we target our QA technology to support
increasing developer velocity and unblocking testers so they can
provide timely and meaningful feedback.
BioWare QA Technology has developed sophisticated tools and
telemetry platforms that are in use all across Electronic Arts.
We’ve applied automation technology to solve traditionally non-
QA problems, improved the content and development pipelines,
and partnered with higher education to apply scientific research in
addressing our QA needs.
Game Quality
comes from iteration
& experimentation.
QA
TECHNOLOGY
@BIOWARE
Pipeline Improvements
Increase developer velocity by enhancing build and
content pipelines. Discover defects as close to creation as
possible, as well as repairing them as they occur.
Data Visualization & Telemetry
Increase knowledge about the product at all stages of
development. Things we historically used automation for
are often better achieved via development telemetry.
Enhanced Tools
A broad range of bespoke tools, scripts, state editors and
supporting technologies increase tester capacity and
accuracy. Turn average testers into ‘rockstars’.
Machine Learning
Leverage the capabilities of our engineers and
partnerships to deliver context aware technology. Notable
achievements on DA:I include duplicate bug detection
and callstack analysis.

4. GAJAH ANNUAL REPORT 2015 | ‹#›
Dragon Age: Inquisition
QA
TECHNOLOGY
EXAMPLES

5. Using FBScript to Support Testing
ON DRAGON AGE INQUISITION
FBScript is an integrated Python-based scripting
language that allows the QA Technology team to
rapidly locate and enumerate assets within the
game.
Allows QA to identify ALL the assets in the game,
including new and deprecated assets, where they
exist, and any dependencies that may inject risk into
the product.
For 200 hours of script development time, 4800 staff
hours of verification were saved/discovered.
81% of the scripts provided verification of systems
that where not scheduled for testing due to
feasibility concerns.
Data-Driven
TESTING
Significant ROI
Games have massive amounts of data. Asset counts on Dragon Age run into the millions. Models, shaders, scripts, textures, animation,
etc. are all referenced frequently across the code base, sometimes with 48-level-deep-object references, and this creates a unique level
of challenge for QA. The number of unique permutations is astronomical and so, much of the ‘game’ cannot feasibly be verified.
Additionally, this content is largely undocumented (with cause) and so much of this risk is also unknown. For example, how many
unique ‘rock’ models are there in the game? How many are being used? Where are they being used? Are they being used correctly?
Using data-driven testing via FBScript allowed us to answer these questions. QA was able to form a picture of the game ‘as it was’ and
not ‘as it was intended to be.’
QA, in addition to being able to verify whole swathes of previously untestable content, was able to pinpoint ares of particular risk and
adjust testing strategy accordingly.
1
10
100
1000
Net Gain/Loss (hours)
Total Dev Time (hours)

6. Pre-Flight Build System
DRAGON AGE SERIES
Builds are typically an expensive process in games,
and with the large number of developers on a title,
along with many product SKUs to support, builds can
break frequently.
CIS catches broken builds quickly, but developers are
still blocked while waiting on a fix, and QA is not
testing latest code.
By submitting to pre-flight instead of source-control,
bad code is bounced, good code is checked-in, and
CIS produces ‘always working’ and ‘latest code’ builds.
NO BROKEN
BUILDS
Case Study: Pre-Flight Build System on Dragon Age Origins
• Automated testing of change-lists across multiple SKUs and configurations.
• Does not replace CIS. Works with it by preventing bad code from entering source control in the first place.
• Number of monthly broken builds decreased from ~300 at peak to less than 30. Remaining build breaks where due to developers
circumventing the pre-flight system.
• Supported significant increase in developer velocity by ensuring developers and QA always had a working, up to the minute build.
Remember, game quality comes, in part, from iteration opportunity.
0
100
200
300
400
MAR '06 AUG '07
Pre-Flight
Solution
Deployed
E3
Demo

7. HEAT MAPS
• Aggregate telemetry to pinpoint trends and adjust the
design of the game accordingly
• Identify performance problems; frame-rate, memory
usage, latency
• Support QA in making coverage evaluations
SESSION PLAYBACK
• All gameplay session events are recorded via
integrated telemetry system
• Events, bugs, crashes are all visualized both
spatially and temporally
• Supports GPS-style playback to aid design
balancing and crash reproduction
• QA is better able to target testing and communicate
concerns
DASHBOARDS
• Effective delivery of key data for the project:
• Integrated and presented alongside our game
telemetry
• Surfaced throughout the studio on TVs in common
areas
• Branded to our projects
• High visual fidelity and ‘coolness’ factor increased use
REPORTS
• Customized implementation of Kibana & Elastic Search
• Kibana supports rapid report creation and the flat
database gives us fast access
• Multiple customers:
• Design use for game balance
• Programming use for performance
• Production & QA use for project health
DATA
VISUALIZATION
AND TELEMETRY

8. ADVANCED
BUG FILING
CUSTOM
BUG TOOLS
• Custom interface that sits on top of
existing industry standard tools like JIRA
and DevTrack:
• Unique interface depending on
project role: QA vs. Dev
• Details are filled out procedurally
• Hot-links to content in editor for rapid
fixes
• Advanced duplicate issue detection via
machine learning partnership with local
university.
INTEGRATED WITH TELEMETRY
VISUALIZATION
• All bugs are aligned with session telemetry
• Active bugs can be visualized via a web-
based 2D map (featured) or in-game
• Aids in identifying problem areas, untested
areas and rapid regression of ‘claimed
fixed’
• Crashes are filed automatically,
aggregated with ‘like’ and, due to captured
session data, quickly resolved by
programmers.

9. The games BioWare is developing for the future will be larger and more complex
than anything we’ve ever made before. Procedural content generation is
becoming common place in the industry, and if games were hard to test before,
it’s only going to get more difficult.
To meet this challenge BioWare will continue to invest in QA Technology as a
means to help deliver quality experiences for our players. As the scope of our
games increase, we must invent new ways to assure quality in both our
processes and the complex systems that define our games, while at the same
time becoming better at informing development on the ‘fun’ of our product.
Player Modelling
Formalize models of our players to better
understand their preferred play-styles
Run player models against ‘beat reports’
that track the rhythm of game activities, to
ensure high-levels of continuous
engagement
MVB: Minimum Valuable Bug
Building an automated bug pipeline by adding
procedurally generated repro-steps, indexed
video aligned with telemetry, and additional
machine learning
Create effective, resolvable bugs for
development AND reduce tester burden when
filing them
Heuristic Automation
Procedural content generation is driving need
for procedural quality evaluation
Automatically apply value models for ‘fun’ or
‘interest’ against content
Reduce the amount of content that needs to
have eyes on it during formative development
stages
LOOKING FWD