In this presentation we introduce the concept quality assurance in video games along with the most important concepts, team members and testing phases. These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.

1. Practical Lesson 1:
The testing process

2. What is a bug?

3. Quality is the absence of bugs.
Bug is a behavior not in someone else’s expectations.

4. Different Sorts of Bugs
• Quality is no bugs, but, what is a bug or defect?
A bug is caused by a mistake, it may imply an error (in values), a lack of accordance to
an specification/standard, and (it may) cause a system failure.
1. By classifying bugs into categories, we can understand their importance and learn
how to spot them.
2. By classifying bugs into types, we can understand where they could be originated.
3. By classifying bugs into severity levels, we can understand their priority in the
development process.
Make sure to investigate invisible walls very closely to avoid being NAB’d (when a
developer re-classifies your “bug” as “Not a Bug”). By the way, this also implies you
don’t know how to do your job!)

5. Per Olin classified bugs according to categories (Levy et al. 2009, Game Development
Essentials: Game QA & Testing; p. 77).

6. Visual: missing texture.
[https://steamcommunity.com/sharedfiles/filedetails/?id=135782335]

7. Physics: unrealistic gravity.
[http://www.smosh.com/smosh-pit/photos/10-funny-video-game-glitches-gifs]

8. Visual: clipping.
[http://www.smosh.com/smosh-pit/photos/10-funny-video-game-glitches-gifs]

9. Orthogonal Defect Classification (ODC) system developed by IBM allows to classify
defects into types in order to reveal how they were introduced, be found or avoided
(Schultz, 2011, Game Testing All in One; p. 42).
• A Function error is one that affects a game capability or how the user experiences
the game. The code providing this function is missing or incorrect in some or all
instances where it is required.
• A Assignment type error is the result of incorrectly setting or initializing a value used
by the program or when a required value assignment is missing.
• A Checking defect type occurs when the code fails to properly validate data before it
is used. For instance, the quantity of ‘ammo’ in a shooter.
• Timing defects have to do with the management of shared and real-time resources.
• Build/package/merge or, simply Build defects are the result of mistakes in using the
game source code library system, managing changes to game files, or identifying and
controlling which versions get built.
• Algorithm defects include efficiency or correctness problems that result from some
calculation or decision process.
• Documentation defects occur in the fixed data assets that go into the game. This
includes text, audio, and graphics file content.
• Interface defects occurs where information is being transferred or exchanged.

10. Bug severity levels (Per Olin)
It can also be tagged with a number on the range 1 to 4.

11. Normal bugs vs. Tough bugs (Per Olin)
Advices to deal with a ‘tough bug’:
• Once a tough bug appears, take a screenshot with the dev menu, debug data, etc.
• Immediately talk to the lead tester.
• Tell everybody to reproduce this (the exact steps).

12. The lifecycle of a Build
A testing process aims at finding bugs at every build (or version) of the game.
A basic game testing process in a QA/QC dept. consists of the following steps:
1. Plan and design the test.
2. Prepare for testing.
3. Perform the test.
4. Report the results.
5. Repair the bug.
6. Return to step 1 and re-test.

13. The lead tester, primary tester, or any other tester tasked with test creation should
draft these documents prior to the distribution of the build.
They tell the tester “go for X build version and Y devices and take test suite Z”.
Sometimes when a test suite is generalist a test case “does not apply”.
This is a test suite for minesweeper

14. Lead Tester receives the build…
Builds submitted to testing that don’t meet the basic entry criteria are likely to waste
the time of both testers and programmers.
The countdown to testing should stop until the test “launch” criteria are sufficiently
met. These are the criteria:
• The game code should be built without compiler errors. Any new compiler
warnings that occur are analyzed and discussed with the test team.
• The code release notes should be complete and provide the detail that testers
need to plan which tests to run or re-run for this build.
• Defect records for any bugs closed in the new release should be updated so they
can be used by testers to make decisions about how much to test in the new build.
• Tests and builds should be properly version controlled, as described in the
following sidebar (version control is for everyone, developers and testers).
• When you are sufficiently close to the end of the project, you also want to receive
the game on the media that it will ship on. Check that the media provided
contains all of the files that would be provided to your customer.

15. Configuration Preparation (config prep)
Before the test team can work with the new build, some housekeeping is in order.
The test equipment must be readied for a new round of testing.
• Wiping the “old” build in the computer. You need a clean computer. Save the old
files in a server or common space.
• The latest drivers for all components of the computer. This not only includes the
obvious video card and sound card drivers, but also chipset drivers, motherboard
drivers, ethernet card drivers, and so on.
• The latest versions of any “helper apps” or middleware the game requires to run.
These can range from Microsoft’s DirectX multimedia libraries to multiplayer
matchmaking software such as GameSpy Arcade.
The only other software on the computer should be the new build.

16. Ready, tester? Wait a second.
The next step after accepting a new build and preparing to test it is to certify
that the build is worthwhile to formally test.
• Smoke Testing. At a minimum, it should consist of a “load & launch,” that is,
the lead or primary tester should launch the game, enter each module from
the main menu, and spend a minute or two playing each module.
• So the build is distributed. Time to test for new bugs, right? Not just yet.
Before testing can take a step forward, you must take a step backward and
verify that the bugs the development team claims to have fixed in this build
are indeed fixed. They are in the “knockdown list”. This process is known as
Regression testing.
Sometimes critical bugs are not retired from the regression testing and the
regression testing may last for the entire development.

17. Reporting a bug
Possible audience of a bug: lead tester, project manager, marketing, third
parties, customer service,… obviously other testers.
In fact, the primary tester should be allowed to modify all fields in the bug
database except for Priority, Status, Assigned To, and Developer Comments.
How to write good reporting?
• Only facts. “The guard’s hat should be blue” or “the AI is weak”.
• Brief Description or title. The first sentence should contain a brief and clear
description of the bug:
“Crash to desktop when choosing Options from Main Menu”. YES
“Game crashed after I killed all the guards and doubled back through the level to
get all the pick-ups and killed the first re-spawned guard.” NO. too much detail.
“Game crashed after guards respawned”. YES

18. Advice: write the full description first and then the brief description (title).
• Full description. This provides all the necessary details. Rather than a prose
discussion of the defect, the full description should be written as a series of
brief instructions so that anyone can follow the steps and reproduce the bug.
The steps should be written in second person imperative, as though you were
telling someone what to do. The last step is a sentence (or two) describing the bad
result.
1. Launch game.
2. Choose Multiplayer.
3. Choose Skirmish.
4. Choose “Sorrowful Shoals” map.
5. Choose two players.
6. Start game.
Or much better:
1. Start a two-player skirmish game on “Sorrowful Shoals.”

19. EXAMPLE:
1. Create a multiplayer game.
2. Click Game Settings.
3. Using the mouse, click any map on the list. Remember the map you clicked on.
4. Press up or down directional keys on your keyboard.
5. Notice the highlight changes. Highlight any other map.
6. Click Back.
7. Click Start Game.
Expected Result: Game loads map you chose with the keyboard and mouse.
Actual Result: Game loads map you chose with the mouse.
Advice 2: Consider introducing the expected results as it may not be obvious for the
programmer.
It is also a good reminder of how test cases are produced.
Programmers should be educated in the testing process.

20. Other things we MUST include:
• Components versions. Build version, specific drivers versions, etcetera.
• Bug priority. The tester sets the possible priority (1-4 or minor, major to critical).
• Bug category. The tester should use the category “audio”, “artificial intelligence”,
“physics”, etc, so it narrows the scope
• Bug possible origin. The tester can include it in a line near the beginning, so it may
help the programmer in understanding the bug. The tester may say it comes from
“function”, “assignment”, wrong “build”, etc.
Things to avoid:
• Humor. A In a high-stress environment it may fuel the tempers.
• Jargon! Write in a clear language. Testers should avoid very technical terms
(source of confusion if not used properly).
EXERCISE: check in the Internet for a bug found by a player on a game already
released. Where is it reported? Is it reported properly according to all the fields and
recommendations?

21. Some Common Mistakes
· Non-summing summary line. "Found a bug." No kidding! Like all these other bugs
in the database aren't bugs that somebody found? That's like sending an email with
the subject line, "Hey" - or "I wrote an email" - or "From me." Sum up the problem!
Say what the problem is. Just like when you write an email, you give the recipient an
idea what the email is about before he or she opens it.
· Too-long summary line. "The slithy toves gyred and gimbled in the wabe when the
borogoves were all mimsy and the mome raths outgrabe." Dude, just say "The slithy
toves gyred and gimbled." That condenses the essence of the problem. You can give
us the details about all the excessively mimsy borogoves in the body of the report.
· Tester as designer. "The slithy toves need to have a 5-second cooldown after gyring
so they don't gimble too soon." No, don't tell the developer how to fix it. Just say
what the problem is. It's the designer's job to figure out what's the best way to
balance the slithy toves.
· Not giving step by step instructions. Tell the developer what to do, so he can see
the problem himself. Don't just say "I did this, then I did that." Tell him, "do this, then
do that." Give step-by-step instructions, in detail, in the second person (imperative).
[http://www.sloperama.com/advice/faq75.htm]

22. · Unclear basis for an expectation. "Usually, pressing X causes the door to open."
What do you mean, "usually"? Do you mean that's how one opens doors in other
games? Do you mean that's how one opens other doors in this game? Do you mean
doors in your home open when you press an X button? What does "usually" mean??
Be specific, dude!
· Confusing "player" with "player character". The player is the human who's holding
the game controller. That digital being on the screen is a character. Don't use the
terms interchangeably.
· Wishy-washy observation step. "Then watch to see if the problem happens or not."
Wrong. Tell the developer he will see the problem. Tell him to observe that
it does happen.
· Inappropriate use of the word "should" as in: "After you follow these steps, you
should see the bug happen." Um, what? The bug should not happen -- that's why it's
a bug! If it was supposed to happen, then it wouldn't be a bug. So you shouldn't say
"should" in this way. Just say "after following these steps, observe that the bug
happens."
[http://www.sloperama.com/advice/faq75.htm]

23. True or False:
• A “Verify Fix” status on a reported bug means it will remain in at least one more
test cycle.
• A tester should write as many steps as possible when reporting a bug to ensure
the bug can be reliably re-created.

24. Lesson 1: Why Quality Matters
Third year course in Quality Assurance and Game Balance
Bachelor Degree in Video Game Design and Production
Third term, April 2019 Dr. Marc Miquel Ribé

25. Overview of the Lesson
In this lesson we will see the following topics:
What is Quality in Games
1. Quality Matters
2. How Can We Improve Quality
3. Different Sorts of Testing
4. Approaches: Blackbox and Whitebox
5. Being a Black-box Tester.
QA and QM: Team, Phases and Documentation
1. TDD and Test plan
2. Testing phases
3. Quality management measurements

26. What is Quality in Games?

27. What is quality?
• Quality is ”how well we do something”
• Quality is “being reliable”
• Quality is “being special”
• Quality is ”being superior”
Quality may mean many things ‘in general’…
Some questions about quality ‘in particular’:
• What is quality in software? Does quality matter in software?
• What is quality in video games? Does quality matter in video games?
• How does user experience relate to video game quality?
• How do features relate to video game quality?

28. Think about a Space shuttle, think about Microsoft Word, think about a car.
• Is quality the same important for them?
In certain services or products, no quality means risking people’s lives or work.

29. Quality is “managing expectations”.
In video games, new mechanics and new experiences may matter more than quality…
But still, in a competing market, defects can ruin experiences.
We need to work for quality. Mainly, we need to test quality.
[https://www.gamesradar.com/top-7-horrendously-buggy-games-we-loved-anyway/]

30. Quality is “managing expectations”.
Quality is about working properly according to someone else’s expectations: game
console manufacturer, customer, etcetera.
Quality is the degree in which some characteristics are adapted to the requirements
put into a product or service (ISO 9000:2000), i.e. (in games: the mechanics, the visuals
requirements, etc.).
Quality is ‘what is required’: functionalities, expectations and the absence of bugs
(anomalies or non-defined behaviors).
“The general aim of testing is to affirm the quality of software systems by
systematically ex- ercising the software in carefully controlled circumstances” (1981).
E. F. Miller. Introduction to Software Testing Technology. Second edition.

31. Quality is the absence of bugs.
Bug is a behavior not in someone else’s expectations.

32. In the world of project management, quality, cost and time are key constraints that
shape any project. They are the “Iron triangle”.
Every project is a fight between:
1. what is required (quality)
2. the date for delivery
3. the agreed budget/cost
QUESTION: Which of the following puts pressure on a game project?
a) New funding b) Taking people away to have them work on another game c) A
requirement to add more levels to be comparable to a competitor’s recently
announced title d) b and c.
Why Do We Ship Buggy Games? - A Look Behind the Scenes - Extra Credits
[https://www.youtube.com/watch?v=s1_50T5GwZ8]

33. @TesterGame_com (QA agency for
Software, Games & Apps. Game
Testing Specialists) says:
“Sólo un 34.8% de las empresas
españolas considera el testing como
un perfil imprescindible en el
desarrollo de un videojuego
#libroblancoDEV #gamedev #indiedev
#indiegame #indiegamedev #qa”
These are results from the survey
“Libro Blanco de los Vídeojuegos”
from 2017.
[https://twitter.com/TesterGame_com/
status/951411246296903682]

34. 1. Quality Matters
ALERT:
§ Quality is a key service aspect. Quality is part of the management, as project
success depends on it. However, while quality is important, it is sometimes
considered “unoriginal” or the “last part”.
It is easy that games have bugs, but quality matters.
• Game software is complex.
• Software tools are used to produce games and these tools are not perfect.
• Games must work on multiple platforms/devices with a variety of configurations.
• Games can be played by a large number of people.
Online reputation depends on critics opinions and quality matters.
Quality must be planned and tested.
§ Coding and testing are different enough to justify the quality department.
As we get more and more professional, we can locate the quality responsibility into
one department.

35. • Trust is at the base of (video game) software development, and so every
stakeholder (product managers, press, etc.) asks for deliveries and expects each
others’ good performance. But… reality is not as beautiful as expected.
• The quality department does not know if something works until tested. They
cannot trust anyone.
• The quality department is given “no time” and “no resources” but still is asked
quality. Tension.
• How do you balance quality, cost and time?
In other words, how do you balance trust and testing? This is where QA comes in.

36. § Testing is the process of verifying that something works properly. Testing is
the most essential part required to ensure quality.
What do they always tell to the tester?
Trust no one.
Less trust implies more testing.

37. Tester: “Please, Trust No One”
A general form of statements to watch out for is “X happened, so (only/don’t) do Y.”
Here are some examples:
• “Only a few lines of code have changed, so don’t inspect any other lines.”
• “The new audio subsystem works the same as the old one, so you only need to
• run your old tests.”
• “We added foreign language strings for the dialogs, so just check a few of them in
one of the languages and the rest should be okay too.”
• “We only made small changes so don’t worry about testing <insert feature name
here>.”
• “You can just run one or two tests on this and let me know if it works.”
• “We’ve gotta get this out today so just ....”
WRONG! Quality control needs to test whatever considered relevant, sometimes
not listening to the recommendation and constraints coming from other
departments. This is the real meaning of “Trust no one”.

38. Tester: “Don’t stress: You are in a rational world”.
So, as a tester, you should follow these advice:
• Know your role on the team based on the responsibilities assigned to you.
• Execute your tasks aggressively and accurately.
• Do the most important tests first (priority).
• Do the tests most likely to find defects often.
• Make emotion-free and objective decisions to the best extent possible.
It is not always your fault as a tester if there are still some bugs after the final release
(they might be introduced late, they could be hidden because you spent time on other
bugs, etc.). Remember the ‘iron triangle’.
Conclusion: Testing is not enough to guarantee quality.

39. 3. Different Sorts of Testing
Remember: when they say quality, you may ask: ‘according to what or who?’.
I want you to structure the topic in questions, as it is easier to remember.
• Depending on the “Who” and the “What“ (requirements) with which they define
quality, we may find a different sort of testing.
• Functionality testing, Item testing (game design, producer)
• Compliance testing (platform)
• Localization testing (customer environment)
• Compatibility testing (components configurations)
• Depending on the rest of questions, other testing may appear:
• “How”: Blackbox and Whitebox testing.
• “When”: Smoke testing, regression testing, beta testing.
Let’s leave aside Usability testing and Playtesting, as they are User Experience field:
• Usability testing is aimed at finding aspects which block the ease of using an interface.
• Playtesting is aimed at understanding the user experience and game aspects (e.g. balance)

40. • Compliance
For ensuring that your game complies with the standards and guidelines of major first
party developers like Apple, Google, Amazon, Microsoft, Sony and Nintendo.
• Guidelines are known as TRC (Technical Requirements Checklist) because of Sony.
While Microsoft calls it TCR (Technical Certification Requirements), and Nintendo
calls it “Lot Check” since 1980 with Super Mario Bros.
Some usual compliance requirements:
• The game must not display any religious images.
• The game should pause once the controllers are not attached.
• Before formatting a memory card or a hard drive, a message warns the user that all the data
will be erased.
• All copyrighted brands or names are accompanied by their respective copyright or trademark
signs.
• If data gets corrupted, the game warns the user and suggests reformatting the memory card
or hard drive.
• System setting for a language.
Guidelines are at all levels: design features, fluency, defects, compatibility, etcetera.

41. • QA Team from the manufacturer will test it against their TRC again to be make
certain the game complies with platform conventions. The answer is binary: yes/no.
• Some companies publishes job offers with “supervise some TRCs” as some of the
tasks, as it may be interesting to dedicate someone’s job to be familiar with each
platform.
• As said, anything may be included in the TRC: the manufacturer branding is also
dependent on that.

42. 4. Approaches: Blackbox and Whitebox
Depending on the approach, testing is either Blackbox or Whitebox. This is extensible
to software testing.
1. White-box testing – focuses on the architectural, integration and systemic aspects
of mobile game: how third-party components, databases, social media/external
entities, as well as graphics/game engines, audio play and so on are integrated in to
your game.
2. Black-box testing – focuses on the functional and overall playability aspects of the
game. Menus, graphical elements, special effects, animations, and the actual
gameplay are those under test with Black-box approach.
Why does Blackbox exists? Programmers don’t fully test their own games (anymore).
They don’t have time to, and even if they did, it’s not a good idea.
Job segmentation allows programmers to be more professionals and focus on their
task.
Blackbox is the only testing that guarantees “real conditions”.

43. White-box Testing
White-box testing gives the tester an opportunity to exercise the source code directly
in ways no end user ever could. It takes into account he internal mechanism of a system
or component.
Tests performed by developers prior to submitting new code for integration with the
rest of the game.
Testing code modules that will become part of a reusable library across multiple games
and/or platforms; modules that are parts of a game engine or middleware product;
modules that can be used by third-party developers or “modders”.

44. • Unit testing: it is the testing of individual hardware or software units or groups of
related Units. Testers (usually developers) verify that the code does what it is
intended; it is usually done within a class or a component.
o Unit testing checks a single assumptions about the behavior of one system.
• Integration testing: it is the testing in which software components, hardware
components, or both are combined and tested to evaluate the interaction between
them. Although it is also done in black-box testing, software developers verify that
units work together and check whether they give some error or not (data might get
lost across an interface, messages might not get passed properly, or interfaces might
not be implemented as specified).
o Integration testing checks multiple systems are correctly connected.
• Automated testing: it is the testing in which certain configurations or tests are run
automatically in order to create situations that are difficult to repeat at large scale
by manual testers. Some of the purposes of doing automated testing are to test
loading times, graphics performance, server loading, etcetera.
QUESTION: Why does White-box testing is more considered QA than QC?

45. What kind of tests might have been performed to this in order not to detect the flaw?

46. 2 unit tests. 0 integration tests.

47. Black-box Testing
By their nature, black-box test cases are designed and run by people who do not see
the inner workings of the code. They play as if they were players.

48. • Test Case: it is a set of inputs, the expected results and the actual results (outputs).
• Test Suite: it is a set of test cases.
• Test Plan: it is a document with the test suites and their test cases.
The minimal ’unit’ of black-box testing is the Test Case

49. IMPORTANT: Test Case can be defined as an statement ”placement of an object is that”
to be resolved as “passes/fails” or as a description of the test case, expected results
and actual results (and “passes/fails”).
You can create a test case as:
• Description (situation and order) > Expected Results > Actual Results > Pass/Fails.
The Expected Result is the way the game should work according to its design
specification.
The Actual Result is anomalous behavior observed when you used the game, caused by
a software defect.
Or you can create it as:
• Description (order) with the expected results > Pass/Fails > Description.

50. Test Suite for Monopoly “Getting in Jail”
Requirements according to
some Monopoly rules:
When a user lands on the “Go
to Jail” cell, the player goes
directly to jail, does not pass
go, does not collect $200.
On the next turn, the player
must pay $50 to get out of jail
and does not roll the dice or
advance.
If the player does not have
enough money, he or she is
out of the game.

51. Test cases and suites will be very different depending on the game genre. In a saga, test
suites can be reused. You can generate a test case for the previous ‘characteristics’ you
already know what to expect from.
Again:
• A test plan defines the overall structure of the testing cycle.
• A test case is one specific question or condition the code is operated and evaluated
against.
• A test suite is a group of test cases.
PROPOSED EXERCISE:
Level Test Checklist. Develop this into a test suite.
Check the placement and behavior of objects in the level
Check the placement and behavior of NPCs
Check the fit and form of each unique tile, mesh, and texture used in the level
Check the function and load times of transitions from one level to another

52. 5. Being a Black-box Tester
• Being a black-box tester is being in a loop. Playing only happens once. In a ‘test
suite’, it is not playing, and once you found the first ‘defect’, you are in the middle of
the loop. The loop is called “Piano TV”.
1. Play to crash it, not to have fun.
2. Identify the bug. It may be not important, blocking, etc. Finding bugs is this part.

53. 3. Amplifying the bug. This means that the tester will narrow the possibilities in which
the bug appears. You may have found a bug in a place, but it may occur more than you
think. Amplifying implies testing again (look for all the places which “relate”).
4. Notifying the team. Once the bug is found, you need to report it using a defect
tracking system, describing the most useful information (category and type of bug) and
extra things you can do to help without being too verbose.
The replicability is the most fundamental part (just like science). It is important to put a
clear title and set the priority according to the severity. It is useful to include server
logs, screenshots, saved files, etcetera.
5. Testify to others. In this moment, the tester sends the report and acknowledges the
different teams (lead tester and developer). Remember: testers get emotionally
attached to their found defect (they are sort of medals in the testing sport).
6. Verify the fix. Once the bug is fixed by the developer, the tester needs to try to
reproduce it. Once it is fixed, the tester may close it.

54. Depending on your personality, you may be one type of tester or another.
If you are a Judger, you prefer a structured, ordered, and fairly predictable
environment, where you can make decisions and have things settled.
If you are a Perceiver, you prefer to experience as much of the world as possible.
You like to keep your options open and are most comfortable adapting.
Judger
Runs the tests for...
Conventional game playing
Repetitive testing
User manual, script testing
Factual accuracy of game
Step-by-step or checklist-based
May rely too much on test details
Concerned about game contents
Perceiver
Finds a way to...
Unconventional game playing
Testing variety
Realistic experience of game
Open-ended or outline-based testing
May stray from the original test purpose
Concerned about game context

55. QA and QM:
Team, Phases and
Documentation

56. Testing is the execution. It is the method and the process by which QC ensures
that all the defects are found.
1. How Can We Improve Quality?
Testing may not be enough. We need to plan for quality. We
need three departments.
These are the quality ‘roles and departments’ in Project
Management:
• Quality Control & Testing – controlling a process to ensure
that outcomes are as expected. Quality control is the lower
layer: it includes testing.
• Quality Assurance – obtaining confidence that a product or
service will be satisfactory, as well as planning for it.
• Quality Management – directing an organization so that it
optimizes its performance through analysis and
improvement.

57. Why is there quality assurance if there is already quality control and testing?
In order to focus on quality earlier in the process.
If we could test on forever, there would be no need for planning. Planning is putting
priorities and doing things to optimize results.
Defining the testing and planning the testing are part of Quality Assurance.
Quality Assurance developed from the realization that quality could be improved by
looking 'further up the line'. It is aimed at preventing nonconformities/defects.
There are two kinds of defects that justify QA:
1. The one which is defined according to the design requirements.
2. The one which is perceived by the customer and was not even foreseen by the
development team.
The two must be taken into account by Quality Assurance in order to improve the game
and evaluate the quality.

58. 2. Team and roles
Game teams come in different sizes, shapes, locations, and skill sets. They can vary by
company, game title, or platform.
• It is usual to find a leader defined hierarchy: development lead, build lead, etc. So
responsibilities are assigned.
• QA department may have a leg in the development team and in testing team.
The testing team may be dynamic and grow according to the project needs.
• Lead tester (QA lead) is assigned according to a game.
o She defines the “testability” requirements, procedures and standards.
o She represents the test team in planning and meetings.
o She may do some of the game testing (in small teams).
• Testers (QC). They are the group of final testers who execute the test plan.
• Test Engineer: breaks stuff while programmers try to make it work (basically through
White-box testing). She works tightly with programmers.
• Beta testers (optionally). “Volunteer army” in case of doing pre-releases.

59. Outside the testing team:
• Art Team (animators, artists, etc.), Game Designers (level designers, etc.), Sound
Team (sound engineer, music producer).
• User Experience (responsible for playtesting, surveys, interviews; creating
hypothesis; evaluating the GUI).
• Product Manager / Producer: getting the game done on time and within budget.
Evaluate risks. More and more. The management team and production is QM.

60. The decision-makers are: Producer (QM), QA Manager and Lead Tester.
• Producer Questions: how do we keep the calendar? How do we fit the budget?
• QA Manager Questions: how big is the game? (platforms, map size, etc.) how much
time do we have? How much money do we have?
• Lead tester Questions: What test suites are most important? How can all testers
work on something important? How many bugs are open and which category?
The main responsibilities of the Lead testers are:
managing the test team, designing and implementing the overall test plan, “owning”
the bug database.
The Lead tester must be very a ‘team’ focused person.
Defining documents is (mainly) for the three decision-makers.

61. 3. Documentation
• Quality does not start at testing but on the planning phase. Good Quality
Assurance and Management requires creating a good set of documents.
• Good planning is saving resources (development, testing). Therefore, it is important
to determine the Scope of testing the project will require.
• Testing better earlier can get problems fixed sooner and cheaper. Earlier means
white-box and testing specific parts. It is about balance.
• Documentation is key to coordinate. The most important documents for the test
manager are the game design document (GDD), the test design document (TDD),
software quality assurance plan (SQAP), and the the test plan document (TPD).
For whom is documentation most important?
• GDD—everyone.
• TDD—tech lead, art director, producer, project manager.
• SQAP—project manager, producer, development lead, test lead, QA lead, and
engineer(s). à This document is Quality Management.
• Test plan document—project manager, producer, development lead, key testers.
• Test suites—feature developer, testers.
(QA & QM)

62. Technical Design Document (TDD)
The technical design document sets out the way your tech lead plans to transform the
game design from words on a page to software on a machine.
• It describes the core tools that will be used to build the game, and whether they are
already in the house (not to be bought or created).
• It documents the list of software and hardware, and changes in the organization
infrastructure (storage capacity, backup plans, network speed).
• TDD is sometimes part of the GDD (Game Design Document). When both
documents exist, it is recommended that they do not overlap each other in content.
Ø GDD is more focused on story, characters, gameplay, art, UI (what and why),
instead of the “how” to achieve that.
• It is important to keep track of the versions of the document, and each change
should be documented in a log along with its creator (as a table).
• Different companies may have different TDD outline but the same content.

63. • Its Outline should be something like this:
1. Technical Team: Tech Lead, developers, etcetera.
2. Executive Summary: project overview from a tech. perspective, delivery
platforms, etc.
3. Engine Evaluation (internal solutions, external solutions, middleware)
4. Platform-specific issues (delivery platform #1, #2)
5. Development plan (use cases, game mechanics, main loop, data structures,
data flow, physics, AI, graphics, collision, GUI, fonts, audio/video, especial
requirements)
6. Coding standards (programming standards, style guide, code review
procedures)
7. Scheduling (programming tasks, man-month scheduling, calendar month
scheduling, milestone schedule and deliverables, major event planning).
8. Recruitment (personnel, hiring, risk plan for handling delays in acquiring
resources).
9. Equipment budget and costs (personnel hardware sets, team wide tools,
software tools)
10. Miscellaneous
Game Design, Second Edition. Bob Bates, 2004. p. 209

64. Software Quality Assurance Plan (SQAP)
An SQAP is strictly concerned with the independent monitoring and correction of
product and process quality issues. It is a QM document.
It should address the following topics:
• QA personnel
• Standards to be used
• Reviews and audits that will be conducted
• QA records and reports that will be generated
• QA problem reporting and correction
• QA tools, techniques, and methods
• QA metrics
• Supplier control
• QA records collection
• QA training required
• QA risks

65. Test Plan Document (TPD)
A test plan acts as the playbook for the test team. It identifies the test team’s goals
along with the resources (staff, time, tools, and equipment) and methods that are
necessary to achieve them.
It should contain:
1. QA Team (QA Lead, internal testers, external testers, etc.)
2. Testing Procedures (Daily Activities such as Generate a daily build, important
regression tests, etc.; Integration of reports, different kinds of testing – test suites)
3. Testing Requirements Origin (generated in meetings, generated by producer).
4. Bug Tracking Software (package, instructions how-to, etc.)
5. Bug Classification and Bug Tracking (Who classifies, who assigns, what happens
when fixed, what happens when the fix is verified)
6. Scheduling and Loading (rotation plan, when testers are in and out, how many
testers needed in total)
7. Equipment budget and costs (team members’ hardware, software tools needed,
console debug kit, etc.)
The Test Plan Document is as detailed as the Technical Design Document. Again, it
should not overlap each other.
Example: [https://sites.google.com/site/knightsvszombies/test-plan]

66. 4. Testing Phases and Development
No matter what size the game and how long the production schedule, the testing of
the game should always follow the same basic structure: Pre-production, Kickoff, Alpha,
Beta, Gold, Release and Post-release.
The interesting things are “how these phases are defined” and the “test kickoff”.
In each game, a Lead Tester is assigned and its up to him to specify the criteria for each
phase of testing (in an ideal world, in a real one, other issues matter). On large teams,
more than one primary tester may be assigned (e.g. for multiplayer mode, etc.).
Each phase needs: an entry criteria (set of tests that a build must pass before being
able to tell the game is already at that phase) and the target date for it.

67. Test Kickoff (After Pre-Production)
Test kickoff activities are broken into two parts: tester preparation and the kickoff
meeting, which is conducted according to the kickoff agenda.
For the test kickoff meeting, the tester prepares in the following ways:
• Reads the requirements and/or documentation for the game feature being tested
• Gathers equipment, files, and programs needed for the test
• Reads through the tests
Test kickoff meeting is very important
1. Giving a feature overview
2. Addressing feature questions
3. Bringing up any special test instructions
4. Addressing any test execution questions or issues
• It prepares the teams, it sets ready the equipment, it familiarizes the tester, it
resolves test instruction conflicts, it provides a forum for test improvement.

68. Alpha Phase Entry Criteria (usual ones)
1. All major game features exist and can be tested.
2. A tester can navigate the game along some path from start to finish.
3. The code passes at least 50% of platform TRC.
4. Basic interface is complete and preliminary documentation is available to QA.
5. First-party controllers work.
6. Online multiplayer can be tested.
7. Minimal Performance (e.g. 30 fps)
Beta Phase Entry Criteria (usual ones)
1. All features are implemented.
2. The game can be navigated in all paths.
3. The game passes the 100% of the TRC.
4. The entire user interface is final.
5. The game logic and AI is final.
6. All controllers work.
7. Final artwork is implemented.
The design lock (or feature lock) happens when this phase is concluded.

69. Gold Testing
In all gold testing, the game should be in a similar state to this:
1. All known Severity 1 bugs (crashes, hangs, major function failures)
2. Greater than 90% of all known Severity 2 bugs are fixed.
3. Greater than 85% of all known Severity 3 bugs are fixed.
4. Release-level performance (60-fps frame rate)
Upon these criteria, the game is declared to be at “code lock” and the final versions are
release candidate or gold master candidates (GMCs).
The game is sent to the platform manufacturer, which conducts its own testing on the
TRC. They may find “showstopper bugs”, which need to be fix and resubmitted.
Post-release testing. Patches.

70. 5. Six-sigma and Phase Containment (PCE)
• Quality Management and Quality Assurances aim at improving the testing process,
as it is a way of improving the resources efficiency and product quality.
• Game Quality Measurements can help us in determining “how good” is game
software according to the number of defects it has. They are managed in the SQAP.
Two measures are “Sigma Level” and the “Phase Containment”.
1. Six-sigma is a set of techniques and tools for process improvement. It is based on
statistical methods. The “Sigma Level” of code is computed according to the
number of errors per line (in assembly-equivalent lines of code, AELOC).
2. Phase containment is the ability to detect faults in the project phase in which they
were introduced. Phase containment effective (PCE) coefficient is a measure of
how well the development and testing process is being done.
Both are simple calculations which are fast to compute and easy to understand.
(QM & QA)

71. Sigma Level
Sigma-Level is based on the Gaussian distribution. For instance, 38 errors for 250.000
lines of code is equivalent to a 5.1 Sigma Value.
Released Defects per (AELOC) Sigma
20.000 100.000 250.000 1.000.000 Value
124 621 1552 6.210 4
93 466 1165 4.660 4,1
69 347 867 3.470 4,2
51 256 640 2.560 4,3
37 187 467 1.870 4,4
27 135 337 1.350 4,5
19 96 242 968 4,6
13 68 171 687 4,7
9 48 120 483 4,8
6 33 84 337 4,9
4 23 58 233 5
3 15 39 159 5,1
2 10 27 108 5,2
1 7 18 72 5,3
0 4 12 48 5,4
0 3 8 32 5,5
0 2 5 21 5,6
0 1 3 13 5,7
0 0 2 9 5,8
0 0 1 5 5,9
0 0 0 3,4 6
In this process, the defects being counted must
include both the game defects you know about
that have not been fixed, whatever defects your
customers have already found, and your
projection of defects that remain in the software
which haven’t been discovered yet.

72. Phase Containment (PCE)
Faults that are found in the phase in which they are introduced are known as in-phase
faults or “errors.” If found later, they are “defects” or bugs.
Calculate PCE by dividing the number of in-phase faults by the sum of faults found in all phases to
come up with the PCE for that phase. When new testing phases are added, the PCE can only be
reduced.

73. The higher the PCE, the better, because it means that the bugs created at one phase
are also found in the same phase.
QUESTION: What strategies can be useful to have a higher PCE?
- Improve knowledge of the subject matter and provide relevant training.
- Have successful team members provide mentoring to less-successful members.
- Document methods used by successful individuals and deploy them throughout the
team.
- Increase compliance with existing methods and standards.
- Add standards which, by design, help prevent faults.
- Add checking tools that run during the creation process, such as color-coded and
syntax-aware editors.
- Add checking tools that run after the creation process, such as stronger compilers and
memory leak checkers.

74. 6. Testers Effectiveness and Performance
• Quality Management uses data regarding the testing process (the editors’ work) in
order to understand how well it fits the needs and expectations of the overall
project (in terms of duration, new hiring, etc.).
• It is useful to contrast planned and actual testing execution data (about the number
of tests suite performed), globally and at a tester level. This is contained in the SQAP.
(QM)

75. It is important to take into account the tester “overhead” (trainings, meetings,
preparing for demos, etc.) so, only counting a 75% of the time at best performance, 50
at average performance and so on.
It is useful to count the tests in number of days. This way it is possible to calculate in an
Excel file how many extra testers at part-time or full-time can help in pushing the
project forward.
• It usual that testing teams grow at peak moments of the development process.
Monitoring the team to stay on a short-term commitment help in maintaining long-
term goals on track. However, monitoring is not all good.
Pros:
- Some people will do better if they know they are being “watched”.
- It provides a measurable basis to detect who are the “elite” and avoid favoritism.
- Testers seeking better numbers may interact more with developers to prevent bugs.
Cons:
- Some testers may not be perceived as valuable as they really are (they mentor, etc.).
- It may trigger some jealousy among the team if not channeled properly.
- Testers may argue over who gets the credit for certain defects.

76. Testing Effectiveness
In a similar way to Phase Containment, testing effectiveness is a measurement which
aims at improving the quality control. In this case, it provides a ratio between the
number of defects according to the number of tests performed in a release.
PCE could be used as a reference.
For instance, if you know a PCE coefficient should be 0.60 and 30 tests are performed,
then, at least 2 defects should be found.
Coders may delay a code release until more defects are found.

77. Testing Effectiveness
Testers can be also measured according to their effectiveness.
However, you see “what” is happening, but you do not understand “why” is happening
that way. You do not know the ‘severity’ of the bugs found by each tester.

78. REWIND: Some innocent questions about the test phases
• All bugs MUST be fixed before a game can be certified as GMC. True or false?
• Online multiplayer features can be tested in Alpha. True or false?
• Feature lock should happen in Alpha. True or false?
• The Beta build is the version that will be sent to manufacturing. True or false?
• Describe whether each of the following is an appropriate topic to discuss during a
test execution kick-off, and why:
a) Possible contradictions in the feature requirements
b) Ideas for new tests
c) Company stock prices
d) Identical tests already being run in other test suites
e) How “buggy” this feature was in the previous release
f) Recent changes to the game data file formats
g) Lack of detail in the test case documentation.

79. Summary
• Quality is ‘what is required’: functionalities, expectations and the absence of
bugs (anomalies or non-defined behaviors).
• Quality Management, Quality Assurance and Quality Control are the three
areas which deal with quality in the game development process.
• Testing is the execution of quality control with the aim of finding defects or
bugs. They can be described according to their external characteristics, possible
cause and severity.
• Quality is not coding better or solving bugs, it is detecting what is not working
properly or in best case scenario planning the method to detect. Solving bugs is
development.
• White-box and Black-box testing are two approaches to bug detection usually in
different phases. One is from ”the inside”, as a coder, and the other is “from the
outside”, as a player.
• Black-box testers cannot trust anyone, and need to plan test suites and perform
a rigorous and documented process in order to find bugs, report them and
verify them.