How much information should your test cases (or test missions, charters, or other types or similar test artifacts) include? What are the pros and cons of adding lots of detailed information in your test cases? These are questions I will discuss in this article, based on my experience with testing.

1. Codifying Game & Test Knowledge in Tests?
How much information should your test cases (or test missions, charters, or
other types or similar test artifacts) include? What are the pros and cons of
adding lots of detailed information in your test cases? These are questions I will
discuss in this article, based on my experience with testing.
So imagine you have a test case, which just states that you should test the
feature, and confirm that you experience the desired outcome. If you have no
previous experience with the game and it’s features, performing such a test will
be very difficult, and most likely counter-productive. If you are an experienced
game tester who has worked with the game and it’s features extensively, it may
be enough information for you.
So the key point here is:
 Context-specific domain knowledge (knowing the game and the features)
affects the game tester’s requirements on the test cases (or similar test
artifacts)
What if you are very experienced in general but do not have any game specific
knowledge? Most likely you would run some exploratory testing on the feature
before the actual test to understand how it works, and then discuss your findings
with a developer or designer. This way you get the context-specific domain
knowledge you need to perform the tests using the very vague test cases.
Key point:
 General game testing experience can allow you to mitigate the lack of
context-specific domain knowledge, if you have the means to bridge this
gap through exploration of the game/feature and having the proper
communication channels with designers/developers
But there are obviously exceptions. If you have to enter certain settings into test
tools or environments to be able to test the feature, and this information is not
readily available, it will be difficult to perform the tests. This could be part of the
test case, or of some other test documentation, depending on the context.
Key point:
 Specific test tool and test environment information needs to be available
to the tester regardless of experience
So why not just write very detailed test cases, with intricate step-by-step
instructions for how to do every single thing, so that even someone without any
experience could do it? That would make all of the above irrelevant.
There are a few reasons.

2. Firstly it is very costly to write these kinds of test cases. The game testers have to
spend a lot of time creating these test cases. Time they could have spent doing
more valuable things, like actually testing the game, or other preparations for
testing.
Secondly these test cases will require much more maintenance. Every time a
small detail in the feature changes, the tests need to be updated. This is an
additional cost apart from the upfront creation cost.
Thirdly these types of detailed, scripted tests restrict the tester and reduce the
variation and innovation in the tests performed. You will test everything exactly
the same every time, and there is a greater risk of missing bugs at the periphery
of the test. The tester is executing a test case, rather than focusing on testing the
feature. Running the exact same test over and over again is often not an optimal
use of resources.
Fourthly it gives false confidence that inexperienced game testers can just
perform these tests, and that the confidence in this results will be the same as if
an experienced tester performed the tests. Even though an inexperienced tester
can run through a detailed, scripted test, this does not mean that an experienced
game tester wouldn’t have done it better. And I don’t mean faster – I mean
finding tricky bugs in the area, which the detailed test artifact intends to cover.
You cannot separate test design from test execution and think that test execution
is something mindless which can be outsourced to random people. Good test
execution requires experience and opportunities to utilize that experience when
you perform your tests.
Key points:
 High up-front cost
 High maintenance cost
 Reduces variation and innovation in tests
 False confidence - detailed test cases cannot replace experience
 Test execution cannot be seen as a mindless follow-up to test design
But what about if you are an inexperienced tester – isn’t detailed test cases a
good way to learn the feature/game, and get the initial experience you need?
This is a tempting proposition. An experienced tester transfers his knowledge to
an inexperienced tester through a test case. Personally I am unconvinced this is a
good way to transfer knowledge. I think the best way is either through pair-testing,
mentorships, or through good communication channels. I just don’t think
detailed test cases are a cost-effective way of transferring knowledge.
With this in mind I would like to edit the first key-point I presented:
 Context-specific domain knowledge (knowing the game and the features)
affects the game tester’s requirements on the test cases (or similar test

3. artifacts), or other means to acquire knowledge of how to test the
game/features
So how do you reach the optimal level of detail in a test artifact?
1. Make sure that mandatory things like information about test tools and
test environment is available for the tester when running the test case –
either in the test case or other test documentation
2. Let experienced testers write tests and let other experienced testers try to
run those tests and review the results
3. Make sure you have the framework in place to train inexperienced testers
on the job – for example pair-testing, mentors, or good communication
channels
But even experienced game testers have to understand what needs to be tested,
so that they do not forget to test specific parts of the game or feature. So a test
case that states: “Test the game”, would be problematic for anyone, regardless of
experience.
The goal with a test case (or similar test artifact) should be:
 To help the game tester understand what needs to be tested, not how it
should be tested in detail
If an experienced game tester knows what needs to be tested, then the game
tester can use experience and knowledge to deduct how to test that in the best
way, with regards to the current context, using minimal documentation.
These are my views right now in any event.
/Johan Hoberg