Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Codifying Knowledge in Tests

573 views

Published on

How much information should your test cases (or test missions, charters, or other types or similar test artifacts) include? What are the pros and cons of adding lots of detailed information in your test cases? These are questions I will discuss in this article, based on my experience with testing.

Published in: Engineering
  • Be the first to comment

Codifying Knowledge in Tests

  1. 1. Codifying Game & Test Knowledge in Tests? How much information should your test cases (or test missions, charters, or other types or similar test artifacts) include? What are the pros and cons of adding lots of detailed information in your test cases? These are questions I will discuss in this article, based on my experience with testing. So imagine you have a test case, which just states that you should test the feature, and confirm that you experience the desired outcome. If you have no previous experience with the game and it’s features, performing such a test will be very difficult, and most likely counter-productive. If you are an experienced game tester who has worked with the game and it’s features extensively, it may be enough information for you. So the key point here is:  Context-specific domain knowledge (knowing the game and the features) affects the game tester’s requirements on the test cases (or similar test artifacts) What if you are very experienced in general but do not have any game specific knowledge? Most likely you would run some exploratory testing on the feature before the actual test to understand how it works, and then discuss your findings with a developer or designer. This way you get the context-specific domain knowledge you need to perform the tests using the very vague test cases. Key point:  General game testing experience can allow you to mitigate the lack of context-specific domain knowledge, if you have the means to bridge this gap through exploration of the game/feature and having the proper communication channels with designers/developers But there are obviously exceptions. If you have to enter certain settings into test tools or environments to be able to test the feature, and this information is not readily available, it will be difficult to perform the tests. This could be part of the test case, or of some other test documentation, depending on the context. Key point:  Specific test tool and test environment information needs to be available to the tester regardless of experience So why not just write very detailed test cases, with intricate step-by-step instructions for how to do every single thing, so that even someone without any experience could do it? That would make all of the above irrelevant. There are a few reasons.
  2. 2. Firstly it is very costly to write these kinds of test cases. The game testers have to spend a lot of time creating these test cases. Time they could have spent doing more valuable things, like actually testing the game, or other preparations for testing. Secondly these test cases will require much more maintenance. Every time a small detail in the feature changes, the tests need to be updated. This is an additional cost apart from the upfront creation cost. Thirdly these types of detailed, scripted tests restrict the tester and reduce the variation and innovation in the tests performed. You will test everything exactly the same every time, and there is a greater risk of missing bugs at the periphery of the test. The tester is executing a test case, rather than focusing on testing the feature. Running the exact same test over and over again is often not an optimal use of resources. Fourthly it gives false confidence that inexperienced game testers can just perform these tests, and that the confidence in this results will be the same as if an experienced tester performed the tests. Even though an inexperienced tester can run through a detailed, scripted test, this does not mean that an experienced game tester wouldn’t have done it better. And I don’t mean faster – I mean finding tricky bugs in the area, which the detailed test artifact intends to cover. You cannot separate test design from test execution and think that test execution is something mindless which can be outsourced to random people. Good test execution requires experience and opportunities to utilize that experience when you perform your tests. Key points:  High up-front cost  High maintenance cost  Reduces variation and innovation in tests  False confidence - detailed test cases cannot replace experience  Test execution cannot be seen as a mindless follow-up to test design But what about if you are an inexperienced tester – isn’t detailed test cases a good way to learn the feature/game, and get the initial experience you need? This is a tempting proposition. An experienced tester transfers his knowledge to an inexperienced tester through a test case. Personally I am unconvinced this is a good way to transfer knowledge. I think the best way is either through pair-testing, mentorships, or through good communication channels. I just don’t think detailed test cases are a cost-effective way of transferring knowledge. With this in mind I would like to edit the first key-point I presented:  Context-specific domain knowledge (knowing the game and the features) affects the game tester’s requirements on the test cases (or similar test
  3. 3. artifacts), or other means to acquire knowledge of how to test the game/features So how do you reach the optimal level of detail in a test artifact? 1. Make sure that mandatory things like information about test tools and test environment is available for the tester when running the test case – either in the test case or other test documentation 2. Let experienced testers write tests and let other experienced testers try to run those tests and review the results 3. Make sure you have the framework in place to train inexperienced testers on the job – for example pair-testing, mentors, or good communication channels But even experienced game testers have to understand what needs to be tested, so that they do not forget to test specific parts of the game or feature. So a test case that states: “Test the game”, would be problematic for anyone, regardless of experience. The goal with a test case (or similar test artifact) should be:  To help the game tester understand what needs to be tested, not how it should be tested in detail If an experienced game tester knows what needs to be tested, then the game tester can use experience and knowledge to deduct how to test that in the best way, with regards to the current context, using minimal documentation. These are my views right now in any event. /Johan Hoberg

×