2ContentsPurposeOverviewTest CasesManual Test ScriptsAutomated Test Scripts
3Purpose These slides were put in place over a period oftime, as I found myself often having to leadteams on fast delivery projects, where testershad no previous experience or training. It isvery much a quick guide and assumes that theslides are supported by a skilled practitionerleading the team. The approach assumesbasic level tooling. Obviously with specialisttooling the approach will be adapted. There is also an assumption that other trainingis also available to support test analysis tocreate test cases and the use of specific tools.
4Overview This presentation is written to help those whoare new to test scripting, who may have hadlittle previous experience in this task. The slides address: The layout and management of Test Cases. Naming requirements for Test Scripts. Limitations imposed by Test Tooling. Creation and content of Test Scripts. Structuring of Automated Test Scripts.
5Test CasesA Test Case is described as: “A set of inputs, execution, preconditions, andexpected outcomes developed for a particularobjective, such as to exercise a particular programpath or to verify compliance with a specificrequirement.” BS7925-1 Tools like Classification Tree Editor (CTE) allow usersto generate Test Cases in terms of lists identifyinginput parameters (or Test Vector sets). NOTE: Aseparate presentation is available that deals with CTE. The Test Cases will therefore form a list of what needsto be tested, against a Test Case will be hooked inone or more Test Scripts which provide step by stepinstructions in how to implement the Test Case.
6Test Case and Test Script A test case defines the parameters that arethe vector values which are input for a specificset of tests. Defined with the input values arecorresponding output values that are expectedfor the test outcome. A test script details the step by step actionsand observations to implement the input of testvectors and observations. Often a test case may be implemented in anumber of different scripts. This might occur,for example, if there are different inputmethods.
7Creation of Test Cases Test cases are identified through systematic analysis, to identifythe test vectors to be used for testing. Techniques such as boundary analysis, equivalence partitioning,state transition testing, etc can be found in BS7925-2. Thesetechniques can be applied to system as well as unit testing.Note: Details of BS7925-1 and BS7925-2 can be found at:http://www.testingstandards.co.uk/ An additional technique is classification tree (CT), which is alsosupported by the classification tree editor (CTE) tool.Note: The CT approach and the CTE tool are described in aseparate presentation.
8Test Case Work Packages Following test analysis a series of work packages areidentified. These are the individual test cases, or testvector sets. These test cases then need to be developed into aseries of test scripts. The test scripts outline the starting assumptions andpreparation for implementing a test, followed by stepby step clear repeatable actions and the observationsto make and what is expected to be observed for asuccessful outcome. Later a test manager will want to run specific scriptsassociated with test cases.
9Structuring Test Cases – A Practical Approach Test Cases are structured into well organised foldersto allow efficient management. During a testprogramme, the Test manager will need to: Assign test cases for the creation of test scripts. Run scripts for specific test case sets for different softwaredeliveries and project milestone phases. Assign specific work packages to testers (so a tester will begiven a defined set of tests to run within a period of time). Prioritise testing for a defect fix, software delivery orconfidence test. Consequently Test Cases and Scripts need to be wellorganised and the organisation well understood andeasily recognised by the team.
10Early Organisation of Test Cases Test Cases are usually organised under aseries of folders. The organisation may differfrom project to project. In structuring TestCases or Scripts the following will need to beconsidered as a possible workable approachfor organising Tests: Requirements Business Use Cases Functional Grouping Equipment Functionality Data Type Other….
11Getting Started When creating Test Cases, it may be that theTest Cases have been created with toolingsuch as CTE and the test cases have beencreated as a flat table or series of flat tables.The test manager will want to group these forassigning to testers to create Test Scripts. It is often easier to initially assign the testcases to high level folders and then as theTest Scripts are created, the Test Cases aremoved to a more appropriate folder structuregrouping.
12Setting Out Test Cases Test Cases canbe storedinitially at a highlevel and assorted thentransferred tolower levelfolders.PortalServer AgentProject ABCAdminUser Set-UpSystem User FunctionsPositive TestNegative Test
13Considerations for Priority and Regression Typically during testing a test manager will need to do thefollowing, which can be helped by good Test Case structure: Assign specific sets of Test Cases to specialist testers. Prioritise the creation of Test Cases by Positive testing. Positive tests check that the normal functional path is working for normalinputs. In contrast a negative test will provide an illegal entry and will thus testthe resilience and error handling capability of the functionality. Oftennegative tests are more efficient at finding less obvious defects andspecifically those defects that may cause a major system failure. Prioritise Test Cases for regression test sets. Prioritise Test Cases for automation. Choose Test Cases to test a defect fix. Choose a small subset of Test Cases to decide if a build is fit foraccepting into testing – eliminating bad build injection into test. Choose a subset of Test Cases for reuse in Acceptance Testing.Hence the reason for structure needs to be understood by theteam.
14Dealing with Test Phases It is often simpler tocreate a library ofTest Scripts, wherethese are maintainedcentrally and thencopy down TestScripts to appropriatelevel folders for thetest phase:Project ABCIntegrationBuild1FATUATSATOATCOPYTestSub-FoldersSpecialist testmanagement toolingallows results andscripts to be managedwithout the need forcopying. Howeversome tools are betterthan others.
15Limitations on Folder Structure Be aware that some test tools and olderversions of well established tooling, havelimitations on path length. The total number of characters for the TestCase name (path) and Test Script (file) arelimited. If this happens to be the case then appropriateabbreviations will need to be used that are wellunderstood by the team.
16Agile If using an Agile approach, then you will needto consider how you add Test Cases and mapthese to Agile Stories and at the same timeallow your administration of RegressionTests. Ensure that Test Cases for Performance andSecurity are added early. The content willimprove over the life span of the test delivery,however they need to be created early on inthe test cycle.
17Warning Take great care to maintain a sustainable test scriptlibrary. There is always a danger that with a poorly definedsystem, tests are not maintained and new and evenrepeated tests are added. This is dangerous for aproject because: Older tests will eventually break, causing further delays to testruns. There is a greater maintenance cost as more tests get added. The time to run a suite of tests increases, adding significantcosts to testing. Make certain the test team understand the need formaintenance and make sure that tests for specificfunctions can be quickly identified.
18Test Script Overview Test Scripts have 3 key elements.: Step Number – providing the sequence for Action (which includes Observation). Action – A unique step in a sequence of Action or Observation. The Expected Outcome from the result, against which a Pass or Fail is measured. Note: When using traditional Test Management Tools, further columns are added forPass/Fail and Comments. However if making do with tools such as Excel, there is aneed to add columns for this and have a way of referencing any attachments – such asscreen captures. So you will need to add a column for comments, defect ID crossreference and for any hyperlink to screen captures, which will need to be stored in astructured directory. With an excel solution, there also needs to be an approach for making sure the issue ofthe test script is maintained and the results of [past test runs are mapped to a specificissue of the script. Test script maintenance can also be achieved by each Excel workbook having a masterblank script, which is copied within the file and used for completion during a specificrun. Try to avoid the need to edit multiple scripts, so work maintenance is minimised.Step Action Expected Result1
19Setting Out Test Actions #1 Test Script Actions need to be set out in alogical order of events. Step 1 – will detail any test assumptions. Doconsider that often a test run may at timesvary. So if a test is for example about loggingin with a user name of 3 characters, then statethis early. This then avoids the situation of abusy test team using a standard assigned username with more characters. Or avoiding theautomated tester writing a replacementautomated test script which misses the point ofthe test.
20Setting Out Test Actions #2 In setting out steps, we are ensuring that wecan carry out tests in a precise and orderedapproach that is repeatable. The developershould be able to repeat the test step andobtain the same result and understand exactlywhat the failure was. Test Actions also include the action ofObservation. So for specific observationsensure that these are included as single steps.This then guides the automated tester toensure that specific observations need to becaptured.
21Setting Out Test Actions #3Finally when all the script test actionsare complete, there is a final step whichwill either:Set the system to a known state ready foranother test ORDirect the user to run a cleanup script.
22Level of Detail Required There is a fine balance in ensuring that there is sufficientinformation that can be used to understand what action needs tobe taken and the amount of information that needs to bemaintained. If testing is likely to utilise an army of untrained testers, then moredetail will be required within a test script. This has to beconsidered early on. Depending on the company this couldinclude: undergraduates, new graduates with no or little previousexperience, users, administration staff, etc. So if this is likelythen the scripting needs to be addressed early on in developmentof the scripts. If code is likely to change significantly then you do not want tocreate a large number of test scripts with lots of fine detail that willtake longer to maintain than it takes to change the code. So inthis instance test scripts will be general in description and be lessprone to the need of continuous maintenance. However the keypurpose of the test case and the choice of vectors needs to bestated clearly.
23Test Script Referencing #1 In Test Management Tooling, Test Scripts are often contained ina flat table with a file reference. These files are then hooked intoTest Cases. This approach has a specific advantage that if a testcase is copied and repeated several times, they will alwaysreference the same batch of test scripts, that can be maintainedcentrally. Changing one test script will update all instances of theTest Script. Do ensure that a test run results are not impacted by changes inthe working Test Script. Any change in a Test Script should notimpact any recorded running history, where the previous versionwas used. Not all Test Tools are helpful in this instance. If thiscauses a problem you will need to ensure that any configurationcontrol deployed compensates for the flaw in the tooling. The reason for this is that if you needed to re-run an old versionof a test to check a result, or if development roll back code to aprevious build, you need to ne able to reference the appropriateversion of the test script. If you do not have control of this, thenyou will be heading for problems.
24Test Script Referencing #2 In creating Test Scripts, the Script name willneed to reflect a number of criteria: Identify any specific order that a series of scriptshas to be run in. Identify any priority (High, Medium, Low) that isrequired for automation and regression testing. Identify, where necessary any version controlreference. Identify the functionality being tested. This caninclude reference to positive or negative testing.
25When to Automate Automation creates a level of administration which can at timesbe unsustainable. So in automating ensure: Code has stabilised and is unlikely to need automated tests to beupdated. Do not rely on record and play, where possible use coding to alterdata inputs. This means that scripts are far more easily adaptableand maintainable. Take care with bit map comparisons. Different graphics cards havein the past given rise to sufficient differences to cause failure, wherea pass is correct. Modern GUI automation tools allow the thresholdfor bit map comparison to be adjusted to compensate. Also takecare not to compare a whole screen as routine. Positions may varyslightly and so if looking for example a logo, it is possible in sometools to highlight the image that is to appear on the screen and willpass the test if the image appears anywhere on the screen, withinspecific set boundaries.
26Cutting Duplication in Test Scripting This applies to both automated and manual testing. If there is a set-upsequence say of 12 steps, then if you have to repeat that for 100 testscripts that is 1,200 steps that you need to maintain. This is veryexpensive. Instead consider setting up a preliminary set up script (say called“preliminary_set_up_test24v1”. This would then have those 12 steps.Any test that needed this would then have a step that says “first runpreliminary_set_up_test24” (note no version, however the date and issuecontrol will allow anyone to identify the version run). Any automatedscripting would then run the preliminary test before the extension.However any Test Script name would need to indicate that a preliminarytest needs to be run first. This is then easier to check a regression suitefor the inclusion of any preliminary test. For Manual Testing this is easily implemented by the tester running therepeat step, with a print out. While in the single repeat step will have afailure that will correspond to one of many steps in the preliminarypreparation script, the results should have the ability to detail in whichstep of the preliminary script the failure occurred. However in practice ifone had may repeated failures in a preliminary script, one would raise adefect against the preliminary script and abandon testing on any scriptthat relied on that preliminary test being carried out.