Demise of test scripts rise of test ideas
Upcoming SlideShare
Loading in...5

Like this? Share it with your network


Demise of test scripts rise of test ideas

Uploaded on

Tired of doing upfront test script creation in your testing efforts? Feeling bad for demotivating your testers? Want something to replace this sickening approach to software testing? This......

Tired of doing upfront test script creation in your testing efforts? Feeling bad for demotivating your testers? Want something to replace this sickening approach to software testing? This presentation outlines why test scripts are not useful, and how test ideas are the new way forward to better testing. Coverage, traceability, reporting, automation and skills are all covered. Take a quick look and see if you can see there is another way to do software testing that is actually pure common sense.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 314 156 97 23 15
http://deheremap4840 8 8 3 2 1 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Demise of Test Scripts,Rise of Test IdeasRichard Robinson8 April 2013
  • 2. Who is Richard?● 10 year old tester● Consultant / contractorBlog www.hellotestworld.comTwitter @richrichnzSkype richrob79● Began thinking sinceDice Game in 2009● Co-organiser of Sydney Testers● Background in Psychand Computer Science● Enjoys running, piano,photography, travel
  • 3. Subject Matter"Demise of Test Scripts, Rise of Test Ideas"What do I mean by test scripts:● upfront analysis● templated● preconditions, steps, expected result,actual result, pass/fail● not automated
  • 4. One of these...
  • 5. Or these....
  • 6. TerminologyCHARTER - Learning and execution log that captures allrelated information of the feature analysis. Includesmetacognitive comments, tests, questions, bugobservations, modelling and anything else relevant to thetest session.FEATURE - Any chunk of a system or product. Logical sliceof functionality. eg. an interface, a webpage screen, aworkflow scenario.SILO Scratchpad for holding ideas of a common naturesuch as questions, test ideas, or problems.TEST IDEAS - Lets discuss test scripts first, then get on totest ideas...
  • 7. Test scripts - hands upHow do you feel about test scripts?What do you think about when you are readinga test script document?frustrated!RepetitionRepetitionRepetitionRepetitioRepetitionRepOMG!
  • 8. Cem Kaner on test scripts● Myth that scripts teach testers the system● In fact, they do the opposite● Steps switch off learning, intuition,questioning... focus on steps themselves● Testers learn faster when they discoverconcepts, rather than by being told them
  • 9. There are books on this stuff...Adults benefit fromactivity based anddiscovery-based styles oflearning, not scriptedinstruction-based.The Adult LearnerKnowles, Holton, & Swanson
  • 10. Scripts promote confirmation bias● Humans have selective attention● A scripts expected result biases them tofind it, and miss everything else.● This is known as confirmation bias.
  • 11. Inattentional blindness...Test scripting focussed the mind on a strict setof steps and expected outputs.So what happens when something goes wrongthat is not related to the test?Answer: blindness
  • 12. Keith Klain on test scripts...● Testers ignore everthing off-script● Testers cannot react, learnand change their approach● Factory model outdated● Offshoring becoming costly● Focus moving to skilledtesting
  • 13. Lets do something about this...Test scripts paint a bad picture for softwaretesting, quality and dont add much value to aproduct or system.Is there abetter way?
  • 14. Introducing test ideas...● Single words or phrases that represent aconcept to be considered for testing● A checklist used as a guide to inspirevaluable tests relevant to the feature● Exist in any intended granularity● Unit of testing currency that help focustester on what is important
  • 15. Here are some test ideas...Happy pathUsabilityMain flowAlternate flowNegative testsError testsBoundary conditionsDatabase dataConfigurationUser personasProduction recordsProduction systemsPrivacyStandardsSecurityReconciliation testsPlatform performanceMobile
  • 16. Why use test ideas?● Act as a guide, not a control● Antifragile and evolve with project● Does not assume complete specifications● Traceability and progress reporting● Reporting on what was NOT tested● More likely to achieve insight● Empowers and motivates tester● Simplified, natural way of testing
  • 17. Compatible with how people learnWhen faced with a newfeature, if a testers initialunderstanding is engaged,then the tester is morelikely to succeed inunderstanding newconcepts and information.
  • 18. But how do I use test ideas?● Can have none, one or many tests● Can be of any granularity that seemsrelevant to the feature under test● Can be captured in a checklist at any time● Needs to be evaluated against the contextof the testing effort● Should be explored from simple to complex,alongside the learning process● Need to be captured in the summary reportfor the feature under test
  • 19. Demo feature: login pageYour mission is to test the latest feature to bedeveloped, the login page.Go test!
  • 20. Demo of test ideas in a method...Start here:● Developer demo● Repeat simple path● Tour the feature● Early discovery
  • 21. Demo of test ideas in practice..Feature introductionShould behave like any other log in screen.Our data is very important to us, so we should notallow a robot to attack the log in to steal useraccounts.We pride ourselves on appearance so usabililtymight be important.Requires user accounts to be set up. These oneshave already been set up: - dev:password123● Begin charter document
  • 22. Demo of test ideas in practice..● Begin charter documentInfo sourcesMyee, the developerCompare with other log in functions such asfacebook, ebay there a standard for log in screens? Yes - foundone: the spec and data dictionary for detail aroundworkflows, found here
  • 23. Demo of test ideas in practice..● Begin charter documentInitial test ideasInput field boundaries anddefinitionFailed login attemptsLocked accountReset account functionRequest password and usernameStay signed inConfigurationCookiesEncryption of dataSessions and public terminalsMultiple sessions on multipledevicesSecurityUsability - tabbing, enter tosubmit?NotificationsError messages - different for badusername and bad password?Help linkTimeoutRe-captcha on reattempts
  • 24. Demo of test ideas in practice..● Begin charter documentInitial questionsWhat browsers are important?Is mobile browsing supported?How much time do I have to test this?Where can a user log in from?Images didnt display, is this a bug?Is this functionality out of the box or custom built?What is the name of the function?What platform are we operating on?What version of html and ajax are we using?Is the page compliant with YSlow?
  • 25. Model moving partsLog in attempts: first time, less than max, at maxAccount state: unlocked, lockedLog-in state: logged in, logged out, session expiredlogged outTime: recent session expired, first time login on thispc, 2nd time (cookies)Demo of test ideas in practice..● Begin charter document
  • 26. And the rest of the method...● Execute some standard test ideas● Gather more test ideas, seek moreinformation● Investigate deeper test ideas● Consider automation● Consider performance● Consider security● Consider compliance● Document tests● Tester demo
  • 27. Finally the coverage and report
  • 28. Questions?
  • 29. James Bach offers his opinion...Play in browser
  • 30. Method Analysis1. Engage3. Info sources2. Dev demo4. Tour &charter ideas5. Model6. Gather info7. Order ideas8. Investigate9. Automate10. Performance11. Security12. Compliance13. Document14. Test demo
  • 31. Step 1: Engage● Take early notes on the feature● What is important?● Why test at all?● Start your charter
  • 32. Step 2: Dev demo● Best done on testers machine● Prove that simple path works● Prove that config ok● Builds relations● Early questions● Keep it brief
  • 33. Step 3: Identify info sources● People● Drivers● Project state● Contract?● Vendors● Relationship managers● Data dictionary● Database access● Standards
  • 34. Step 4: Tour and charter ideas● System tours at your will● Early tests● Early learning● Gather test ideas● List questions● Identify testability challenges● Get curious!
  • 35. Step 5: Model moving parts● List states of things● List transitions● List actor types● List groups of relevant partitions● List key scenarios around time● List anything that moves● List key business rulesYou may also like to jot down test cases fromsimple, standard flow, negative, complexcases
  • 36. Step 6: Gather more info● Read and gather more information● Process more test ideas from it● Refer to accronym heuristics○ FEWHICCUPS, SFDPOT● Ask questions● Learn learn learn!
  • 37. Step 7: Order your ideas● Checkpoint● Consider wider context● Stop look listen○ Time○ Business drivers○ Stakeholder messages○ Contractual requirements○ Trouble areas (developer, history, complexity)● Order tests by complexity○ Main flows / alternate flows○ Error cases○ Complex interactions
  • 38. Investigate test ideas strength-based● Gradually spiral outwards, from simple teststo complexTimeDepthBADevTester
  • 39. Step 8: Begin test idea experiments!● You can start this any time, but you shouldbe launching into it by now● You are feeling confident about the feature● Your learning is now being cashed in● Conduct regular checkpoints● Raise bugs● Draft test procedures that may be reused
  • 40. Step 9: Consider automationSupport automation● What tests require code to execute?● How can you improve the efficiency?● What tools might assist the testing?Publish to auto framework● Propose some tests for inclusion● Purpose is to check only, not test● Monitoring function
  • 41. Step 10: Consider performance● Performance is a major concern at go live● How does load impact your feature?● How can you increase your volume?● Is this consideration relevant?
  • 42. Step 11: Consider security● Security compliance is a majorconsideration for most systems● Sensitive data● Encryption● Hacking● Basic penetration tests
  • 43. Step 12: Consider compliance● Non-compliance can halt a project dead● Compliance standards● Safety of data● Global standards
  • 44. Step 13: Document tests● List out tests into specification● Indicate which are simple, standard, andcomplex tests● Offers evidence of process for auditors● Charters are usually messy● Useful for identifying tests for regression
  • 45. Step 14: Tester demo● Tester demonstrates feature findings● Chance to give back info discovered● Chance to demo skills and test cases
  • 46. Wise words...
  • 47. Final word...Play in browser
  • 48. ResourcesMetrics and MeasurementMetrics, Qualitative Measurement, andStakeholder, 2009Software Engineering Metrics: What Do TheyMeasure and How Do We Know? & Bond, 2004Testing CoverageSoftware Negligence and Testing Coverage, 1995Exploratory TestingExploratory Testing Explained, 2003Session Based Test Management Bach, 2000Test Heuristics Cheat Sheet, 2006Mnuemonic mind map to Adjust to the Changing Face of Software Testing?, March, 2013Blog: FEW HICCUPPS, July 2012
  • 49. Appendix: Meetup responses"What do you think of test scripts?"
  • 50. Appendix: Charter templateCHARTERHYP/MISSION:TASKS:MODEL DATA:INITIAL QUESTIONS:INFORMATION:EVIDENCE LOCATION:ACCESS:TESTS:SCOPE:VERIFICATION METHODSLEARNING TASKS=============DEV DEMO -OPERATION LEVEL TESTS -READ KEY DOCUMENTATION -REVIEW APPROACH DESCRIPTIONMODEL MOVING PARTSREVIEW SPECIFIC TEST IDEASDB Field Check -Mandatory Fields -Optional Fields -Field Length Boundaries -Field TypesFull/Partial RecordsDuplicate RecordsXSD Structure Check -Validation Rules EnforcedNegative Pathway TestsMultiple Sets of DataLogical Flow TestsError Coverage -Intent Based Tests -Check Logging for ErrorsPrivacy & EncryptionStandards -Risk EvaluationLight Load Tests -Race Conditions -UNIT TESTS IN SOAPUISCENARIO IDENTIFICATION -DEV QUESTION & DEMOINTEGRATE FULLY WITHFRAMEWORK -DOCUMENT TEST CASESDEMO TO TEAMSUMMARY OF KEY FINDINGS============REPORT TO PM============
  • 51. Appendix: Skills framework● Self-organised modules● Written by testers for testers● Closely related to specific test ideaseg. DB Sql statements, bash scripts forlogging, manipulation of data in np++