Top Challenges in Testing Requirements


Published on

Studies show that at least half of all software defects are rooted in poor, ambiguous, or incomplete requirements. For decades, testing has complained about the lack of solid concrete requirements, claiming that this makes our task more difficult and in some instances—impossible. Lloyd Roden challenges these beliefs and explains why having detailed requirements can be at best damaging and at worst can even be harmful to both testing and the business. Rather than constantly complaining, Lloyd shows how testers and test managers can rise to the challenges of testing without requirements, testing with evolving requirements, testing with vague requirements, and testing with wrong requirements. To help make your testing more effective, Lloyd provides practical tips and techniques for each of these testing challenges.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Top Challenges in Testing Requirements

  1. 1. T13 Requirements 5/8/2014 1:30:00 PM Top Challenges in Testing Requirements Presented by: Lloyd Roden Lloyd Roden Consultancy Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ ∙
  2. 2. Lloyd Roden Lloyd Roden Consultancy With more than twenty-eight years in the software industry, Lloyd Roden has worked as a developer, test analyst, and test manager for many different organizations. Lloyd was a consultant/partner with Grove Consultants for twelve years. In 2011 he created Lloyd Roden Consultancy, an independent UK-based training and consultancy company specializing in software testing. Lloyd’s passion is to enthuse, excite, and inspire people in the area of software testing. He has spoken at conferences worldwide including STAREAST, STARWEST, Better Software, EuroSTAR, AsiaSTAR, and Special Interest Groups in software testing in several countries. In 2004, he won the European Testing Excellence award.
  3. 3. Top challenges in testing requirements Keynote Presentation Written by Lloyd Roden Version 1_0 © Lloyd Roden
  4. 4. © Lloyd RodenTesting Requirements 131014 testing requirements-1 Top Challenges in Testing Requirements 1 Contents Introduction Top challenges for testing requirements Rising to the challenge 2
  5. 5. © Lloyd RodenTesting Requirements 131014 testing requirements-2 What is a requirement? Definition: !  something that is required/needed !  something essential to the existence of something else why do we have requirements? to make money requirement for a new product to improve requirement for an existing system to remain relevant requirement for changing technology 3 What do requirements look like? specifications use cases user stories story boards emails verbal some forms are better than others 4
  6. 6. © Lloyd RodenTesting Requirements 131014 testing requirements-3 Root Cause Analysis shows… 56% 27% 10% 7% Defects Originate in Project Phase Requirements Design Code Other similar statistics from other projects •  58% requirement bugs for large mobile phone company •  73% requirement bugs for large financial company Source: Binder & Associates 5 Problems that we have with requirements "   non-existent requirements "   expectations are not fully understood "   evolutionary requirements "   constantly changing the scope of the project "   wrong or bad requirements "   wasted time in development and testing "   vague or ambiguous requirements "   leading to misunderstandings "   too much detail in the requirements "   lack of creativity "   too complex requirements "   unable to test effectively "   too many requirements "   overbearing and frustrating 6
  7. 7. © Lloyd RodenTesting Requirements 131014 testing requirements-4 Contents Introduction Top challenges for testing requirements Rising to the challenge 7 Challenge #1: No requirements "   can we test with no requirements? "   we do it all the time "   performance, usability… "   should we test with no requirements? "   challenge is knowing whether something is right “I know it when I see it” “I want something, but I don’t know what” “and by the way, could you give me an estimate” 8
  8. 8. © Lloyd RodenTesting Requirements 131014 testing requirements-5 requirement Challenge #2: Evolutionary requirements requirement idea design code test 9 The good and bad of evolutionary requirements "   excellent when customers don’t really know what they want "   often produces systems that meet customer’s expectations "   high collaboration "   documentation can be sparse/non-existent "   tester’s nightmare "   automation usually not possible until final iteration 10
  9. 9. © Lloyd RodenTesting Requirements 131014 testing requirements-6 Challenge #3: Bad or wrong requirements some bad products… •  all started out as “bad” ideas/requirements •  time “wasted” testing them we need to recognise and challenge any bad/wrong requirements 11 Challenge #4: Vague/ambiguous requirements which do you consider to be “good” requirements "   the help messages must be meaningful and informative "   the fields provided must all have context sensitive help "   the system must be user-friendly "   the system must be reliable "   navigation must be consistent across all applications "   the system updates must be fast "   the website must sustain 5000 simultaneous users "   the mobile phone application must be secure "   developers must be able to fix a bug within 2 days why are some requirements better than others? 12
  10. 10. © Lloyd RodenTesting Requirements 131014 testing requirements-7 What is wrong with these requirements? "   requirement 1 "   requirement 2 online orders can be made at anytime, but will be processed on the next working day. Payment must be made using PayPal or standard credit/debit cards. Orders will be dispatched within 2 days. Customers will be notified via email if a partial order has been sent and can return their order within 28 days and obtain a full refund. working day standard within partial within full A computer program plays chess with one user at a time. It displays the board and pieces on the screen. Moves are made by dragging the pieces 13 Challenge #5: Too much detail in the requirement "   is there such a thing of having too much detail in requirements? "   why are too much detail in requirements a challenge? "   it often prevents creativity in development and test "   mistakes are made due to a lack of questioning "   too much detail could lead to conflicting requirements 14
  11. 11. © Lloyd RodenTesting Requirements 131014 testing requirements-8 Challenge #6: Complex requirements and we call this progress… 1990 2013 1990 2013 1990 2013 “yo, I am outside your door. lol 15 Challenge complexity at every opportunity "   simplicity seen as weak and uninteresting "   who wants a “basic mobile phone?” "   complex is seen as good "   I don’t understand this, so it must be really good (everyone else understands) "   $1m pen suggestion: challenge requirements and design documents at every opportunity to see whether complexity is needed 16
  12. 12. © Lloyd RodenTesting Requirements 131014 testing requirements-9 Features and functions used Jim Johnson XP2002 Standish Study Group Features and Functions Used 16% 13% 7% 19% 45% Sometimes Used Often Used Always Used RarelyUsed Never Used Features and Functions Used 20% 64% 16% Often and Alw ays Used Rarely or Never Used Sometimes this means we have driven up complexity by putting in things that are not required 17 .. If there is too little demand on them, people are bored. If there is too much for them to handle, they get anxious. Flow occurs in that delicate zone between boredom and anxiety. Performance Arousal level Flow Strained concentration Boredom Distracted Csikszentmihalyi Challenge #7: Too many requirements low medium high lowhighmedium Stress Laid back Anxious Panic/Anger/Fear Optimum Performance 18
  13. 13. © Lloyd RodenTesting Requirements 131014 testing requirements-10 MoSCoW "   Must "   describes a requirement that must be satisfied in the final solution for the solution to be considered a success "   Should "   represents a high-priority requirement that should be in the solution if possible. It is often a critical requirement but one which can be satisfied in other ways if necessary "   Could "   describes a requirement which is considered desirable but not necessary. This will be included if time and resources permit "   Wont "   represents a requirement that stakeholders have agreed will not be implemented this time but may be considered for the future 19 Contents Introduction Top challenges for testing requirements Rising to the challenge 20
  14. 14. © Lloyd RodenTesting Requirements 131014 testing requirements-11 Choose the most appropriate lifecycle Integration Testing Acceptance Testing System Testing Component Testing Code Design Specification System Specification Business Requirements sequential iterative agile •  detailed requirements •  complex requirements •  vague requirements •  no requirements •  evolutionary requirements •  vague requirements •  too many requirements •  complex requirements •  vague requirements there is no lifecycle applicable for bad/wrong requirements 21 "   an agile development approach to component testing Use Test-Driven Development pick a req. yesno need more tests for requirement? write enough code to pass the test write component test automate test using a tool 22
  15. 15. © Lloyd RodenTesting Requirements 131014 testing requirements-12 Use test design techniques as a review technique "   example requirement: by reading this requirement we may have a few questions, or we might think we understand what is required. However… A thermostat displays the temperature of a greenhouse from 15o C to 35o C. If the temperature falls below 20o C then “too cold” appears on the display and a blue light flashes. It the temperature exceeds 30o C then “too hot” appears and a red light flashes. 23 Review using test case design techniques "   sharpens our understanding "   generates tests "   finds more defects with requirements Tests for our example: Temperature Expected result 17oC, 15oC, Too Cold, blue light flash 20oC, 25oC, 30oC No display, no light? 32oC, 35oC Too hot: red light flash 10oC, 40oC ??? 2 lights or 1 light? whole degrees? up to and including… where is the temperature taken…isolated or whole room? 24
  16. 16. © Lloyd RodenTesting Requirements 131014 testing requirements-13 Produce some requirement guidelines do not use pronouns do not use unqualified adjectives and adverbs do not use the word “etc” is the requirement is testable? do we have the necessary expertise? are the requirements prioritised? does the requirement need breaking down? is the requirement clear and understood? 25 Use walkthroughs to clarify and understand "   walkthroughs are a powerful static testing technique "   to educate, aid understanding and find bugs in documents "   the power and full potential of the walkthrough is often not recognized or utilized "   watch for what is said and how it is said "   do not underestimate body language "   resist peer pressure and challenge when you don’t understand 26
  17. 17. © Lloyd RodenTesting Requirements 131014 testing requirements-14 Use the Quality Attribute Template Concept Test Scale Plan Must Now Best completeness compare menu and help items percentage of menu items in the help 100% 90% 40% 100% ease of access sample 25 features average number of keystrokes to find a feature 3 7 12 2 accuracy collect data each month number of reported defects in the help <5 20 50? 0 Usability On-linehelp high level attribute sub attribute description of attribute how is it to be m easured what is being m easured m easure to be achieved worse m easure acceptable current m easure ultim ate m easure particularly useful for vague non-functional requirements 27 Perform Exploratory Testing "   it is not obvious what the next test should be "   we want to go beyond the obvious tests "   we want to assess the product quickly "   we want to explore areas of the system that are unclear "   creative skills in testing are being encouraged "   we don’t know all the detail of the requirements ET is useful when… 28
  18. 18. © Lloyd RodenTesting Requirements 131014 testing requirements-15 Summary "   various forms of requirements exist "   there are many challenges that we are often faced with when testing requirements "   testing must rise to these challenges by using a variety of techniques, methods and tools in order to be as effective as we can in testing the requirements 29