Paradigms of Black Box Software Testing

2,161 views

Published on

  • Be the first to comment

  • Be the first to like this

Paradigms of Black Box Software Testing

  1. 1. Paradigms of Black Box Software Testing Tutorial at Quality Week, 2002 Cem Kaner, J.D., Ph.D. Florida Institute of Technology kaner@kaner.com www.kaner.com www.badsoftware.com This material is based upon work supported by the National Science Foundation under Grant No. EIA-0113539 ITR/SY+PE “Improving the Education of Software Testers.” Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).
  2. 2. About Cem Kaner Professor of Computer Sciences, Florida Institute of Technology Career focus--improving software customer satisfaction and safety. I’ve worked as a programmer, tester, writer, teacher, user interface designer, software salesperson, organization development consultant, development consultant, manager / director of almost all aspects of software development, and as an attorney focusing on the law of software quality. Senior author of: Testing Computer Software (1988; 2nd edition with Hung Nguyen and Jack Falk,1993). This received the Award of Excellence in the Society for Technical Communication’s Northern California Technical Publications Competition. Bad Software: What To Do When Software Fails (with David Pels, 1998). Ralph Nader called this book “a how-to book for consumer protection in the Information Age.” and Lessons Learned in Software Testing (with James Bach & Bret Pettichord, 2001). B.A. in Arts & Sciences (Math, Philosophy); Ph.D. in Experimental Psychology (Human Perception & Performance: Psychophysics), J.D. (law degree). Member of the American Law Institute. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 2
  3. 3. Paradigms of Black Box Testing ACKNOWLEDGEMENT This section is based on work done jointly with James Bach, and uses slides that James and I prepared jointly for presentation at the USPDI and STAR conferences. We also thank Bob Stahl, Brian Marick, Hans Schaefer, and Hans Buwalda for several insights. ABOUT JAMES BACH • Principal of Satisfice, Inc., a software testing consulting company. • Writes, speaks, and consults on software quality asssurance and testing. In his sixteen years in Silicon Valley-style software companies, including nine years at Apple Computer, Borland International, and Aurigin Systems, he's been a programmer, tester, QA manager, and roving problem-solver. For three years, James was Chief Scientist at ST Labs, an independent software testing company in Seattle that performs substantial testing and tester training for Microsoft. • James is a frequent speaker and writer on the subject of software quality assurance and development processes. James edited the "Software Realities" column in IEEE Computer magazine, is a former section chair of the American Society for Quality and the Silicon Valley Software Quality Association, and was part of the ASQ committee that created the Certified Software Quality Engineer program. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 3
  4. 4. Background Thoughts on Test Case Design If the purpose of testing is to gain information about the product, then a test case’s function is to elicit information quickly and efficiently. In information theory, we define “information” in terms of reduction of uncertainty. If there is little uncertainty, there is little information to be gained. A test case that promises no information is poorly designed. A good test case will provide information of value whether the program passes the test or fails it. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 4
  5. 5. Dimensions of Test Case Quality Think of a test case as a question you ask of the program. What are the attributes of the excellent question? • Information » Reduction of uncertainty. How much do you expect to learn from this test? • Power » If two tests have the potential to expose the same type of error, one test is more powerful if it is more likely to expose the error. • Credibility » Failure of the test should motivate someone to fix it, not to dismiss it. • Feasibility » How hard is it to set up the test, run the test, and evaluate the result? Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 5
  6. 6. Models A Model is… • A map of a territory • A simplified perspective • A relationship of ideas • An incomplete representation of reality • A diagram, list, outline, matrix… No good test design has ever been done without models. But models are often implicit, partially because of incomplete specs and conflicting stakeholder interests A key goal is to become aware of how you model the product, and learn different ways of modeling. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 6
  7. 7. The Puzzle Black box testing groups vary widely in their approach to testing. Tests that seem essential to one group seem uninteresting or irrelevant to another. Big differences can appear even when both groups are composed of intelligent, experienced, dedicated people. Why? Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 7
  8. 8. Paradigms A paradigm provides a structure of thinking about an area. • Typically, the description of a paradigm includes one or a few paradigmatic cases -- key example. Much of the reasoning within the paradigm is based on analogies to these key cases. • The paradigm creates a mainstream of thinking--it provides insights and a direction for further research or work. But it implicitly also defines limits on what is relevant, interesting, or possible. Things outside of the paradigm are uninteresting. People who solve the types of puzzles that are characteristic of the paradigm are respected, whereas people who solve other types of puzzles instead are outsiders, and not well respected. • A testing paradigm would define the types of tests that are (to the person operating under this paradigm) relevant and interesting. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 8
  9. 9. Paradigms (Kuhn) See Thomas Kuhn, The Structure of Scientific Revolutions. He describes paradigms as follows: • A paradigm is a model based on a shared experiment that includes law, theory, application and instrumentation together and from which springs particular coherent traditions of scientific research. • A paradigm includes a concrete scientific achievement as a locus of professional commitment, prior to the various concepts, laws, theories and points abstracted from it. • The pre-paradigm period . . . is regularly marked by frequent and deep debate over legitimate methods, problems, and standards of solution, though these serve rather to define schools than to produce agreement. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 9
  10. 10. Black Box Testing Paradigms • There are strikingly different paradigms (or sub- paradigms) within black box testing. • This list reflects our (Kaner’s/Bach’s) observations in the field and is not exhaustive. • We put one on the list if we’ve seen credible testers drive their thinking about black box testing in the way we describe. A paradigm for one person might merely be a technique for another. • We recommend that you try to master the “right” combination of two or three approaches. They are not mutually exclusive. The right combination will depend on your situation. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 10
  11. 11. A Quick Tour of the Paradigms • Function testing • Domain testing • Specification-based testing • Risk-based testing • Stress testing • Regression testing • User testing • Scenario testing • Exploratory testing • Stochastic or Random testing • <<combination testing>> Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 11
  12. 12. Function Testing Tag line • “Black box unit testing.” Fundamental question or goal • Test each function thoroughly, one at a time. Paradigmatic case(s) • Spreadsheet, test each item in isolation. • Database, test each report in isolation Strengths • Thorough analysis of each item tested Blind spots • Misses interactions, misses exploration of the benefits offered by the program. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 12
  13. 13. Domain Testing Tag lines • “Try ranges and options.” • “Subdivide the world into classes.” AKA partitioning, equivalence analysis, boundary analysis Fundamental question or goal: • This confronts the problem that there are too many test cases for anyone to run. This is a stratified sampling strategy that provides a rationale for selecting a few test cases from a huge population. General approach: • Divide the set of possible values of a field into subsets, pick values to represent each subset. Typical values will be at boundaries. More generally, the goal is to find a “best representative” for each subset, and to run tests with these representatives. • Advanced approach: combine tests of several “best representatives”. Several approaches to choosing optimal small set of combinations. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 13
  14. 14. Domain Testing Paradigmatic case(s) • Equivalence analysis of a simple numeric field. • Printer compatibility testing (multidimensional variable, doesn’t map to a simple numeric field, but stratified sampling is essential.) • In classical domain testing • Two values (single points or n-tuples) are equivalent if the program would take the same path in response to each. • The classical domain strategies all assume • that the predicate interpretations are simple, linear inequalities. • the input space is continuous and • coincidental correctness is disallowed. • It is possible to move away from these assumptions, but the cost can be high, and the emphasis on paths is troublesome because of the high number of possible paths through the program. Clarke, Hassell, & Richardson, p. 388 Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 14
  15. 15. Domain Testing Strengths • Find highest probability errors with a relatively small set of tests. • Intuitively clear approach, generalizes well Blind spots • Errors that are not at boundaries or in obvious special cases. • Also, the actual domains are often unknowable. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 15
  16. 16. Specification-Driven Testing Tag line: • “Verify every claim.” Fundamental question or goal • Check the product’s conformance with every statement in every spec, requirements document, etc. Paradigmatic case(s) • Traceability matrix, tracks test cases associated with each specification item. • User documentation testing Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 16
  17. 17. Specification-Driven Testing Strengths • Critical defense against warranty claims, fraud charges, loss of credibility with customers. • Effective for managing scope / expectations of regulatory- driven testing • Reduces support costs / customer complaints by ensuring that no false or misleading representations are made to customers. Blind spots • Any issues not in the specs or treated badly in the specs /documentation. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 17
  18. 18. Traceability Matrix Var 1 Var 2 Var 3 Var 4 Var 5 Test 1 X X X Test 2 X X Test 3 X X X Test 4 X X Test 5 X X Test 6 X X Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 18
  19. 19. Traceability Matrix The columns involve different test items. A test item might be a function, a variable, an assertion in a specification or requirements document, a device that must be tested, any item that must be shown to have been tested. The rows are test cases. The cells show which test case tests which items. If a feature changes, you can quickly see which tests must be reanalyzed, probably rewritten. In general, you can trace back from a given item of interest to the tests that cover it. This doesn’t specify the tests, it merely maps their coverage. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 19
  20. 20. Risk-Based Testing Tag line • “Find big bugs first.” Fundamental question or goal • Define and refine tests in terms of the kind of problem (or risk) that you are trying to manage Paradigmatic case(s) • Failure Mode and Effects Analysis (FMEA) • Sample from predicted-bugs list. • Stress tests, error handling tests, security tests, tests looking for predicted or feared errors. • Equivalence class analysis, reformulated. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 20
  21. 21. Risk-Based Testing Strengths • High power tests • Intuitive tests Blind spots • Risks that were not identified or imagined Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 21
  22. 22. Stress Testing Tag line • “Overwhelm the product.” • “Drive it through failure.” Fundamental question or goal • Learn about the capabilities and weaknesses of the product by driving it through failure and beyond. What does failure at extremes tell us about changes needed in the program’s handling of normal cases? Paradigmatic case(s) • Buffer overflow bugs • High volumes of data, device connections, long transaction chains • Low memory conditions, device failures, viruses, other crises. Strengths • Expose weaknesses that will arise in the field. • Expose security risks. Blind spots • Weaknesses that are not made more visible by stress. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 22
  23. 23. Regression Testing Tag line • “Repeat testing after changes.” Fundamental question or goal • Manage the risks that (a) a bug fix didn’t fix the bug or (b) the fix (or other change) had a side effect. Paradigmatic case(s) • Bug regression (Show that a bug was not fixed) • Old fix regression (Show that an old bug fix was broken) • General functional regression (Show that a change caused a working area to break.) • Automated GUI regression suites Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 23
  24. 24. Regression Testing Strengths • Reassuring, confidence building, regulator-friendly Blind spots / weaknesses • Anything not covered in the regression series. • Repeating the same tests means not looking for the bugs that can be found by other tests. • Pesticide paradox. • Low yield from automated regression tests. • Maintenance of this standard list can be costly and distracting from the search for defects. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 24
  25. 25. Regression Automation Regression tools dominate the automated testing market. Why automate tests that the program has already passed? • Percentage of bugs found with already-passed tests is about 5-20% • Efficiency of regression testing shows up primarily in the next version or in a port to another platform. If the goal is to find new bugs quickly and efficiently, we should use a method based on new test cases. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 25
  26. 26. User Testing Tag line • Strive for realism • Let’s try this with real humans (for a change). Fundamental question or goal • Identify failures that will arise in the hands of a person, i.e. breakdowns in the overall human/machine/software system. Paradigmatic case(s) • Beta testing • In-house experiments using a stratified sample of target market • Usability testing Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 26
  27. 27. User Testing Strengths • Design issues are more credibly exposed. • Can demonstrate that some aspects of product are incomprehensible or lead to high error rates in use. • In-house tests can be monitored with flight recorders (capture/replay, video), debuggers, other tools. • In-house tests can focus on areas / tasks that you think are (or should be) controversial. Blind spots • Coverage is not assured (serious misses from beta test, other user tests) • Test cases can be poorly designed, trivial, unlikely to detect subtle errors. • Beta testing is not free, beta testers are not skilled as testers, the technical results are mixed. Distinguish marketing betas from technical betas. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 27
  28. 28. Scenario Testing Tag lines • “Do something useful and interesting” • “Do one thing after another.” Fundamental question or goal • Challenging cases that reflect real use. Paradigmatic case(s) • Appraise product against business rules, customer data, competitors’ output • Life history testing (Hans Buwalda’s “soap opera testing.”) • Use cases are a simpler form, often derived from product capabilities and user model rather than from naturalistic observation of systems of this kind. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 28
  29. 29. Scenario Testing The ideal scenario has several characteristics: • It is realistic (e.g. it comes from actual customer or competitor situations). • There is no ambiguity about whether a test passed or failed. • The test is complex, that is, it uses several features and functions. • There is a stakeholder who has influence and will protest if the program doesn’t pass this scenario. Strengths • Complex, realistic events. Can handle (help with) situations that are too complex to model. • Exposes failures that occur (develop) over time Blind spots • Single function failures can make this test inefficient. • Must think carefully to achieve good coverage. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 29
  30. 30. Exploratory Testing Tag line • “Simultaneous learning, planning, and testing.” Fundamental question or goal • Software comes to tester under-documented and/or late. Tester must simultaneously learn about the product and about the test cases / strategies that will reveal the product and its defects. Paradigmatic case(s) • Skilled exploratory testing of the full product • Guerrilla testing (lightly but harshly test an area for a finite time) • Rapid testing • Emergency testing (including thrown-over-the-wall test-it-today testing.) • Third party components. • Troubleshooting / follow-up testing of defects. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 30
  31. 31. Exploratory Testing Simultaneously: • Learn about the product • Learn about the market • Learn about the ways the product could fail • Learn about the weaknesses of the product • Learn about how to test the product • Test the product • Report the problems • Advocate for repairs • Develop new tests based on what you have learned so far. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 31
  32. 32. Exploratory Testing Strengths • Customer-focused, risk-focused • Takes advantage of each tester’s strengths • Responsive to changing circumstances • Well managed, it avoids duplicative analysis and testing • High bug find rates Blind spots • The less we know, the more we risk missing. • Limited by each tester’s weaknesses (can mitigate this with careful management) • This is skilled work, unsupervised juniors aren’t very good at it. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 32
  33. 33. Random / Statistical Testing Tag line • “High-volume testing with new cases all the time.” Fundamental question or goal • Have the computer create, execute, and evaluate huge numbers of tests. » The individual tests are not all that powerful, nor all that compelling. » The power of the approach lies in the large number of tests. » These broaden the sample, and they may test the program over a long period of time, giving us insight into longer term issues. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 33
  34. 34. Random / Statistical Testing Paradigmatic case(s) • Oracle-driven, validate a function or subsystem (such as function equivalence testing) • Stochastic (state transition) testing looking for specific failures (assertions, leaks, etc.) • Statistical reliability estimation • Partial or heuristic oracle, to find some types of errors without overall validation. Strengths • Regression doesn’t depend on same old test every time. • Partial oracles can find errors in young code quickly and cheaply. • Less likely to miss internal optimizations that are invisible from outside. • Can detect failures arising out of long, complex chains that would be hard to create as planned tests. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 34
  35. 35. Random / Statistical Testing Blind spots and Concerns • Need to be able to distinguish pass from failure. Too many people think “Not crash = not fail.” • Also, these methods will often cover many types of risks, but will obscure the need for other tests that are not amenable to automation. • Inappropriate prestige difference between automators and skilled manual testers Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 35
  36. 36. Combination Testing I don’t think combination testing is really a paradigm, but it involves a far-from-universal structured way of thinking about testing. Tag line • “Systematically and efficiently test variables in combination. Fundamental question or goal • To find some defects (or replicate some failures) you must control (consciously set) the values of two or more variables at the same time. • Reduce a vast number of possible combinations to a sensible subset. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 36
  37. 37. Combination Testing Paradigmatic cases: • Combining boundaries of several variables • Combinatorial testing (e.g. all-pairs) • Orthogonal arrays • Cause-effect graphing Contrast with scenario testing • Coverage vs. credibility • Systematic exploration of variable values vs. systematic exploration of market (stakeholder) concerns Strengths • If well done, broad combination coverage with relatively few cases Concerns • Tests are typically artificial, may not be credible Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 37
  38. 38. Paradigms and Techniques The paradigms overlap. For example, you can do: • Spec-based regression testing • User scenario testing • Risk-based automated GUI regression testing • Exploratory domain testing The difference lies in what’s in the mind of the tester. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 38
  39. 39. Paradigms and Techniques A paradigm is a mental framework within which you design tests. A technique is a recipe for creating tests. The design of any test case involves at least five dimensions. A technique specifies one or more of these dimensions: • Coverage: What you’re testing • Risks: Problems you are trying to find • Evaluation: How you know a test passed or failed • Activities: How you test • Testers: Who is testing Any of the paradigms could describe a technique or family of techniques, but a “technique” user is likely to use several techniques, not rely primarily on one as the prototypic approach to testing. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 39
  40. 40. A Closer Look at the Paradigms Function Testing Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 40
  41. 41. Some Function Testing Tasks Identify the program’s features / commands • From specifications or the draft user manual • From walking through the user interface • From trying commands at the command line • From searching the program or resource files for command names Identify variables used by the functions and test their boundaries. Identify environmental variables that may constrain the function under test. Use each function in a mainstream way (positive testing) and push it in as many ways as possible, as hard as possible. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 41
  42. 42. A Closer Look at the Paradigms Domain Testing Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 42
  43. 43. Domain Testing Some of the Key Tasks • Partitioning into equivalence classes • Discovering best representatives of the sub-classes • Combining tests of several fields • Create boundary charts • Find fields / variables / environmental conditions • Identify constraints (non-independence) in the relationships among variables. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 43
  44. 44. Domain Testing: The Classic Example The program reads three integer values from a card. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral. » From Glen Myers, The Art of Software Testing Write a set of test cases that would adequately test this program. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 44
  45. 45. Another Standard Example Here is the program’s specification: • This program is designed to add two numbers, which you will enter • Each number should be one or two digits • The program will print the sum. Press Enter after each number • To start the program, type ADDER Before you start testing, do you have any questions about the spec? Refer to Testing Computer Software, Chapter 1, page 1 Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 45
  46. 46. Brainstorm Test Ideas Brainstorming Rules: • The goal is to get lots of ideas. You are brainstorming together to discover categories of possible tests. • There are more great ideas out there than you think. • Don’t criticize others’ contributions. • Jokes are OK, and are often valuable. • Work later, alone or in a much smaller group, to eliminate redundancy, cut bad ideas, and refine and optimize the specific tests. • Facilitator and recorder keep their opinions to themselves. We’ll work more on brainstorming and, generally, on thinking in groups later. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 46
  47. 47. Brainstorm Test Ideas What? Why? ______________ ________________ ______________ ________________ ______________ ________________ ______________ ________________ ______________ ________________ ______________ ________________ ______________ ________________ ______________ ________________ ______________ ________________ ______________ ________________ Refer to Testing Computer Software, pages 4-5, for examples. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 47
  48. 48. Domains: Brainstormed Test Ideas There are too many tests. There are 199 x 199 = 39,601 test cases for valid values: • definitely valid: 0 to 99 • might be valid: -99 to –1 There are infinitely many invalid cases: • 100 and above • -100 and below • anything non-numeric Some people want to automate these tests. • Can you run all the tests? • How will you tell whether the program passed or failed? We cannot afford to run every possible test. We need a method for choosing a few tests that will represent the rest. Equivalence analysis is the most widely used approach. • refer to Testing Computer Software pages 4-5 and 125-132 Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 48
  49. 49. Domains: Classical Equivalence Class and Boundary Analysis • To avoid unnecessary testing, partition (divide) the range of inputs into groups of equivalent tests. • Then treat an input value from the equivalence class as representative of the full group. • Two tests are equivalent if we would expect that an error revealed by one would be revealed by the other. • Boundaries mark the point or zone of transition from one equivalence class to another. The program is more likely to fail at a boundary, so these are good members of equivalence classes to use. Myers, Art of Software Testing, 45 Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 49
  50. 50. Domains: Back to the Test Ideas For each test idea on the brainstormed list: • What class does this belong to? » How can we decide what class? Well, what kind of defect is this test well suited to detect? • What other members of the class are there? » What other tests are well suited to detect this type of defect? • Which tests in this class are the most powerful -- most likely to detect this type of defect? » These are your best representatives. Often, but not always, they are boundary cases. • Note the broad range of test ideas, associated with the broad range of risks. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 50
  51. 51. Expanding the Notion of Equivalence Consider these cases. Are these paired tests equivalent? (55+56, 56+57) (57+58, 58+59) (59+60, 60+61) (61+62, 62+63) (63+64, 64+65) (65+66, 66+67) (67+68, 68+69) (69+70, 70+71) Example: The Hockey Game Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 51
  52. 52. Another Example: Boundaries May Not Be Obvious Character ASCII Code / 47 lower bound 0 48 1 49 Refer to Testing 2 50 Computer Software, pages 9-11 3 51 4 52 5 53 6 54 7 55 8 56 upper bound 9 57 : 58 A 65 a 97 Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 52
  53. 53. Equivalence Classes and Partitions are Subjective. My working definition of equivalence: Two test cases are equivalent if you expect the same result from each. This is fundamentally subjective. It depends on what you expect. And what you expect depends on what errors you can anticipate: Two test cases can only be equivalent by reference to a specifiable risk. Two different testers will have different theories about how programs can fail, and therefore they will come up with different classes. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 53
  54. 54. Equivalence Classes and Boundaries Two tests belong to the same equivalence class if you expect the same result (pass / fail) of each. Testing multiple members of the same equivalence class is, by definition, redundant testing. Boundaries mark the point or zone of transition from one equivalence class to another. The program is more likely to fail at a boundary, so these are the best members of (simple, numeric) equivalence classes to use. Note how the boundary case has two ways to fail. It can fail because the program’s treatment of the equivalence class is broken OR because the programmer’s treatment of inequalities is broken. More generally, you look to subdivide a space of possible tests into relatively few classes and to run a few cases of each. You’d like to pick the most powerful tests from each class. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 54
  55. 55. Boundary Analysis Table V ariab le E q u ivalen ce A ltern ate B o u n d aries N o tes C lass E q u ivalen ce an d S p ecial C lass C ases F irst n u m b er -99 to 99 > 99 99, 100 < -99 -99, -100 digits non-digits /, 0, 9, : leading spaces or 0s expressions null entry 2n d n u m b er sam e as first sam e as first sam e Sum -198 to 198 ??? ??? A re there other -127 to 127 -198 to –128 127, 128, - sources of data 128 to 198 127, -128 for this variable? W ays to feed it bad data? Note that we’ve dropped the issue of “valid” and “invalid.” This lets us generalize to partitioning strategies that don’t have the concept of “valid” -- for example, printer equivalence classes. (For discussion of device compatibility testing, see Kaner et al., Chapter 8.) Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 55
  56. 56. Domain Testing In classical domain testing • Two values (single points or n-tuples) are equivalent if the program would take the same path in response to each. The classical domain strategies all assume • that the predicate interpretations are simple, linear inequalities. • the input space is continuous and • coincidental correctness is disallowed. “It is possible to move away from these assumptions, but the cost can be high. The emphasis on paths is troublesome because of the high number of possible paths through the program.” Clarke, Hassell, & Richardson, p. 388 Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 56
  57. 57. Exercise For each of the following, list • The variable(s) of interest • The valid and invalid classes • The boundary value test cases. 1. FoodVan delivers groceries to customers who order food over the Net. To decide whether to buy more more vans, FV tracks the number of customers who call for a van. A clerk enters the number of calls into a database each day. Based on previous experience, the database is set to challenge (ask, “Are you sure?”) any number greater than 400 calls. 2. FoodVan schedules drivers one day in advance. To be eligible for an assignment, a driver must have special permission or she must have driven within 30 days of the shift she will be assigned to. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 57
  58. 58. Domains: Notes on Myers’ Exercise Several classes of issues were missed by most students. For example: • Few students checked whether they were producing valid triangles. (1,2,3) and (1,2,4) cannot be the lengths of any triangle. » Knowledge of the subject matter of the program under test will enable you to create test cases that are not directly suggested by the specification. If you lack that knowledge, you will miss key tests. (This knowledge is sometimes called “domain knowledge”, not to be confused with “domain testing.”) • Few students checked non-numeric values, bad delimiters, or non-integers. • The only boundaries tested were at MaxInt or 0. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 58
  59. 59. Domains: Myers’ Answer • Test case for a valid scalene triangle • Test case for a valid equilateral triangle • Three test cases for valid isosceles triangles (a=b, b=c, a=c) • One, two or three sides has zero value (5 cases) • One side has a negative • Sum of two numbers equals the third (e.g. 1,2,3) is invalid b/c not a triangle (tried with 3 permutations a+b=c, a+c=b, b+c=a) • Sum of two numbers is less than the third (e.g. 1,2,4) (3 permutations) • Non-integer • Wrong number of values (too many, too few) Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 59
  60. 60. Domain Testing Some Relevant Skills • Identify ambiguities in specifications or descriptions of fields • Find biggest / smallest values of a field • Discover common and distinguishing characteristics of multi-dimensional fields, that would justify classifying some values as “equivalent” to each other and different from other groups of values. • Standard variable combination methods, such as all-pairs or the approaches in Jorgensen and Beizer’s books Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 60
  61. 61. Domain Testing Practice exercises at home • Find the biggest / smallest accepted value in a field • Find the biggest / smallest value that fits in a field • Partition fields • Read specifications to determine the actual boundaries • Create boundary charts for several variables • Create standard domain testing charts for different types of variables • For finding variables, see notes on function testing Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 61
  62. 62. Domains: Notes on Myers’ Exercise Several classes of issues were missed by most students. For example: • Few students checked whether they were producing valid triangles. (1,2,3) and (1,2,4) cannot be the lengths of any triangle. » Knowledge of the subject matter of the program under test will enable you to create test cases that are not directly suggested by the specification. If you lack that knowledge, you will miss key tests. (This knowledge is sometimes called “domain knowledge”, not to be confused with “domain testing.”) • Few students checked non-numeric values, bad delimiters, or non-integers. • The only boundaries tested were at MaxInt or 0. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 62
  63. 63. Domains: Myers’ Answer • Test case for a valid scalene triangle • Test case for a valid equilateral triangle • Three test cases for valid isosceles triangles (a=b, b=c, a=c) • One, two or three sides has zero value (5 cases) • One side has a negative • Sum of two numbers equals the third (e.g. 1,2,3) is invalid b/c not a triangle (tried with 3 permutations a+b=c, a+c=b, b+c=a) • Sum of two numbers is less than the third (e.g. 1,2,4) (3 permutations) • Non-integer • Wrong number of values (too many, too few) Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 63
  64. 64. Additional Basic Exercises Analyze some common dialog boxes, such as these in MS Word: • Print dialog • Page setup dialog • Font format dialog For each dialog • Identify each field, and for each field » Name the type of the field (integer, rational, string, etc.) » List the range of entries that are “valid” for the field » Partition the field and identify boundary conditions » List the entries that are almost too extreme and too extreme for the field » List a small number of test cases for the field and explain why the values you have chosen are best representatives of the sets of values that they were selected from. » Identify any constraints imposed on this field by other fields Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 64
  65. 65. Domain Testing: Interesting Papers • Thomas Ostrand & Mark Balcer, The Category-partition Method For Specifying And Generating Functional Tests, Communications of the ACM, Vol. 31, No. 6, 1988. • Debra Richardson, et al., A Close Look at Domain Testing, IEEE Transactions On Software Engineering, Vol. SE-8, NO. 4, July 1982 • Michael Deck and James Whittaker, Lessons learned from fifteen years of cleanroom testing. STAR '97 Proceedings (in this paper, the authors adopt boundary testing as an adjunct to random sampling.) • Richard Hamlet & Ross Taylor, Partition Testing Does Not Inspire Confidence, Proceedings of the Second Workshop on Software Testing, Verification, and Analysis, IEEE Computer Society Press, 206-215, July 1988 Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 65
  66. 66. A Three-Variable Boundary Example I, J, and K are integers. The program calculates K = I*J. For this question, consider only cases in which you enter integer values into I and J. Do an equivalence class analysis from the point of view of the effects of I and J (jointly) on the variable K. Identify the boundary tests that you would run (the values you would enter into I and J) if • I, J, K are unsigned integers • I,J, K are signed integers Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 66
  67. 67. Interaction example K can run from MinInt (smallest integer) to MaxInt. For any value of K, there is a set of values of I and J that will yield K • The set is the null set for impossible values of K • The set might include only two pairs, such as K = MaxInt, when MaxInt is prime (7 could be a MaxInt) » (I,J) Є {(1, MaxInt), (MaxInt, 1)} • The set might include a huge number of pairs, such as when K is 0: » (I,J) Є {(0,0), (1, 0), (2, 0), … (MaxInt,0), (0,1),…,(0, MaxInt,1)} • A set of pairs of values of I and J can be thought of as an equivalence set (they all yield the same value of K) and so we ask which values of K are interesting (partition number 1) and then which values of I and J would produce those K-values, and do a partitioning on the sets of (I,J) pairs to find best representatives of those. • As practitioners, we do this type of analysis often, but many of us probably don’t think very formally about it. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 67
  68. 68. Fuzzy Boundaries In theory, the key to partitioning is dividing the space into mutually exclusive ranges (subsets). Each subset is an equivalence class. This is nice in theory, but let’s look at printers. Problem: • There are about 2000 Windows-compatible printers, plus multiple drivers for each. We can’t test them all. But maybe we can form equivalence classes. An example from two programs (desktop publishing and an address book) developed in 1991-92. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 68
  69. 69. Fuzzy Boundaries Primary groups of printers at that time: • HP - Original • HP - LJ II • PostScript Level I • PostScript Level II • Epson 9-pin, etc. LaserJet II compatible printers, huge class (maybe 300 printers, depending on how we define it) 1. Should the class include LJII, LJII+, and LIIP, LJIID- compatible subclasses? 2. What is the best representative of the class? Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 69
  70. 70. Fuzzy Boundaries Example: graphic complexity error handling • HP II original was the weak case. Example: special forms • HP II original was strong in paper-handling. We worked with printers that were weaker in paper-handling. Examples of additional queries for almost-equivalent printers • Same margins, offsets on new printer as on HP II original? • Same printable area? • Same handling of hairlines? (Postscript printers differ.) What exercises can we do to support development of this type of analysis? Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 70
  71. 71. Partitioning Device compatibility testing illustrates a multidimensional space with imperfect divisions between classes and with several different failure risks. From an equivalence class of “LaserJet II compatibles” you get several different, uniquely powerful, class representatives. The key to success is to remember that PARTITIONING IS MERELY A SAMPLING STRATEGY. The goal is to work from a rational basis in order to select a few valuable representatives from a much larger population of potential tests. A strong sampling strategy rests on our knowledge of the world, not just of the specification. If you can think of different ways that the program can fail in its interaction with a device (such as a printer), then FOR EACH TYPE OF ERROR, you look for the specific device (model, version of printer) that is most likely to confound the program. We might miss testing a key device because we didn’t think through all of the failure modes we were looking for. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 71
  72. 72. Partitioning The problem (as laid out by a client): “Equivalence class methods would be very valuable in situations where someone wants every OEM system, every sound and video card, every operating system, multiple technologies (e.g., DirectX 3 and 5), etc. “How can the engineer be confident in developing an equivalence matrix that provides good coverage?” The disappointing but very real answer: “Despite good analysis and execution, we might simply miss a device or driver or a combination of devices and drivers that has a bug associated with it. If we convince a stakeholder that equivalence class analysis provides “complete” coverage, she will be justifiably upset if our ‘complete coverage’ misses a bug.” Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 72
  73. 73. Equivalence Classes: A Broad Concept The notion of equivalence class is much broader than numeric ranges. Here are some examples: • Membership in a common group » such as employees vs. non-employees. (Note that not all classes have shared boundaries.) • Equivalent hardware » such as compatible modems • Equivalent event times » such as before-timeout and after • Equivalent output events » perhaps any report will do to answer a simple the question: Will the program print reports? • Equivalent operating environments » such as French & English versions of Windows 3.1 Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 73
  74. 74. Variables Suited to Equivalence Class Analysis There are many types of variables, such as: input variables output variables internal variables hardware and system software configurations, and equipment states. Any of these can be subject to equivalence class analysis. Here are some examples: Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 74
  75. 75. Variables Well Suited to Equivalence Class Analysis There are many types of variables, including input variables, output variables, internal variables, hardware and system software configurations, and equipment states. Any of these can be subject to equivalence class analysis. Here are some examples: ranges of numbers size of the sum of character codes variables, or of some other how many times something computed value (think is done binary and think digits) (e.g. shareware limit on size of a number that you number of uses of a product) enter (number of digits) or size of a character string (e.g. how many times you can do it before you run out size of a concatenated of memory) string how many names in a size of a path specification mailing list, records in a size of a file name database, variables in a size (in characters) of a spreadsheet, bookmarks, document abbreviations Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 75
  76. 76. Variables Well Suited to Equivalence Class Analysis size of a file (note special amount of available memory values such as exactly 64K, (> 128 meg, > 640K, etc.) exactly 512 bytes, etc.) visual resolution, size of size of the document on the screen, number of colors page (compared to page operating system version margins) (across different variations within a group of page margins, page sizes) “compatible” printers, size of a document on a sound cards, modems, etc. page, in terms of the equivalent event times memory requirements for (when something happens) the page. This might just be in terms of resolution x timing: how long between page size, but it may be event A and event B (and in more complex if we have which order--races) compression. • length of time after a timeout equivalent output events (from JUST before to way (such as printing after) -- what events are documents) important? Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 76
  77. 77. Variables Well Suited to Equivalence Class Analysis speed of data entry (time • transitions between between keystrokes, algorithms (optimizations) menus, etc.) (different ways to compute a • speed of input--handling of function) concurrent events • most recent event, first • number of devices event connected / active • input or output intensity • system resources (voltage) consumed / available (also, • speed / extent of voltage handles, stack space, etc.) transition (e.g. from very date and time soft to very loud sound) Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 77
  78. 78. Closing Notes on Boundary Analysis • In the traditional approach, we analyzed the code from the point of view of the programmer and the specification. • The specification provides us with only one of several classes of risk. We have to test against a broader set. • Experts in different subject matters will see different opportunities for failure, and different classes of cases that might reveal these failures. • Thinking outside of the box posed by the explicit design is at the essence of black box testing. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 78
  79. 79. Paradigms of Black Box Software Testing Reusable Test Matrices (and Domain Testing) Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 79
  80. 80. Using Test Matrices for Routine Issues After testing a simple numeric input field a few times, you’ve learned the drill. The boundary chart is reasonably easy to fill out for this, but it wastes your time. Use a test matrix to show/track a series of test cases that are essentially the same. • For example, for most input fields, you’ll do a series of the same tests, checking how the field handles boundaries, unexpected characters, function keys, etc. • As another example, for most files, you’ll run essentially the same tests on file handling. The matrix is a concise way of showing the repeating tests. • Put the objects that you’re testing on the rows. • Show the tests on the columns. • Check off the tests that you actually completed in the cells. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 80
  81. 81. Typical Uses of Test Matrices • You can often re-use a matrix like this across products and projects. • You can create matrices like this for a wide range of problems. Whenever you can specify multiple tests to be done on one class of object, and you expect to test several such objects, you can put the multiple tests on the matrix. • Mark a cell green if you ran the test and the program passed it. • Mark the cell red if the program failed and write the bug number of the bug report for this bug. • Write (in the cell) the automation number or identifier or file name if the test case has been automated. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 81
  82. 82. Matrices Problems? • What if your thinking gets out of date? (What if this program poses new issues, not covered by the standard tests?) • Do you need to execute every test every time? (or ever?) • What if the automation ID number changes? -- We still have a maintenance problem but it is not as obscure. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 82
  83. 83. Reusable Test Matrix Numeric (Integer) Input Field number of digits or Empty field (clear the default value) UB of value + 1 digits or chars digits or chars LB of value - 1 UB number of Outside of UB LB number of UB of value LB of value Non-digits Negative Nothing chars 0 This includes only the first few columns of a matrix that I’ve used commercially, but it gets across the idea. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 83
  84. 84. Examples of integer-input tests Nothing Outside of UB number of digits or Valid value chars At LB of value Non-digits At UB of value Wrong data type (e.g. decimal into At LB of value - 1 integer) At UB of value + 1 Expressions Outside of LB of value Space Outside of UB of value Non-printing char (e.g., Ctrl+char) 0 DOS filename reserved chars (e.g., " * . :") Negative Upper ASCII (128-254) At LB number of digits or chars Upper case chars At UB number of digits or chars Lower case chars Empty field (clear the default value) Modifiers (e.g., Ctrl, Alt, Shift-Ctrl, etc.) Function key (F2, F3, F4, etc.) Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 84
  85. 85. Error Handling when Writing a File full local disk full network disk almost full local disk almost full network disk write protected local disk write protected network disk damaged (I/O error) local disk damaged (I/O error) network disk unformatted local disk remove network disk after opening remove local disk from drive after file opening file timeout waiting for network disk timeout waiting for local disk to keyboard / mouse I/O during save to come back online network disk keyboard and mouse I/O during save other interrupt during save to to local disk network drive other interrupt during save to local local power out during save to drive network power out during save to local drive network power during save to network Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 85
  86. 86. Paradigms of Black Box Software Testing Specification-Based Testing Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 86
  87. 87. Specification Tasks • review specifications for » Ambiguity » Adequacy (it covers the issues) » Correctness (it describes the program) » Content (not a source of design errors) » Testability support • Create traceability matrices • Document management (spec versions, file comparison utilities for comparing two spec versions, etc.) • Participate in review meetings Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 87
  88. 88. Specification Skills • Understand the level of generality called for when testing a spec item. For example, imagine a field X: » We could test a single use of X » Or we could partition possible values of X and test boundary values » Or we could test X in various scenarios » Which is the right one? • Ambiguity analysis » Richard Bender teaches this well. If you can’t take his course, you can find notes based on his work in Rodney Wilson’s Software RX: Secrets of Engineering Quality Software Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 88
  89. 89. Specification Skills • Ambiguity analysis » Another book provides an excellent introduction to the ways in which statements can be ambiguous and provides lots of sample exercises: Cecile Cyrul Spector, Saying One Thing, Meaning Another : Activities for Clarifying Ambiguous Language Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 89
  90. 90. Breaking Statements into Elements Make / read a statement about the program Work through the statement one word at a time, asking what each word means or implies. • Thinkertoys describes this as “slice and dice.” • Gause & Weinberg develop related approaches as » “Mary had a little lamb” (read the statement several times, emphasizing a different word each time and asking what the statement means, read that way) » “Mary conned the trader” (for each word in the statement, substitute a wide range of synonyms and review the resulting meaning of the statement.) » These approaches can help you ferret out ambiguity in the definition of the product. By seeing how different people could interpret a key statement (e.g. spec statement that defines part of the product), you can see new test cases to check which meaning is operative in the program. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 90
  91. 91. Breaking Statements into Elements: An Example Quality is value to some person • Quality » » » • Value » » » • Some » • Who is this person? » • How are you the agent for this person? » • How are you going to find out what this person wants? • Person • How will you report results back to this person? » • How will you take action if this person is mentally absent? Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 91
  92. 92. Reviewing a Specification for Completeness Reading a spec linearly is not a particularly effective way to read the document. It’s too easy to overlook key missing issues. We don’t have time to walk through this method in this class, but the general approach that I use is based on James Bach’s “Satisfice Heuristic Test Strategy Model” at http://www.satisfice.com/tools/satisfice-tsm-4p.pdf. • You can assume (not always correctly, but usually) that every sentence in the spec is meant to convey information. • The information will probably be about » the project and how it is structured, funded or timed, or » about the product (what it is and how it works) or » about the quality criteria that you should evaluate the product against. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 92
  93. 93. Reviewing a Specification for Completeness Spec Review using the Satisfice Model, continued • The Satisfice Model lists several examples of project factors, product elements and quality criteria. • For a given sentence in the spec, ask whether it is telling you project, product, or quality-related information. Then ask whether you are getting the full story. As you do the review, you’ll discover that project factors are missing (such as deadline dates, location of key files, etc.) or that you don’t understand / recognize certain product elements, or that you don’t know how to tell whether the program will satisfy a given quality criterion. • Write down these issues. These are primary material for asking the programmer or product manager about the spec. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 93
  94. 94. Getting Information When There Is No Specification (Suggestions from some brainstorming sessions.) • Whatever specs exist • Marketing presentations, selling • Software change memos that the concept of the product to come with each new internal management version of the program • Bug reports (responses to them) • User manual draft (and previous • Reverse engineer the program. version’s manual) • Interview people, such as • Product literature •development lead • Published style guide and UI •tech writer standards •customer service • Published standards (such as C- •subject matter experts language) •project manager • 3rd party product compatibility • Look at header files, source code, test suites database table definitions • Published regulations • Specs and bug lists for all 3rd • Internal memos (e.g. project mgr. party tools that you use to engineers, describing the • Prototypes, and lab notes on the feature definitions) prototypes Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 94
  95. 95. Getting Information When There Is No Specification • Interview development staff from • Localization guide (probably one the last version. that is published, for localizing • Look at customer call records products on your platform.) from the previous version. What • Get lists of compatible equipment bugs were found in the field? and environments from Marketing • Usability test results (in theory, at least.) • Beta test results • Look at compatible products, to • Ziff-Davis SOS CD and other find their failures (then look for tech support CD’s, for bugs in these in your product), how they your product and common bugs designed features that you don’t in your niche or on your platform understand, and how they explain • BugNet magazine / web site for their design. See listserv’s, common bugs NEWS, BugNet, etc. • News Groups, CompuServe • Exact comparisons with products Fora, etc., looking for reports of you emulate bugs in your product and other • Content reference materials (e.g. products, and for discussions of an atlas to check your on-line how some features are geography program) supposed (by some) to work. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 95
  96. 96. Paradigms of Black Box Software Testing Risk-Based Testing Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 96
  97. 97. Equivalence and Risk Our working definition of equivalence: Two test cases are equivalent if you expect the same result from each. This is fundamentally subjective. It depends on what you expect. And what you expect depends on what errors you can anticipate: Two test cases can only be equivalent by reference to a specifiable risk. Two different testers will have different theories about how programs can fail, and therefore they will come up with different classes. A boundary case in this system is a “best representative.” A best representative of an equivalence class is a test that is at least as likely to expose a fault as every other member of the class. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 97
  98. 98. Evaluating Risk • Several approaches that call themselves “risk- based testing” ask which tests we should run and which we should skip if we run out of time. • These seem more like risk-based test management. I’d rather focus on test design. • If a key purpose of testing is to find defects, a key strategy for testing should be defect-based. Every test should be questioned: » How will this test find a defect? » What kind of defect do you have in mind? » What power does this test have against that kind of defect? Is there a more powerful test? A more powerful suite of tests? Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 98
  99. 99. Risk-Based Testing Many of us who think about testing in terms of risk, analogize testing of software to the testing of theories: • Karl Popper, in his famous essay Conjectures and Refutations, lays out the proposition that a scientific theory gains credibility by being subjected to (and passing) harsh tests that are intended to refute the theory. • We can gain confidence in a program by testing it harshly (if it passes the tests). • Subjecting a program to easy tests doesn’t tell us much about what will happen to the program in the field. In risk-based testing, we create harsh tests for vulnerable areas of the program. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 99
  100. 100. Risk-Based Testing Tasks • Identify risk factors (hazards: ways in which the program could go wrong) • For each risk factor, create tests that have power against it. • Assess coverage of the testing effort program, given a set of risk-based tests. Find holes in the testing effort. • Build lists of bug histories, configuration problems, tech support requests and obvious customer confusions. • Evaluate a series of tests to determine what risk they are testing for and whether more powerful variants can be created. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 100
  101. 101. Risk-Based Testing: Definitions • Hazard: » A dangerous condition (something that could trigger an accident) • Risk: » Possibility of suffering loss or harm (probability of an accident caused by a given hazard). • Accident: » A hazard is encountered, resulting in loss or harm. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 101
  102. 102. Risks: Where to look for errors New things: newer features may fail. New technology: new concepts lead to new mistakes. Learning Curve: mistakes due to ignorance. Changed things: changes may break old code. Late change: rushed decisions, rushed or demoralized staff lead to mistakes. Rushed work: some tasks or projects are chronically underfunded and all aspects of work quality suffer. Tired programmers: long overtime over several weeks or months yields inefficiencies and errors Adapted from James Bach’s lecture notes Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 102
  103. 103. Risks: Where to look for errors Other staff issues: alcoholic, mother died, two programmers who won’t talk to each other (neither will their code)… Just slipping it in: pet feature not on plan may interact badly with other code. N.I.H.: external components can cause problems. N.I.B.: (not in budget) Unbudgeted tasks may be done shoddily. Ambiguity: ambiguous descriptions (in specs or other docs) can lead to incorrect or conflicting implementations. Adapted from James Bach’s lecture notes Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 103
  104. 104. Risks: Where to look for errors Conflicting requirements: ambiguity often hides conflict, result is loss of value for some person. Unknown requirements: requirements surface throughout development. Failure to meet a legitimate requirement is a failure of quality for that stakeholder. Evolving requirements: people realize what they want as the product develops. Adhering to a start-of-the-project requirements list may meet contract but fail product. (check out http//www.agilealliance.org/) Complexity: complex code may be buggy. Bugginess: features with many known bugs may also have many unknown bugs. Adapted from James Bach’s lecture notes Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 104
  105. 105. Risks: Where to look for errors Dependencies: failures may trigger other failures. Untestability: risk of slow, inefficient testing. Little unit testing: programmers find and fix most of their own bugs. Shortcutting here is a risk. Little system testing so far: untested software may fail. Previous reliance on narrow testing strategies: (e.g. regression, function tests), can yield a backlog of errors surviving across versions. Weak testing tools: if tools don’t exist to help identify / isolate a class of error (e.g. wild pointers), the error is more likely to survive to testing and beyond. Adapted from James Bach’s lecture notes Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 105
  106. 106. Risks: Where to look for errors Unfixability: risk of not being able to fix a bug. Language-typical errors: such as wild pointers in C. See • Bruce Webster, Pitfalls of Object-Oriented Development • Michael Daconta et al. Java Pitfalls Criticality: severity of failure of very important features. Popularity: likelihood or consequence if much used features fail. Market: severity of failure of key differentiating features. Bad publicity: a bug may appear in PC Week. Liability: being sued. Adapted from James Bach’s lecture notes Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 106
  107. 107. Risks: Where to look for errors Quality Categories: • Capability • Reliability Each quality category is a • Usability risk category, as in: • Performance “the risk of unreliability.” • Installability • Compatibility – Efficiency • Supportability – Localizability • Testability – Portability – Maintainability Derived from James Bach’s Satisfice Model Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 107
  108. 108. Bug Patterns as a Source of Risk Testing Computer Software lays out a set of 480 common defects. You can use these or develop your own list. • Find a defect in the list •Ask whether the software under test could have this defect • If it is theoretically possible that the program could have the defect, ask how you could find the bug if it was there. • Ask how plausible it is that this bug could be in the program and how serious the failure would be if it was there. • If appropriate, design a test or series of tests for bugs of this type. For an up to date analysis along these lines, see Giri Vijayaraghavan’s paper in this conference. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 108
  109. 109. Build Your Own Model of Bug Patterns Too many people start and end with the TCS bug list. It is outdated. It was outdated the day it was published. And it doesn’t cover the issues in your system. Building a bug list is an ongoing process that constantly pays for itself. Here’s an example from Hung Nguyen: • This problem came up in a client/server system. The system sends the client a list of names, to allow verification that a name the client enters is not new. • Client 1 and 2 both want to enter a name and client 1 and 2 both use the same new name. Both instances of the name are new relative to their local compare list and therefore, they are accepted, and we now have two instances of the same name. • As we see these, we develop a library of issues. The discovery method is exploratory, requires sophistication with the underlying technology. • Capture winning themes for testing in charts or in scripts-on-their-way to being automated. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 109
  110. 110. Risk-Based Testing Exercises • Given a list of ways that the program could fail, for each case: » Describe two ways to test for that possible failure » Explain how to make your tests more powerful against that type of possible failure » Explain why your test is powerful against that hazard. • Given a list of test cases » Identify a hazard that the test case might have power against » Explain why this test is powerful against that hazard. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 110
  111. 111. Risk-Based Testing Exercises Take a function in the software under test. Make a hypothetical change to the product. Do a risk analysis: • For N (10 to 30) minutes, the students privately list risk factors and associated tests (ask for 2 tests per factor) • Break • Class brainstorm onto flipcharts--list risk factors. • Pairs project--fill out the charts: » Students work in pairs to complete their chart » Each chart has one risk factor as a title » List tests derived from risk (factor) on body of page » When a pair is done with its page, they move around the room, adding notes to other pairs’ pages. • Paired testing, one chart per pair, test SUT per that risk factor. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 111
  112. 112. Risk-Based Testing Exercises Collect or create some test cases for the software under test. Make a variety of tests: • Mainstream tests that use the program in “safe” ways • Boundary tests • Scenario tests • Wandering walks through the program • Etc. • If possible, use tests the students have suggested previously. For each test, ask: • How will this test find a defect? • What kind of defect did the test author probably have in mind? • What power does this test have against that kind of defect? Is there a more powerful test? A more powerful suite of tests? Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 112
  113. 113. Risk-Based Test Management Project risk management involves • Identification of the different risks to the project (issues that might cause the project to fail or to fall behind schedule or to cost too much or to dissatisfy customers or other stakeholders) • Analysis of the potential costs associated with each risk • Development of plans and actions to reduce the likelihood of the risk or the magnitude of the harm • Continuous assessment or monitoring of the risks (or the actions taken to manage them) Useful material available free at http://seir.sei.cmu.edu http://www.coyotevalley.com (Brian Lawrence) Good paper by Stale Amland, Risk Based Testing and Metrics, 16th International Conference on Testing Computer Software, 1999. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 113
  114. 114. Risk-Driven Testing Cycle Analyze & Perform Prioritize Appropriate Hazards Tests Improve Report & Risk Analysis Resolve Process Analyze Problems Failures: Reassess Risks post-ship pre-ship Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 114
  115. 115. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 115
  116. 116. Risk-Based Test Management Tasks • List all areas of the program that could require testing • On a scale of 1-5, assign a probability-of-failure estimate to each • On a scale of 1-5, assign a severity-of-failure estimate to each • For each area, identify the specific ways that the program might fail and assign probability-of-failure and severity-of-failure estimates for those • Prioritize based on estimated risk • Develop a stop-loss strategy for testing untested or lightly-tested areas, to check whether there is easy-to-find evidence that the areas estimated as low risk are not actually low risk. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 116
  117. 117. Risk-Based Testing: Some Papers of Interest • Stale Amland, Risk Based Testing • James Bach, Reframing Requirements Analysis • James Bach, Risk and Requirements- Based Testing • James Bach, James Bach on Risk-Based Testing • Stale Amland & Hans Schaefer, Risk based testing, a response • Carl Popper, Conjectures & Refutations Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 117
  118. 118. Black Box Software Testing Paradigms: Regression Testing Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 118
  119. 119. Automating Regression Testing This is the most commonly discussed automation approach: • create a test case • run it and inspect the output • if the program fails, report a bug and try again later • if the program passes the test, save the resulting outputs • in future tests, run the program and compare the output to the saved results. Report an exception whenever the current output and the saved output don’t match. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 119
  120. 120. Potential Regression Advantages • Dominant paradigm for automated testing. • Straightforward • Same approach for all tests • Relatively fast implementation • Variations may be easy • Repeatable tests Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 120
  121. 121. The GUI Regression Automation Problem Prone to failure because of difficult financing, architectural, and maintenance issues. Low power (in its traditional form) even if successful. Extremely valuable under some circumstances. THERE ARE MANY ALTERNATIVES THAT CAN BE MORE APPROPRIATE UNDER OTHER CIRCUMSTANCES. If your only tool is a hammer, everything looks like a nail. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 121
  122. 122. Testing Analogy: Clearing Mines mines This analogy was first presented by Brian Marick. These slides are from James Bach.. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 122
  123. 123. Totally Repeatable Tests Won’t Clear the Minefield mines fixes Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 123
  124. 124. Variable Tests are Often More Effective mines fixes Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 124
  125. 125. GUI Regression Strategies: Some Papers of Interest • Chris Agruss, Automating Software Installation Testing • James Bach, Test Automation Snake Oil • Hans Buwalda, Testing Using Action Words • Hans Buwalda, Automated testing with Action Words: Abandoning Record & Playback • Elisabeth Hendrickson, The Difference between Test Automation Failure and Success • Cem Kaner, Avoiding Shelfware: A Manager’s View of Automated GUI Testing • John Kent, Advanced Automated Testing Architectures • Bret Pettichord, Success with Test Automation • Bret Pettichord, Seven Steps to Test Automation Success • Keith Zambelich, Totally Data-Driven Automated Testing Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 125
  126. 126. Paradigms of Black Box Software Testing Exploratory Testing Many of the ideas in these notes were reviewed and extended by my colleagues at the 7th Los Altos Workshop on Software Testing. I appreciate the assistance of the other LAWST 7 attendees: Brian Lawrence, III, Jack Falk, Drew Pritsker, Jim Bampos, Bob Johnson, Doug Hoffman, Chris Agruss, Dave Gelperin, Melora Svoboda, Jeff Payne, James Tierney, Hung Nguyen, Harry Robinson, Elisabeth Hendrickson, Noel Nyman, Bret Pettichord, & Rodney Wilson. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 126
  127. 127. Doing Exploratory Testing • Keep your mission clearly in mind. • Distinguish between testing and observation. • While testing, be aware of the limits of your ability to detect problems. • Keep notes that help you report what you did, why you did it, and support your assessment of product quality. • Keep track of questions and issues raised in your exploration. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 127
  128. 128. Problems to be Wary of… • Habituation may cause you to miss problems. • Lack of information may impair exploration. • Expensive or difficult product setup may increase the cost of exploring. • Exploratory feedback loop my be too slow. • Old problems may pop up again and again. • High MTBF may not be achievable without well defined test cases and procedures, in addition to exploratory approach. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 128
  129. 129. Styles of Exploration Experienced, skilled explorers develop their own styles. When you watch or read different skilled explorers, you see very different approaches. This is a survey of the approaches that I’ve seen. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 129
  130. 130. Characterizing the Styles At the heart of all of the approaches to exploratory testing (all of the styles), I think we find questions and questioning skills. As you consider the styles that follow, try to characterize them with respect to each other: • All of the approaches are methodical, but do they focus on the » Method of questioning? » Method of describing or analyzing the product? » The details of the product? » The patterns of use of the product? » The environment in which the product is run? • To what extent would this style benefit from group interaction? Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 130

×