Upcoming SlideShare
×

# Black box-software-testing-douglas-hoffman2483

1,149 views
1,060 views

Published on

Published in: Education
0 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

Views
Total views
1,149
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
10
0
Likes
0
Embeds 0
No embeds

No notes for slide
• 3
• 3
• 3
• 26,614 Years to send all the combinations of positive case values – assuming no time for generation or comparing of expected results
• 8
• 52
• Was called Equivalence Class analysis Quantify huge possible test space; Dick Bender’s Softtool generates cause-effect graph based test generation from complete requirements his process said that 10 100 possible test cases in a financial application where Hawkings estimates there are 10 90 molecules in the universe! Segment the test universe into sub-domains of [likely] equivalent values, then select the one best representative of the class. Used Laser Jet II as best [worst] representative printer for the class; Trick in non-trivial cases is to select the best representative; see Jorgensen’s “Software Testing: A Craftsman’s Approach.” Also, Beizer’s “Black Box Software Testing” is OK.
• Equivalence classes depend on the risk that you’re testing against. Example is 64+64 looks equivalent to 64+63, but it is not equivalent if we’re dealing with single-byte sums. Hockey game example -- you can’t get into the playoffs with more than 63 games won (more than 127 points scored).
• 45 This illustrates the internal boundaries. Show chi-square as another example Walk through A,b as non-numeric tests. Explain the reversal error (&gt; 57 vs &gt;75, the program would appear to pass on b, but would fail on A)
• 45
• 41 with more than 63 games won (more than 127 points scored).
• 127 The notion of Binary State as a boundary Between algorithms = chi-square approximation numerosity number of usages number of records size sum digits entered time race conditions? timeout followed by , , , , speed of data entry? year 2000 12:00 midnight special cases 24:00 rolls over to 00:01 or 24:01 most recent, first space on page memory &lt; 640K on disk 64 K file system resources, between algorithms, date, intensity, what else?
• 127
• 127
• 125
• 8
• Example similar to real world architecture Testability: Visibility (how close to the point of failure can you look at information) Control (feed the program information, set the program state) [DuChard: Prob(failure|fault) should be high (exposure)] (Density of the error)
• Get a team of developers together and trace through typical activities. Diagram the elements as they are involved.
• War stories - database bug; it didn’t crash. This is the report-automation problem. The report test plan had been automated but no one knew what it did. It printed bad, but internally consistent, output. - vast wasted time interpreting old output - don’t read output
• 83
• 82
• 84
• 85
• 86 Define software increments Use rigorous methods for Spec, Dev, and Certification of each increment Apply strict statistical QC during testing Enforce strict separation of Specification and Design Tasks from Testing Activities
• 87
• 88 Concept central to SDLC and PM Left out of Primer and most texts
• Slide 9 Experience, education , and examination
• ### Black box-software-testing-douglas-hoffman2483

1. 1. Black Box Software Testing Douglas HoffmanB.A., M.S., M.B.A., ASQ-CSQE, ASQ-CQMgr, ASQ Fellow Consultant Software Quality Methods, LLC, (SQM) Winter, 2004 doug.hoffman@acm.org www.SoftwareQualityMethods.com Copyright © 1994-2004 SQM, LLC. All Rights Reserved.
3. 3. Contents18 595 Planning Testing Projects19 609 Metrics20 647 Defect Life Cycles21 657 Impossible Deadlines22 671 Career planning for testers23 679 Job Seeking24 707 Recruiting testers25 729 Learning styles26 741 Resources on the NetCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 7
5. 5. Demographics: How long have you worked in: • software testing 0-3 months ____ 3-6 months ____ • management 6 mo-1 year ____ 1-2 years ____ » Testing group _____ » Any management _____ 2-5 years ____ > 5 years ____ • marketing _____ • programming • documentation _____ » Any experience _____ » Production programming _____ • customer care _____ • test automation • traditional QC _____ » Test development _____ » Tools creation _____Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 9
6. 6. Customization Please vote for your top 10 topics Managing testing Test strategy planning Job Seeking groups Recruiting software Software life cycles Black box automation testers Testing user Role of test group documentation Test case design Metrics Bug advocacy Status reporting Specification analysis Defect life cycle Planning / scheduling Beta testing (in-house Exploratory testing the testing project courses only) Test planning Career paths for Bug tracking system (documentation) testers design (in-houseCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 10
7. 7. Themes of the Course The Themes 1. Management (not Testing) heads up QA. 2. Many development methods work. 3. Testing is done in context. 4. The program doesn’t work. Therefore, your task is to find errors. 5. Complete testing requires a nearly infinite series of tests. 6. Be methodical. 7. Look for powerful representatives of classes of tests. 8. Groups often develop a better range of insights than individuals. 9. Cost-justify your processes. 10. Think in terms of quality costs. 11. Automation is software development. 12. Communication must be effective. 13. Change is normal. 14. Test planning can be evolutionary.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 11
8. 8. Black Box Testing Defined Black Box Testing is based on externally observed characteristics of the program we are testing. The tests are designed without regard to how the program works internally. Glass Box Testing (a.k.a. White Box) draws on knowledge of the internal characteristics of the program. The tests are designed to exercise specific parts of the code using that knowledge.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 20
11. 11. Notes about the Triangle ProblemSeveral classes of issues were missed by most students. For example: • Few students checked whether they were producing valid triangles. (1,2,3) and (1,2,4) cannot be the lengths of any triangle. » Knowledge of the subject matter of the program under test will enable you to create test cases that are not directly suggested by the specification. If you lack that knowledge, you will miss key tests. (This knowledge is sometimes called “domain knowledge,” not to be confused with “domain testing.”) • Few students checked non-numeric values, bad delimiters, or non-integers. • The only boundaries tested were at MaxInt or 0. We will revisit this example in Domain TestingCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 24
13. 13. Oracles An oracle is the principle or mechanism by which you recognize problems. “..it works” Means: “...it appeared to meet some requirement to some degree.”Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 26
17. 17. Testing is about ideas. Heuristics give you ideas.• A heuristic is a fallible idea or method that may help solve a problem.• You don’t comply with a heuristic; you apply it. Heuristics can hurt you when elevated to the status of authoritative rules.• Heuristics represent wise behavior only in context. They do not contain wisdom.• Your relationship to a heuristic is the key to applying it wisely. “Heuristic reasoning is not regarded as final and strict but as provisional and plausible only, whose purpose is to discover the solution to the present problem.” - George Polya, How to Solve ItCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 30
18. 18. Some useful oracle heuristicsConsistent with History: Present function behavior is consistent withpast behavior.Consistent with our Image: Function behavior is consistent with animage that the organization wants to project.Consistent with Comparable Products: Function behavior isconsistent with that of similar functions in comparable products.Consistent with Claims: Function behavior is consistent withdocumented or advertised behavior.Consistent with Specifications or Regulations: Function behavioris consistent with claims that must be met.Consistent with User’s Expectations: Function behavior isconsistent with what we think users want.Consistent within Product: Function behavior is consistent withbehavior of comparable functions or functional patterns within the product.Consistent with Purpose: Function behavior is consistent withapparent purpose.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 31
19. 19. Some font size testing issues… What to cover? • Every font size (to a tenth of a point). • Every character in every font. • Every method of changing font size. • Every user interface element associated with font size functions. • Interactions between font sizes and other contents of a document. • Interactions between font sizes and every other feature in Wordpad. • Interactions between font sizes and graphics cards & modes. • Print vs. screen display. What’s your oracle? • What do you know about typography? • Definition of “point” varies. There are as many as six different definitions (http://www.oberonplace.com/dtp/fonts/point.htm) • Absolute size of characters can be measured, but not easily (http://www.oberonplace.com/dtp/fonts/fontsize.htm) • How closely must size match to whatever standard is chosen? • Heuristic approaches: relative size of characters; comparison to MS Word. • Expectations of different kinds of users for different uses.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 32
21. 21. Risk as a simplifying factorFor Wordpad, we don’t care if font size meetsprecise standards of typography!In general it can vastly simplify testing if we focuson whether the product has a problem that matters,rather than whether the product merely satisfies allrelevant standards.Effective testing requires that we understandstandards as they relate to how our clients valuethe product. Instead of thinking pass vs. fail, Consider thinking problem vs. no problem.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 34
22. 22. Risk as a simplifying factorWhat if we applied the same evaluation approach• that we applied to WordPad• to Open Office or MS Word or Adobe PageMaker? The same tests or results lead to different conclusions in different contextsCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 35
23. 23. The oracle problem and test automationWe often hear that all testing should be automated.Automated testing depends on our ability toprogrammatically detect when the software undertest fails a test. Our ability to automate testing is fundamentally constrained by our ability to create and use oracles.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 36
24. 24. The oracle problem and automationOne common way to define an oracle is as a sourceof expected results. • Under this view, you compare your results to those obtained from the oracle. If they match, the program passes. If they don’t match, the program fails. • The comparison between MS WordPad and MS Word illustrates this approach. • It is important to recognize that this evaluation is heuristic: » We can have false alarms: A mismatch between WordPad and Word might not matter. » We can miss bugs: A match between WordPad and Word might result from the same error in both programs.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 37
25. 25. Automated tests narrow the oracles scopeAn automated test is not equivalent to the mostsimilar manual test: • The mechanical comparison is typically more precise (and will be tripped by irrelevant discrepancies) • The skilled human comparison will sample a wider range of dimensions, noting oddities that one wouldnt program the computer to detect.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 38
26. 26. Black Box Software Testing Part 2. The Impossibility of Complete TestingSelected Readings: • Hoffman, Bug Report (Exhausting Your Test Options) • Kaner, The Impossibility of Complete Testing • Kaner, Software Negligence & Testing CoverageCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 39
27. 27. The Impossibility of Complete TestingIf you test completely, then at the end of testing,there cannot be any undiscovered errors. (After all,if there are more bugs, you can find them if you domore testing. So testing couldnt yet be"complete.”) This is impossible because: 1 The domain of possible inputs is too large. 2 There are too many combinations of data to test. 3 There are too many possible paths through the program to test. 4 And then there are the user interface errors, the configuration/ compatibility failures, and dozens of other dimensions of analysis.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 40
28. 28. 1. Too Many InputsThe domain of possible inputs is too large • Valid inputs » Many variables » Huge number of possible values » In some cases, complete testing makes sense. At MASPAR we made a Massively Parallel computer, up to 65K parallel processors. To test the 32-bit integer square root function, we checked all values (all 4,294,967,296 of them). This took the computer about 6 minutes to run the tests and compare the results to an oracle. There were 2 (two) errors, neither of them near any boundary. (The underlying error was that a bit was sometimes mis- set, but in most of these cases, there was no effect on the final calculated result.) Without an exhaustive test, these errors probably wouldn’t have shown up. But what about the 64-bit integer square root? How could we find the time to run all of these?Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 41
29. 29. 1. Too Many Inputs • Invalid inputs » Don’t forget Easter Eggs. • Edited inputs » These can be quite complex. How much editing is enough? • Variations on input timing » Enter data very quickly or very slowly. » Try testing one event just before, just after, and in the middle of processing a second event. • Now, what about all the error handling that you can trigger with "invalid" inputs? » Think about Whittaker & Jorgensens constraint-focused attacks (Whittaker, How Software Fails) » Think about Jorgensens hostile data stream attacksCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 42
30. 30. 1. Size Of The Testing Problem• Input one value in a 10 character field• 26 UC, 26 LC, 10 Numbers• Gives 6210 combinations• How long at 1,000,000 per second? What is your domain size? We can only run a vanishingly small portion of the possible testsCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 43
31. 31. 2. Too Many CombinationsVariables interact. • For example, a program crashes when attempting to print preview a high resolution (say, 600x600 dpi) image on a high resolution screen. The option selections for printer resolution and screen resolution are interacting. • For example, a program fails when the sum of a series of variables is too large.Suppose there are N variables. Suppose the number of choicesfor the variables are V1, V2, through VN. The total number ofpossible combinations is V1 x V2 x . . . x VN. This is huge.We saw 39,601 combinations of just two variables whose valuescould range only between -99 and 99.Here’s a case that isn’t so trivial. There are 318,979,564,000possible combinations of the first four moves in chess.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 44
32. 32. 3. Too Many Paths B A F D G C X EXIT H E < 20 times I through the loopHere’s an example that shows that there are too many paths to test ineven a fairly simple program. This one is from Myers, The Art ofSoftware Testing. Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 45
33. 33. 3. Too Many PathsThe program starts at A. From A it can go to B or C From B it goes to X From C it can go to D or E From D it can go to F or G From F or from G it goes to X From E it can go to H or I From H or from I it goes to X From X the program can go to EXIT or back to A. It can go back to A no more than 19 times.One path is ABX-Exit. There are 5 ways to get to X and then to the EXITin one pass.Another path is ABXACDFX-Exit. There are 5 ways to get to X the firsttime, 5 more to get back to X the second time, so there are 5 x 5 = 25cases like this. Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 46
34. 34. 3. Too Many Paths There are 51 + 52 + ... + 519 + 520 = 1014 = 100 trillion paths through the program to test, or approximately one billion years to try every path if one could write, execute and verify a test case every five minutes.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 47
35. 35. 3. Sequences – The Telephone Example Idle Caller Ringing hung up You Connected hung up On HoldCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 48
36. 36. 3. Sequences – The Telephone ExampleWhy are we spending so much time on this crazy example?Because it illustrates several important points:• Simplistic approaches to path testing can miss critical defects.• Critical defects can arise under circumstances that appear (in a test lab) so specialized that you would never intentionally test for them.• When (in some future course or book) you hear a new methodology for combination testing or path testing, I want you to test it against this defect. If you have no suspicion that there is a stack corruption problem in this program, will the new method lead you to find this bug?This example also lays a foundation for our introduction torandom/statistical testing.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 49
38. 38. Lets consider the nature of the infinite set of testsThere are enormous numbers of possible tests. To test everything, you would have to: • Test every possible input to every variable (including output variables and intermediate results variables). • Test every possible combination of inputs to every combination of variables. • Test every possible sequence through the program. • Test every hardware / software configuration, including configurations of servers not under your control. • Test every way in which any user might try to use the program. Read Testing Computer Software, p. 17 - 22Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 51
39. 39. Complete coverage?Some people (attempt to) simplify away the problem ofcomplete testing by advocating "complete coverage."What is coverage? • Extent of testing of certain attributes or pieces of the program, such as statement coverage or branch coverage or condition coverage. • Extent of testing completed, compared to a population of possible tests.Typical definitions are oversimplified. They miss, forexample, • Interrupts and other parallel operations • Interesting data values and data combinations • Missing codeThe number of variables we might measure is stunning. I(Kaner) listed 101 examples in Software Negligence &Testing Coverage. Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 52
40. 40. Expanded Black Box Testing Model Test Inputs Test Results Precondition Post-condition Data System Data Under Precondition Post-condition Program State Test Program State Environmental Environmental Inputs ResultsCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 53
41. 41. ConclusionComplete testing is impossible • There is no simple answer for this. • There is no simple, comprehensive oracle to deal with it. • Therefore testers live and breathe tradeoffs. Our job isn’t to design and do all the tests - it’s to choose and do the most important tests.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 54
42. 42. Black Box Software Testing Part 3. Domain TestingSelected Readings: • Cem Kaner, Liability for Product Incompatibility • Clarke, Hassell, & Richardson: A Close Look at Domain Testing, IEEE Transactions on Software Engineering, Vol. SE-8, No. 4, July 1982. • Ostrand & Balcer: The category-partition method for specifying and generating functional tests. Communications of the ACM, Vol. 31 (6), pages 676-686, June 1988. • Michael Deck and James Whittaker, Lessons learned from fifteen years of cleanroom testing. STAR 97 Proceedings (in this paper, the authors adopt boundary testing as an adjunct to random sampling.) • Richard Hamlet & Ross Taylor, Partition Testing Does Not Inspire Confidence, Proceedings of the Second Workshop on Software Testing, Verification, and Analysis, IEEE Computer Society Press, 206-215, July 1988Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 55
43. 43. Domain testing• AKA partitioning, equivalence analysis, boundary analysis• Fundamental question or goal: • This confronts the problem that there are too many test cases for anyone to run. This is a stratified sampling strategy that provides a rationale for selecting a few test cases from a huge population.• General approach: • Divide the set of possible values of a field into subsets, pick values to represent each subset. The goal is to find a “best representative” for each subset, and to run tests with these representatives. Best representatives of ordered fields will typically be boundary values. • Multiple variables: combine tests of several “best representatives” and find a defensible way to sample from the set of combinations.• Paradigmatic case(s) • Equivalence analysis of a simple numeric field. • Printer compatibility testing (multidimensional variable, doesn’t map to a simple numeric field, but stratified sampling is essential.)Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 56
44. 44. Example Test SeriesHere is the program’s specification: • This program is designed to add two numbers, which you will enter • Each number should be one or two digits • The program will print the sum. Press Enter after each number • To start the program, type ADDERBefore you start testing, do you have any questionsabout the spec? Refer to Testing Computer Software, Chapter 1, page 1Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 57
45. 45. The First Cycles of TestingHere’s a basic strategy for dealing with new code:1 Start with mainstream-level tests. Test the program with easy-to-pass values that will be taken as serious issues if the program fails.2 Test each function sympathetically. Learn why this feature is valuable before you criticize it.3 Test broadly, rather than deeply. Check out all parts of the program quickly before focusing.4 Look for more powerful tests. Once the program can survive the easy tests, put on your thinking cap and look systematically for challenges.5 Pick boundary conditions. There will be too many good tests. You need a strategy for picking and choosing.6 Do some exploratory testing. Run new tests every week, from the first week to the last week of the project. Refer to Testing Computer Software, Chapter 1 Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 58
46. 46. 1. A Mainstream Level Test For the first test, try a pair of easy values, such as 3 plus 7. Here is the screen display that results from that test. Are there any bug reports that you would file from this? ?3 ?7 10 ?_ Refer to Testing Computer Software, Chapter 1Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 59
47. 47. 2. Test each function sympathetically• Why is this function here?• What will the customer want to do with it?• What is it about this function that, once it is working, will make the customer happy? Knowing what the customer will want to do with the feature gives you a much stronger context for discovering and explaining what is wrong with the function, or with the functions interaction with the rest of the program.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 60
48. 48. 3. Test broadly before deeply • The objective of early testing is to flush out the big problems as quickly as possible. • You will explore the program in more depth as it gets more stable. • There is no point hammering a design into oblivion if it is going to change. Report as many problems as you think it will take to force a change, and then move on.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 61
49. 49. 4. Look for More Powerful Tests More likely to encounter and recognize errors : • Do more iterations (more classes per function). • Do more combinations (interactions between functions). • Do more things (functional breadth). • Methodically cover the code (look everywhere). • Look for specific errors (similar products’ problems). • Check for more symptoms (more types of errors). • Try to break it (take a perverse view, get creative).Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 62
50. 50. 4a. Look for More Powerful Tests Brainstorming Rules: • The goal is to get lots of ideas. You are brainstorming together to discover categories of possible tests. • There are more great ideas out there than you think. • Don’t criticize others’ contributions. • Jokes are OK, and are often valuable. • Work later, alone or in a much smaller group, to eliminate redundancy, cut bad ideas, and refine and optimize the specific tests. • Facilitator and recorder keep their opinions to themselves.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 63
51. 51. 4. Look for More Powerful Tests What? Why? ______________ ________________ ______________ ________________ ______________ ________________ ______________ ________________ ______________ ________________ ______________ ________________ ______________ ________________ ______________ ________________ ______________ ________________ ______________ ________________ Refer to Testing Computer Software, pages 4-5, for examples.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 64
52. 52. 5. Equivalence Class & Boundary Analysis There are 199 x 199 = 39,601 test cases for valid values: • 99 values: 1 to 99 • 1 value: 0 • 99 values: -99 to –1 __________ 199 values per variable 199 x 199 = 39,601 possible tests So should we test them all? We tested 3 + 7. Should we also test • 4 + 7? 4 + 6? • 2 + 7? 2 + 8? • 3 + 8? 3 + 6? Why? Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 65
53. 53. 5. Reducing the testing burden There are too many possible tests. There are 199 x 199 = 39,601 test cases for valid values: There are infinitely many invalid cases: • 100 and above • -100 and below • anything non-numeric We cannot afford to run every possible test. We need a method for choosing a few tests that will represent the rest. Equivalence analysis is the most widely used approach.Copyright © 1994-2004 Software Quality Methods, LLC. pages 4-5 andReserved. 66 • refer to Testing Computer Software All Rights 125-132
54. 54. 5. Classical Equivalence Class and Boundary (Domain) Analysis • To avoid unnecessary testing, partition (divide) the range of inputs into groups of equivalent values. • Then treat any input value from the equivalence class as representative of the full group. • Two tests are equivalent if we would expect that an error revealed by one would be revealed by the other. • If you can map the input space to a number line, then boundaries mark the point or zone of transition from one equivalence class to another. These are good members of equivalence classes to use since a program is more likely to fail at a boundary. Myers, Art of Software Testing, 45Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 67
55. 55. 5. Traditional Approach (Myers) One input or output field • The “valid” values for the field fit within one (1) easily specified range. • Valid values: -99 to 99 • Invalid values » < -99 » > 99 » non-numeric » expressions » long delays between digits » too many, too few valuesCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 68
56. 56. 5. Myers’ boundary table Variable Valid Case Invalid Case Boundaries Notes Equivalence Equivalence and Special Classes Classes Cases First -99 to 99 > 99 99, 100 number < -99 -99, -100 null entry non-integer 2.5 Second same as first same as first same number The traditional analysis would look at the potential numeric entries and partition them the way the specification would partition them.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 69
57. 57. 5. Myers’ boundary table (continued) Variable Valid Case Invalid Case Boundaries Notes Equivalence Equivalence and Special Classes Classes Cases First -99 to 99 > 99 99, 100 number < -99 -99, -100 null entry 0 non-integer 2.5 non-number / : expressions Second same as first same as first same number It might be useful to consider some other cases, such as special cases that are inside the range (e.g. 0) and errors on a different dimension from your basic "too big" and "too small".Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 70
58. 58. 5. Myers’ boundary table Variable Valid Case Invalid Case Boundaries Notes Equivalence Equivalence and Special Classes Classes Cases First -99 to 99 > 99 99, 100 number < -99 -99, -100 non-number / expressions : 0 null entry Second same as first same as first same number Sum -198 to 198 Are there other sources of data for this variable? Ways to feed it bad data? We should also consider other variables, not just inputs. For example, think of output variables, interim results, variables as things that are stored, and variables as inputs to a subsequent process. ---See Whittaker, How to Break Software.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 71
59. 59. Boundary table as a test plan component • Makes the reasoning obvious. • Makes the relationships between test cases fairly obvious. • Expected results are pretty obvious. • Several tests on one page. • Can delegate it and have tester check off what was done. Provides some limited opportunity for tracking. • Not much room for status. ----------------------------------------Question, now that we have the table, must we doall the tests? What about doing them all each time(each cycle of testing)?Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 72
60. 60. 5. Building the Table (in practice) • Relatively few programs will come to you with all fields fully specified. Therefore, you should expect to learn what variables exist and their definitions over time. • To build an equivalence class analysis over time, put the information into a spreadsheet. Start by listing variables. Add information about them as you obtain it. • The table should eventually contain all variables. This means, all input variables, all output variables, and any intermediate variables that you can observe. • In practice, most tables that I’ve seen are incomplete. The best ones that I’ve seen list all the variables and add detail for critical variables.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 73
61. 61. 6. Exploratory TestingExploratory testing involves simultaneously learning,planning, running tests, and reporting/troubleshooting results.A common goal of exploration is to probe for weak areas ofthe program.Proportions do (and should) vary across the industry. Here ismy preference per week: • 25% of the group’s time developing new tests • 50% executing old tests (including bug regression) • 25% on exploratory testingI recommend running new tests even in the very last daysbefore releasing the product.Different testers do more or less well at exploratory testing.Don’t assign 25/50/25 to everyone. Some people should doalmost no exploration. Others should do it nearly full-time.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 74
63. 63. Triangle Problem, Revisited The program reads three integer values from a card. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral. » From Glen Myers, The Art of Software Testing Write a set of test cases that would adequately test this program. This is a classic example of domain partitioning.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 76
64. 64. Myers’ Answer 1. Test case for a valid scalene triangle 2. Test case for a valid equilateral triangle 3. Three test cases for valid isosceles triangles (a=b, b=c, a=c) 4. One, two or three sides are zero length (7 cases) 5. One side has a negative length 6. Sum of two sides equals the third (e.g. 1,2,3) is invalid because it’s not a triangle (3 permutations a+b=c, a+c=b, b+c=a) 7. Sum of two numbers is less than the third (e.g. 1,2,4) (3 permutations) 8. Non-integer value 9. Wrong number of values (too many, too few)Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 77
65. 65. Example Values1. {5,6,7}2. {15,15,15}3. {3,3,4; 5,6,6; 7,8,7}4. {0,1,1; 2,0,2; 3,2,0; 0,0,9; 0,8,0; 11,0,0; 0,0,0}5. {3,4,-6}6. {1,2,3; 2,5,3; 7,4,3}7. {1,2,4; 2,6,2; 8,4,2}8. {Q,2,3}9. {2,4; 4,5,5,6}Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 78
67. 67. ExerciseFor each of the following, list • The variable(s) of interest • The valid and invalid classes • The boundary value test cases.1. FoodVan delivers groceries to customers who order foodover the Net. To decide whether to buy more more vans, FVtracks the number of customers who call for a van. A clerkenters the number of calls into a database each day. Based onprevious experience, the database is set to challenge (ask,“Are you sure?”) any number greater than 400 calls.2. FoodVan schedules drivers one day in advance. To beeligible for an assignment, a driver must have specialpermission or she must have driven within 30 days of the shiftshe will be assigned to.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 80
68. 68. Understanding domain testingWhat is equivalence? 4 views of what makes values equivalent. Each has practical implications • Intuitive Similarity: two test values are equivalent if they are so similar to each other that it seems pointless to test both. » This is the earliest view and the easiest to teach » Little guidance for subtle cases or multiple variables • Specified As Equivalent: two test values are equivalent if the specification says that the program handles them in the same way. » Testers complain about missing specifications may spend enormous time writing specifications » Focus is on things that were specified, but there might be more bugs in the features that were un(der)specified Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 82
69. 69. Understanding domain testingWhat is equivalence? • Equivalent Paths: two test values are equivalent if they would drive the program down the same path (e.g. execute the same branch of an IF) » Tester should be a programmer » Tester should design tests from the code » Some authors claim that a complete domain test will yield a complete branch coverage. » No basis for picking one member of the class over another. » Two values might take program down same path but have very different subsequent effects (e.g. timeout or not timeout a subsequent program; or e.g. word processors interpretation and output may be the same but may yield different interpretations / results from different printers.) Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 83
70. 70. Understanding domain testingWhat is equivalence? • Risk-Based: two test values are equivalent if, given your theory of possible error, you expect the same result from each. » Subjective analysis, differs from person to person. It depends on what you expect (and thus, what you can anticipate). » Two values may be equivalent relative to one potential error but non-equivalent relative to another. Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 84
71. 71. Understanding domain testingTest which values from the equivalence class?Most discussions of domain testing start from several assumptions: • The domain is continuous [This is easily relaxed -- CK] • The domain is linearizable (members of the domain can be mapped to the number line) or, at least, the domain is an ordered set (given two elements, one is larger than the other or they are equal) • The comparisons that cause the program to branch are simple, linear inequalities “It is possible to move away from these assumptions, but the cost can be high." --- Clarke, Hassell, & Richardson, p. 388If we think in terms of paths, can we use any value that drives the program down the correct path? This approach is common in coverage-focused testing. Unfortunately, it doesnt yield many failures. See Hamlet & TaylorIf you can map the input space to a number line, then boundaries mark the point or zone of transition from one equivalence class to another. These are said to be good members of equivalence classes to use because the program is more likely to fail at a boundary. Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 85
72. 72. Understanding domain testingTest which values from the equivalence class?The program is more likely to fail at a boundary? • Suppose program design: » INPUT < 10 result: Error message » 10 <= INPUT < 25 result: Print "hello" » 25 >=INPUT result: Error message • Some error types » Program doesnt like numbers • Any number will do » Inequalities mis-specified (e.g. INPUT <= 25 instead of < 25) • Detect only at boundary » Boundary value mistyped (e.g. INPUT < 52, transposition error) • Detect at boundary and any other value that will be handled incorrectly • Boundary values (here, test at 25) catch all three errors • Non-boundary values (consider 53) may catch only one of the three errors Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 86
73. 73. Understanding domain testingTest which values from the equivalence class? • The emphasis on boundaries is inherently risk-based • But the explicitly risk-based approach goes further » Consider many different risks » Partitioning driven by risk » Selection of values driven by risk: • A member of an equivalence class is a best representative (relative to a potential error) if no other member of the class is more likely to expose that error than the best representative. Boundary values are often best representatives We can have best representatives that are not boundary values We can have best representatives in non-ordered domainsCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 87
74. 74. Expanding the Notion of Equivalence Consider these cases. Are these paired tests equivalent? • If you tested Would you test 51+52 52+53 53+54 54+55 55+56 56+57 57+58 58+59 59+60 60+61 61+62 62+63 63+64 64+65 65+66 66+67 67+68 68+69Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 88
75. 75. Another Example: Boundaries May Not Be Obvious Character ASCII Code / 47lower bound 0 48 1 49 2 50 Refer to Testing Computer Software, 3 51 pages 9-11 4 52 5 53 6 54 7 55 8 56upper bound 9 57 : 58 A 65 a 97Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 89
76. 76. Another example of non-obvious boundariesStill in the 99+99 programEnter the first valueWait N secondsEnter the second valueSuppose our client application will time out on inputdelays greater than 600 seconds. Does this affect howyou would test?Suppose our client passes data that it receives to aserver, the client has no timeout, and the server timesout on delays greater than 300 seconds. • Would you discover this timeout from a path analysis of your application? • What boundary values should you test? In whose domains? Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 90
77. 77. In sum: equivalence classes and representative values Two tests belong to the same equivalence class if you expect the same result (pass / fail) of each. Testing multiple members of the same equivalence class is, by definition, redundant testing. In an ordered set, boundaries mark the point or zone of transition from one equivalence class to another. The program is more likely to fail at a boundary, so these are the best members of (simple, numeric) equivalence classes to use. More generally, you look to subdivide a space of possible tests into relatively few classes and to run a few cases of each. You’d like to pick the most powerful tests from each class. We call those most powerful tests the best representatives of the class.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 91
78. 78. A new boundary and equivalence tableVariable Risk (potential Classes that Classes that Test cases Notes failure) should not might trigger (best trigger the the failure representatives failure )First input Fail on out-of- -99 to 99 MinInt to -100 -100, 100 range values 100 to MaxInt Doesnt correctly -100, -99,100, discriminate in- 99 range from out-of- range Misclassify Non-digits 0 to 9 0 (ASCII 48) digits 9 (ASCII 57) Misclassify Digits 0 - 9 ASCII other / (ASCII 47) non-digits than 48 - 57 ; (ASCII 58) Note that we’ve dropped the issue of “valid” and “invalid.” This lets us generalize to partitioning strategies that don’t have the concept of “valid” -- for example, printer equivalence classes. (For discussion of device compatibility testing, see Kaner et al., Chapter 8.) Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 92
79. 79. Examples of ordered setsSo many examples of domain analysis involve databases or simple datainput fields that some testers dont generalize. Heres a sample of othervariables that fit the traditional equivalence class / boundary analysis mold.ranges of numbers size of the sum of variables,character codes or of some other computedhow many times something is value (think binary and done think digits)  (e.g. shareware limit on size of a number that you number of uses of a product) enter (number of digits) or  (e.g. how many times you size of a character string can do it before you run out size of a concatenated string of memory) size of a path specificationhow many names in a mailing size of a file name list, records in a database, size (in characters) of a variables in a spreadsheet, document bookmarks, abbreviationsCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 93
80. 80. Examples of ordered setssize of a file (note special amount of available memory (> values such as exactly 64K, 128 meg, > 640K, etc.) exactly 512 bytes, etc.) visual resolution, size ofsize of the document on the screen, number of colors page (compared to page operating system version margins) (across different equivalent event times (when page margins, page sizes) something happens)size of a document on a page, timing: how long between in terms of the memory event A and event B (and in requirements for the page. which order--races) This might just be in terms of resolution x page size, length of time after a timeout but it may be more complex (from JUST before to way if we have compression. after) -- what events are important?equivalent output events (such as printing documents)Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 94
81. 81. Examples of ordered sets speed of data entry (time transitions between between keystrokes, algorithms (optimizations) menus, etc.) (different ways to speed of input--handling of compute a function) concurrent events most recent event, first number of devices connected event / active input or output intensity system resources (voltage) consumed / available (also, speed / extent of voltage handles, stack space, etc.) transition (e.g. from very date and time soft to very loud sound)Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 95
82. 82. Non-ordered setsA sample problem: • There are about 2000 Windows-compatible printers, plus multiple drivers for each. We can’t test them all.They are not ordered, but maybe we can formequivalence classes and choose bestrepresentatives anyway.Here are two examples from programs (desktoppublishing and an address book) developed in 1991-92.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 96
83. 83. Non-ordered setsPrimary groups of printers at that time: • HP - Original • HP - LJ II • PostScript Level I • PostScript Level II • Epson 9-pin, etc.LaserJet II compatible printers, huge class (maybe300 printers, depending on how we define it) 1. Should the class include LJII, LJII+, and LIIP, LJIID- compatible subclasses? 2. What is the best representative of the class?Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 97
84. 84. Non-ordered sets Example: graphic complexity error handling • HP II original was the weak case. Example: special forms • HP II original was strong in paper-handling. We worked with printers that were weaker in paper-handling. We pick different best representatives from the same equivalence class, depending on which error we are trying to detect. Examples of additional queries for almost-equivalent printers • Same margins, offsets on new printer as on HP II original? • Same printable area? • Same handling of hairlines? (Postscript printers differ.)Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 98
85. 85. More examples of non-ordered sets Here are more examples of variables that dont fit the traditional mold for equivalence classes but which have enough values that we will have to sample from them. What are the boundary cases here? Membership in a common group • Such as employees vs. non-employees. • Such as workers who are full-time or part-time or contract. Equivalent hardware • such as compatible modems, video cards, routers Equivalent output events • perhaps any report will do to answer a simple the question: Will the program print reports? Equivalent operating environments • such as French & English versions of Windows 3.1Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 99
86. 86. More examples of non-ordered setsRather than thinking about a single variable with a single range of values, avariable might have different ranges, such as the day of the month, in a date: 1-28 1-29 1-30 1-31We analyze the range of dates by partitioning the month field for the date intodifferent sets: {February} {April, June, September, November} {Jan, March, May, July, August, October, December}For testing, you want to pick one of each. There might or might not be a“boundary” on months. The boundaries on the days, are sometimes 1-28,sometimes 1-29, etc This is nicely analyzed by Jorgensen: Software Testing--A Craftsman’s Approach.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 100
87. 87. An example of interactionInteraction-thinking is important when we think ofan output variable whose value is based on someinput variables. Here’s an example that givesstudents headaches on tests:I, J, and K are integers. The program calculates K= I*J. For this question, consider only cases inwhich you enter integer values into I and J. Do anequivalence class analysis from the point of viewof the effects of I and J (jointly) on the variable K.Identify the boundary tests that you would run(the values you would enter into I and J) if • I, J, K are unsigned integers • I,J, K are signed integersCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 101
88. 88. Interaction exampleK can run from MinInt (smallest integer) to MaxInt.For any value of K, there is a set of values of I and J that will yield K • The set is the null set for impossible values of K • The set might include only two pairs, such as K = MaxInt, when MaxInt is prime (7 could be a MaxInt) » (I,J) Є {(1, MaxInt), (MaxInt, 1)} • The set might include a huge number of pairs, such as when K is 0: » (I,J) Є {(0,0), (1, 0), (2, 0), … (MaxInt,0), (0,1),…,(0, MaxInt,1)}• A set of pairs of values of I and J can be thought of as an equivalence set (they all yield the same value of K) and so we ask which values of K are interesting (partition number 1) and then which values of I and J would produce those K-values, and do a partitioning on the sets of (I,J) pairs to find best representatives of those.• As practitioners, we do this type of analysis often, but many of us probably don’t think very formally about it. Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 102
89. 89. Domain TestingStrengths • Find highest probability errors with a relatively small set of tests. • Intuitively clear approach, easy to teach and understand • Extends well to multi-variable situationsBlind spots or weaknesses • Errors that are not at boundaries or in obvious special cases. » The "competent programmer hypothesis" can be misleading. • Also, the actual domains are often unknowable. • Reliance on best representatives for regression testing leads us to overtest these cases and undertest other valuesCopyright © 1994-2004or almost as, Methods, LLC. that were as, Software Quality good. All Rights Reserved. 103
91. 91. Test Case Design We’ll soon see a wide variety of strategies for designing test cases. Each one offers its own definition of excellence in test case design. This section summarizes some traditional wisdom about good test case design.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 106
92. 92. Test Case DesignIf the purpose of testing is to gain information aboutthe product, then a test case’s function is to elicitinformation quickly and efficiently.In information theory, we define “information” interms of reduction of uncertainty. If there is littleuncertainty, there is little information to be gained.A test case that promises no information is poorlydesigned. A good test case will provide informationof value whether the program passes the test orfails it.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 107
93. 93. Models A Model is… • A map of a territory • A simplified perspective • A relationship of ideas • An incomplete representation of reality • A diagram, list, outline, matrix…No good test design has ever been done without models. The trick is to become aware of how you model the product, and learn different ways of modeling.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 108
95. 95. Implications of the Simple Model • We control the inputs • We can verify results But, we aren’t dealing with all the factors • Memory and data • Program state • System environmentCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 110
96. 96. Expanded Black Box Testing Model Test Inputs Test Results Precondition Post-condition Data System Data Under Precondition Post-condition Program State Test Program State Environmental Environmental Inputs ResultsCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 111
97. 97. Implications of the Expanded Model We don’t control all inputs We don’t verify everything Multiple domains are involved The test exercise may be the easy part We can’t verify everything We don’t know all the factorsCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 112
98. 98. The “Complete” Oracle Test Inputs Test Results Test Results Test Postcondition Data Precondition Data Oracle Postcondition Data Postcondition Precondition System Program State Program State Postcondition Under Program State Test Environmental Environmental Results Inputs Environmental ResultsReprinted with permission of Doug Hoffman Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 113
99. 99. An Example Model For SUTUser GUI Remote GUI User FunctionalAPI Engine Data Set System Under TestCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 114
100. 100. Breaking Down The Testing ProblemUser GUI Remote GUI User FunctionalAPI Engine Data Set System Under Test Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 115
101. 101. Good Test Case DesignAn excellent test case satisfies the followingcriteria: • Reasonable probability of catching an error • Exercises an area of interest • Does interesting things • Doesn’t do unnecessary things • Neither too simple nor too complex • Not redundant with other tests • Makes failures obvious • Allows isolation and identification of errorsCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 116
102. 102. Good Test Case Design: Neither Too Simple Nor Too Complex • What makes test cases simple or complex? (A simple test manipulates one variable at a time.) • Advantages of simplicity? • Advantages of complexity? • Transition from simple cases to complex cases (You should increase the power and complexity of tests over time.) • Automation tools can bias your development toward overly simple or complex tests Refer to Testing Computer Software, pages 125, 241, 289, 433Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 117
103. 103. Good Test Case Design: Make Program Failures Obvious Important failures have been missed because they weren’t noticed after they were found. Some common strategies: • Show expected results. • Only print failures. • Log failures to a separate file. • Keep the output simple and well formatted. • Automate comparison against known good output. Refer to Testing Computer Software, pages 125, 160, 161-164Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 118
104. 104. Make Test Output Obvious: Examples from Printer TestingNormal. Italics. Bold. Bold Italics. Normal. Italics. Bold. Bold Italics.Normal. Italics. Bold. Bold Italics. Normal. Italics. Bold. Bold Italics.Normal. Italics. Bold. Bold Italics. Normal. Italics. Bold. Bold Italics.Normal. Italics. Bold. Bold Italics. Normal. Italics. Bold. Bold Italics.Frame graphical outputPut a one-pixel wide frame around the graphic and/or around theedge of the printable page. Off-by-one-pixel errors will often eraseone of these borders.For color output, create color sample charts.Print a color wheel or some other standard progression (light to darkor rainbow progression) that is visually obvious. Name the colors.Have a comparison page handy.Draw regular line-art that can show distortionsDraw circles and perfect squares. Look for bowing, stretching,dropout. Use some dashed lines, to see if dashes are drawnunequally long.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 119
105. 105. Making a Good Test • Start with a known state • Design variation into the tests » configuration variables » specifiable (e.g. table-loadable) data values • Check for errors • Put your analysis into the test itself • Capture information when the error is found (not later) » test results » environment results • Don’t encourage error cascadesCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 120
106. 106. Black Box Software Testing Part 5 An Overview of Product Development Supplementary Reading: • Coyote Valley Software’s Product Lifecycle Model: http://www.coyotevalley.com/plc/builder.htm • Tom Gilb’s Evolutionary Results Delivery Method. http://stsc.hill.af.mil/swtesting/gilb.aspCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 123
107. 107. Product Development Organization User Interface Programming Design Marketing Glass Box Black Box Black Box Testing Testing Project Testing Multimedia Manager Production Writing Content Customer Development Manufacturing ServiceDuring development, several groups (or people) act as serviceproviders to the Project Manager. They often play other roles beyondthis, but during development, it pays to look at them as members of amanaged group that seeks to ship a high quality, salable product ontime. Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 124
108. 108. Product Development OrganizationProject Manager: Responsible for shipping a salable product on time and within budget.Architect: Design the product’s overall (code & data) structure, relationships among components, and relationships with other expected parts of the customers’ system.User Interface Designers: Design to make the program usable and useful.Programmers: Design and write/test the code.Glass Box Testers: Use knowledge of the internals of the code to drive the testing. Along with glass box test techniques, use code inspections and other code-aware methods to find errors.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 125
109. 109. Product Development OrganizationTech Writers: Write the user manual, help, etc.Content Developers: There may be several different groups here, animating, filming, writing and performing music, writing, etc.Multimedia Producer: Responsible for managing content development, selecting content to include with the product, negotiating rights, etc.Marketing: Manage the product’s overall profit/loss plan. Evaluate competitors, appraise sub-markets (e.g. should we be compatible with a certain printer?), design packaging and advertisements, run trade shows, etc.Black Box Testers: Test from the customer’s perspective.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 126
110. 110. Product Development OrganizationCustomer Service: Help evaluate the effect of design decisions, bugs, and existing problems on customers.Manufacturing: Buy materials. Help everyone control their contribution to the cost of goods. Drive the late-stage schedule. Make the product.Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 127
111. 111. The Waterfall Model for Software Development Idea -- Require- ----- ments Design -------- Detailed ----------------- Design Coding -------------------------- Test -------------------------- Planning Write ----------------------------- manual Testing ------------------------------------------- Customer ------------------------------------------------- Beta Release --------------------------------------------------------------------------------- productCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 128
112. 112. What the Waterfall Really Looks Like Idea --- Announce ------ product Choose ------ team Order T- ---------- shirts Code ------------- Design -------------------- Accept ----------------------- orders Test ------------------------------ Write ------------------------------------- manual Write test ---------------------------------------- plan Write specs ----------------------------------------------- Release --------------------------------------------------------------------------- product Customer ------------------------------------------------------------------------------ complaintsCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 129
113. 113. Evolution V is io n B u ild C o r e D e s ig n D e s ig n D e s ig n D e s ig n Code Code Code Code T est T est T est T est F ix F ix F ix F ix D e s ig n D e s ig n D e s ig n D e s ig n Code Code Code Code T est T est T est T est F ix F ix F ix F ix D e s ig n D e s ig n D e s ig n D e s ig n Code Code Code Code T est T est T est T est F ix F ix F ix F ix See http://stsc.hill.af.mil/SWTesting/gilb.htmlCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 130
115. 115. Mock-Up ApproachRevise ReviseRequirements Prototype Evaluate Prototype Detailed Design Code/Unit Test Integration/Product Verification Implementation/System Test Operation and Maintenance/ Revalidation Copyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 132
116. 116. Spiral Model Determine objectives, Evaluate Alternatives, alternatives, constraints Identify and Resolve Risks Risk Analysis Risk Analysis Risk Analysis Operational Risk Prototype Prototype Anal Proto Prototype Re qu ir e Concept me De nts Detailed ve Pla l op n nts Design me me Initial Int i re Pla egrat nt Pl a R equ Design n io n na nd Te st Implementation Plan Next Phases Develop, Verify Next Level ProductCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 133
117. 117. Decomposition/Integration Requirements Operation System Test Specification n De tio Analysis Integration co ra m teg po Design Unit Test In sit io n DevelopmentCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 134
118. 118. Cleanroom Development Requirements & Spec Team Team Baselined S/W Des and Dev Specification Test & Proof 1st Module 1st Module Development Planning Des and Dev Test & Proof 1st & 2nd 1st & 2nd Verify Module Modules 1st Module Des and Dev Test & Proof Modules Verify Modules 1, 2, 3 1st & 2nd 1, 2, 3 Module Verify Verification/ Modules 1, 2, 3 Certification TeamCopyright © 1994-2004 Software Quality Methods, LLC. All Rights Reserved. 135