Test engineering foundation (v2.01)

2,739 views

Published on

ISTQB ISEB Certification

Published in: Technology, Business
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,739
On SlideShare
0
From Embeds
0
Number of Embeds
14
Actions
Shares
0
Downloads
137
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Test engineering foundation (v2.01)

  1. 1. Test Engineering Foundation Essential Knowledge for Test Professionals Rex Black RBCS, Inc. 31520 Beck Road Bulverde, TX 78163 USA Phone: +1 (830) 438-4830 Fax: +1 (830) 438-4831 www.rexblackconsulting.com rex_black@rexblackconsulting.com
  2. 2. Introductions, Objectives, and Overview Test Engineering Foundation Essential Knowledge for Test Professionals
  3. 3. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 3 The Course Who is this presenter? What is this course? What are the hours? breaks? Must I do exercises? Do I have homework? Can I get certified? May I use my cell phone or read e-mail during the course? Any others???
  4. 4. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 4 The Materials These slides following the Foundation Syllabus 2005 Terms come from the ISTQB Glossary (v1.1) In practice terminological variation is wide Not all terms in syllabus are defined in glossary Reference materials, glossary, and study aids for those pursuing certification For use in exercises Omninet Marketing Requirements Document Omninet System Requirements Document Solutions for the exercises A notepad on which to do the exercises
  5. 5. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 5 ISTQB Foundation Syllabus 2005 Developed by a team of eight authors spanning seven countries: Thomas Müller (chair), Rex Black, Sigrid Eldh, Dorothy Graham, Klaus Olsen, Maaret Pyhäjärvi, Geoff Thompson and Erik van Veendendaal Forty primary reviewers spanning nine countries Final review and approval by 18 National Boards Distills over 1,000 person-years of experience The ISTQB and the authors are the source of the syllabus (copyright © 2005 by the authors and ISTQB) which is used by permission as the basis for this course
  6. 6. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 6 The Main Topics Covered 1. Fundamentals of testing 2. Testing throughout the software lifecycle 3. Static techniques 4. Test design techniques 5. Test management 6. Tool support for testing
  7. 7. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 7 The Exercises In many of the exercises, you are working as a tester on the Omninet project Omninet is a project to deploy a network of public access Internet kiosks in places like malls, theaters, and other public places On this realistic project, you will have a chance to apply many of the techniques we discuss Other exercises have been added to illustrate specific points, too
  8. 8. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 8 Your Course Objectives This material is designed to help you become a more effective and efficient test professional, and to help you obtain ISTQB certification Please spend the next few minutes writing down what you’d like to get out of the course Use the following page for your objectives Don’t let the class end with an objective unfulfilled!
  9. 9. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 9 Course Objectives
  10. 10. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 10 It’s Your Course Please join in  Ask questions  Make comments  Share experiences  Second opinions and disagreements welcome Why are you here? What do you want to learn, discuss, and teach? …as well as this. The best sessions have a lot of this...
  11. 11. Chapter 1: Fundamentals of Testing Test Engineering Foundation Essential Knowledge for Test Professionals
  12. 12. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 12 1. Fundamentals of Testing 1. Why is testing necessary? 2. What is testing? 3. General testing principles 4. Fundamental test process 5. The psychology of testing
  13. 13. Chapter 1: Fundamentals of Testing Section 1: Why is testing necessary? Test Engineering Foundation Essential Knowledge for Test Professionals
  14. 14. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 14 Why is Testing Necessary? Key concepts How bugs can cause harm Bugs and their effects The necessity of testing The role of testing in quality assurance Terms to remember
  15. 15. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 15 The Threat of Bugs Company Damaged reputation for quality High or unpredictable maintenance costs Unexpected delays in release cycles Lack of confidence in system Lawsuits Environment Pollution Waste People, societies, and states Lost jobs Lost lives Lost rights Lost missions Lost wars
  16. 16. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 16 Where Bugs Come From and What Bugs Do People put bugs (defects) into the system Requirements and design specifications Code (business logic and user interface) Documentation (electronic and hard copy) When the implementation of these bugs are executed, failures occur If these failures are visible to customers, users, or other stakeholders, dissatisfaction with system quality results
  17. 17. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 17 From Whence the Bugs and Failures? Bugs occur due to… programmer, analyst, and other individual contributor (including tester) fallibility time pressure complexity of the code, infrastructure, or problem to be solved changing and meshing technologies many system interactions. Failures occur due to bugs and… environmental conditions misuse (deliberate and accidental)
  18. 18. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 18 Testing to Manage Quality Risks Risks and constraints for a software project Features: Right set Schedule: Quickly enough Budget: Acceptably cheap Quality: Ready for customers/release/next step Testing provides the information to guide the project, reduce and manage the risks, and repair the important problems Testing may also address compliance, contractual, and regulatory needs
  19. 19. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 19 What Does “Quality” Mean to You? “Fitness for use” vs. “Conformance to requirements” Testing and quality Tests gives confidence where they find few bugs Passing tests reduce the level of quality risk Failing tests provide a chance to improve quality The test set gives an assessment of quality What are the important quality characteristics for your system? Are you testing them (enough)? Testing, quality assurance, and quality improvement Ideally, testing is part of a larger quality assurance strategy for a project For future projects, analyze the root causes of defects found on current projects and take steps to reduce their incidence
  20. 20. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 20 Terms to Remember Bug, defect, or fault Error or mistake Failure Quality Risk Software Test Test case
  21. 21. Chapter 1: Fundamentals of Testing Section 1: A diversion Test Engineering Foundation Essential Knowledge for Test Professionals
  22. 22. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 22 What Happens When Quality is Lacking
  23. 23. Chapter 1: Fundamentals of Testing Section 2: What is testing? Test Engineering Foundation Essential Knowledge for Test Professionals
  24. 24. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 24 What is Testing? Key concepts Common objectives of testing The purpose of testing… •in software development, maintenance, and operations… •…to find defects, provide confidence and information, and prevent defects Terms to remember
  25. 25. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 25 Testing Objectives Typical general testing objectives Finding bugs and providing programmers with the information they need to fix important bugs Gaining confidence about the level of quality of the system Preventing defects (through early involvement in reviews and advanced test design) Provide information about the most important aspects of the quality of the system under test Help management understand system quality Can you think of others for your projects?
  26. 26. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 26 Unit/Component Test Find bugs in the individual pieces of the system under test before the pieces are fully integrated into the system Integration/String Test Find bugs in the relationships and interfaces between pairs and groups of components in the system under test as the pieces come together System Test Find bugs in the overall and particular behaviors, functions, and responses of the system under test as a whole Acceptance/Pilot Test Demonstrate that the product is ready for deployment/release or to assess quality and give information on the risk of deployment/release Maintenance Test Check for errors introduced during development of the changes Operational Test Assess non-functional system characteristics such as reliability or availability Test Phases and Objectives
  27. 27. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 27 Effectiveness and Efficiency Effective: Producing a decided, decisive, or desired result; impressive To be effective testers, we must select the appropriate objective and desired results Efficient: Productive of desired effect; especially productive without waste To be efficient testers, we must allocate resources (time and money) appropriately These terms are most meaningful in the context of the entire development or maintenance process, not just the testing process.
  28. 28. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 28 Testing vs. Debugging Testing finds failures that are caused by bugs Debugging… …identifies the root cause of a bug… …repairs the code… …and checks that the defect is fixed correctly Confirmation testing ensures the fix resolves the observed failure Different responsibilities: Testers test Programmers debug
  29. 29. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 29 Find-Debug-Confirm
  30. 30. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 30 Beyond Test Execution Testing is not just running tests against a running system Other test activities, before and after execution, include… Planning and control Choosing test conditions Design test cases Checking test results Evaluating exit criteria Test result reporting Closure/end-of-test tasks We will revisit this later
  31. 31. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 31 Terms to Remember Code Debugging Software development Review Requirement Test case Test objective Testing Test basis
  32. 32. Chapter 1: Fundamentals of Testing Section 2: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  33. 33. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 33 Exercise: The Triangle Tests A program accepts three integers representing the lengths of a triangle’s sides. It outputs “scalene” (no equal sides), “isosceles” (two equal sides), or “equilateral” (three equal sides). Write an effective (finds common bugs) and efficient (as few tests as possible) set of test cases. Usually, a test case consists of tester action, data, and expected result, but here the action is generally “input data”. Discuss.
  34. 34. Chapter 1: Fundamentals of Testing Section 3: General testing principles Test Engineering Foundation Essential Knowledge for Test Professionals
  35. 35. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 35 General Testing Principles Key concepts Testing reveals the presence of bugs Impossibility of exhaustive testing Benefits of early testing Lumpiness of bugs: defect clustering Pesticide paradox Testing should adapt to specific needs Absence-of-errors fallacy Terms to remember
  36. 36. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 36 Testing Reveals Presence of Bugs: A Parable You have a beautiful vegetable garden, but one day you see eaten leaves on the tomatoes “Oh no,” you think, “I have hornworms!” You know you have bugs in your garden If you had not seen the symptoms, could you be sure you had no bugs? Some bugs are easy to spot, others aren’t Testing can reveal the presence of bugs, but cannot prove their absence
  37. 37. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 37 Mission Impossible: Exhaustive Testing “Just make sure the software works before we ship it…” This charter is demonstrably impossible The execution paths in non-trivial software are almost infinite Large dataflows separated across space (features) and time (static data) Slight changes can cause regressions which are not linear to the size of the change Myriad usage profiles and field configurations, some unknown and some unknowable
  38. 38. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 38 Defusing Exhaustive Testing Expectations Exhaustive testing as a way to prove the software works is a common (mis)expectation Bad expectations create problems for test professionals and test teams Unachievable high demands on test group Perception of incompetence when these demands aren’t met Testers must be ready to communicate (in words the project stakeholders will understand) how testing can contribute
  39. 39. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 39 Benefits of Early QA and Testing The cost of a bug tends to increase as the project continues Most of the costs associated with pre-release bugs tend to be associated with the effort required to remove them, so the higher cost means longer schedules The more bugs enter a quality assurance or test activity, the more bugs will escape from that activity
  40. 40. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 40 Click here to see the answers
  41. 41. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 41 Defect Clustering Studies have long shown and continue to show the unequal distribution of bugs MVS: 38% of field bugs in 4% of modules IMS: 57% of field bugs in 7% of modules Capers Jones reports that the excessive presence of error-prone modules causes a 50% reduction of productivity in software maintenance
  42. 42. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 42
  43. 43. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 43 Pesticide Paradox Return to your vegetable garden You spray pesticide on your garden, and the hornworms die, but the pesticide is not effective against all bugs Just as pesticides become less effective, so do tests Functional tests can’t find performance bugs Try new test techniques
  44. 44. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 44 These two pictures compare efficiencies of the top 11 bug-finding test suites (out of 27). The top graph shows the first time the set of test suites was run. The bottom graph shows the results for the fifth time that same set of test suites was run. The average test suite efficiency in the first pass, 0.4, is used as the axis crossing point in both graphs. With one exception, the same test suites are in the top 11 test suites, which shows bug clustering again. However, all test suites are less effective after five executions.
  45. 45. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 45 Absence-of-Errors Fallacy Finding and fixing many bugs does not guarantee user, customer, and/or stakeholder satisfaction Many low-defect products have failed in the market place Successful projects balance competing forces in terms of features, schedule, budget, and quality
  46. 46. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 46 Testing Should Adapt to Needs Different projects, organizations, and products have different testing needs Best testing practices exist (and are discussed in this course) but you need to tailor them to your project Failure to adapt the test team and its methods to these needs is a common result of dissolution of test teams
  47. 47. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 47 Terms to Remember Exhaustive testing
  48. 48. Chapter 1: Fundamentals of Testing Section 3: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  49. 49. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 49 Exercise: Test Principles Observed (and Not) Think back on a recent project. Note which of the principles that you can recall were observed. How about principles that you can recall were not observed (i.e., violated)? Discuss.
  50. 50. Chapter 1: Fundamentals of Testing Section 4: Fundamental test process Test Engineering Foundation Essential Knowledge for Test Professionals
  51. 51. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 51 Fundamental Test Process Key concepts Plan, prepare, perform, perfect Planning and control Analysis and design Implementation and execution Evaluating test exit criteria and reporting Test closure activities Terms to remember
  52. 52. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 52 Test Processes: Critical Testing Processes The test engineer’s role focuses on some of these activities, not all, depending on how roles are defined. However, the effective and efficient test engineer must understand how the test process works and how it fits into the overall project from a big picture perspective.
  53. 53. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 53 ISTQB Fundamental Test Process Another way to think of the test process is in terms of the following steps Planning and control Analysis and design Implementation and execution Evaluating test exit criteria and reporting Test closure activities These steps may overlap or take place concurrently
  54. 54. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 54 Planning and Control Planning Determine test scope, risks, objectives, strategies Determine required test resources Implement the test strategies Schedule test analysis and design Schedule implementation, execution and evaluation of tests Determine the test exit criteria Control Measure and analyze results Monitor and document progress, coverage and test exit criteria Initiate corrective actions Make decisions
  55. 55. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 55 Analysis and Design Analysis Review the test basis (e.g., requirements or design specifications, network/ system architecture, quality risks) Identify test conditions, test requirements, or test objectives and required test data based on analysis of test items, its specification, behavior and structure Design Select specific combinations of test data, actions, and expected results to cover the test basis Evaluate testability of the requirements and system Design the test environment Identify any required infrastructure and tools
  56. 56. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 56 Implementation and Execution Implementation Develop and prioritize test cases, create test data, write test procedures Prepare test harnesses and write automated test scripts Organize test suites and sequences of test cases for efficient test execution Verify that the test environment has been set up correctly Execution Execute test cases (manual or automated) Log test results, and the versions of the software under test, test tools and the testware Compare actual and expected results Report and analyze incidents Repeat corrected and/or updated tests Run confirmation and/or regression tests
  57. 57. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 57 Exit Criteria, Reporting, and Closure Exit and reporting Check test logs against the test exit criteria specified in test planning Assess if more tests are needed or if the exit criteria specified should be changed Write a test summary report for stakeholders Closure Confirm test deliverables, final resolution or deferral of bug reports, and the acceptance of the system Finalize and archive testware, test environment and test infrastructure Deliver testware to the maintenance organization Perform a retrospective to capture improvements for future releases, projects, and test processes
  58. 58. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 58 Terms to Remember Testware Test plan Test strategy Test execution Test basis Test log (Test) exit criteria Test summary report Test coverage Test condition
  59. 59. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 59 Terms to Remember Test data Test input Confirmation testing Regression testing Incident
  60. 60. Chapter 1: Fundamentals of Testing Section 4: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  61. 61. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 61 Exercise: Test Steps and Tasks Performed Think back on a recent project. Note which of the steps and tasks of the ISTQB test process were carried out. Note which of the steps and tasks of the ISTQB test process were not carried out. Note whether some steps overlapped or were completely parallel. Discuss.
  62. 62. Chapter 1: Fundamentals of Testing Section 5: The psychology of testing Test Engineering Foundation Essential Knowledge for Test Professionals
  63. 63. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 63 Psychology of Testing Key concepts Psychological factors for testing success Clear objectives Self-testing and independent testing Respectful communication Programmer and tester outlooks for success Terms to remember
  64. 64. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 64 Certainty Versus Progress Thorough research can lead to indisputable bug reports, unquestionably correct test cases, etc. Such certainty is intellectually satisfying and reduces rejected and irreproducible bug reports But certainty also can consume too much time and effort for the payoff and delay forward progress Test projects adapt to schedule pressures or fail Testing computers is test engineering Not a search for truth—that’s science Engineering is making useful objects for customers
  65. 65. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 65 Test Result Misinterpretation On the one hand we could report correct behavior as a bug assign excessively high severity or priority otherwise overstate bug significance On the other hand we could fail to detect or report incorrect behavior assign excessively low severity or priority otherwise understate bug significance Some tips to avoid these errors 1. Have testers take breaks so they don’t miss important events 2. Automate where practical 3. Define expected results as clearly as possible 4. Assign the right testers to each test execution task 5. Use peer reviews for test execution and bug reports However…recognize that perfect test execution takes too long. Define “good enough” for testing and live with that decision
  66. 66. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 66 Professional Pessimism Explore depressing possibilities of failure Anticipate the worst possibilities in order to achieve best obtainable product quality Not adversarial, but a different outlook than the programmers Remember: to assume nothing will fail during testing denies the entire history of computing Caveat: not a license to offend  Don’t target programmers with reports or take glee in failure Challenge: to be positive, pleasant, and the bearer of bad news, all at once Pierre has the pessimism, but perhaps not professionalism?
  67. 67. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 67 Bad News == Bad Guy? Testers are sometimes on receiving end of emotions brought on by news of project problems The key is professional pessimism Don’t take on the role of “Lone Champion of Quality” • Product quality is a business decision • Quality risks are weighed against other risks Don’t second-guess development’s bug-fix efforts Never gloat, even if you were right and everyone else was wrong Juan de Mariana: “The greatest of follies is to exert oneself in vain, and to weary oneself without winning anything but hatred.” Grim reaper or friendly guide?
  68. 68. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 68 Balanced Curiosity Balance need for thoroughness in any one area with need to cover many areas in a short time Effective and efficient test engineers have a talent for spending time where the bugs are Effective and efficient test engineers can do thorough bug isolation quickly Ineffective and inefficient test engineers Write tests to search for unlikely, low-impact bugs Spend hours researching trivial bugs
  69. 69. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 69 Two types of focus problems Pursuing issues narrow-mindedly, losing sight of more important priorities Getting distracted from key tasks Balance and re-evaluate priorities every so often Stay focused on the goals of the test project Focus A seasoned test engineer can find his way towards test project goals, with clear signposts from his test manager
  70. 70. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 70 Defining Tester Skills Reading Specifications, e-mails, test cases, etc. Writing Test cases, bug reports, test documentation, etc. Not “native language” dependent Statistics and other mathematics Pertinent technology, project, and testing skills Technology: Programming languages and more, like operating systems, networking, HTML/Web, etc. Application domain: banking, human factors, office applications, etc. Testing: scripting, exploring and attacking the system, automation, performance modeling, etc.
  71. 71. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 71 Balancing the Skills Effective and efficient tester have right mix of skills for tasks and activities Application domain expert Understands intended behavior Skilled tester Knows quality risks and test techniques Technical guru Aware of technical issues and limitations What is the right mix for… …Internet appliance testing? …nuclear medicine testing? …your project? The appropriate depth and length of each arrow in the figure depends on the project, process, and product
  72. 72. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 72 Terms to Remember Independent testing
  73. 73. Chapter 1: Fundamentals of Testing Section 5: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  74. 74. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 74 Exercise: Psychology in Action Reflect on the attitudes and behaviors of the most successful testers you know. To what extent do they display the psychological aspects discussed in this section? What other elements of their personalities and skillsets do you think lead to success for them? Discuss.
  75. 75. Chapter 2: Testing throughout the Software Lifecycle Test Engineering Foundation Essential Knowledge for Test Professionals
  76. 76. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 76 2. Testing Throughout the Software Lifecycle 1. Software development models 2. Test levels or phases 3. Test types or targets 4. Maintenance testing
  77. 77. Chapter 2: Testing throughout the Software Lifecycle Section 1: Software development models Test Engineering Foundation Essential Knowledge for Test Professionals
  78. 78. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 78 Software Development Models Key concepts The relationship between development and test activities Adapting software development models to the context of the project and product Reasons for different levels of testing Terms to remember
  79. 79. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 79 The “V” Model in Waterfall Projects Usually schedule- and budget-risk-driven Do ever-deeper levels of design, then build, then test Intuitive and familiar model Beats chaos! It’s hard to plan that far in advance! When plans fail, test--at the end--suffers!
  80. 80. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 80 Evolutionary and Incremental Models Schedule-risk-driven to hit market window or delivery date Feature set grown around core functionality Can ship (something) any time once the core functionality is ready Becoming a popular approach Ranges in formality from Extreme Programming to RAD and RUP  Still a temptation to ship a system with buggy features  In the “agile” world, the role of testing still evolving
  81. 81. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 81 System Integration Many projects involve integrating components Risk-mitigation options Integrate, track, and manage vendor testing in distributed test effort Trust vendor component testing Fix vendor testing/quality Disregard and replace their testing (ouch!) Watch politics in last two options Plan on integration and system testing yourself Coupling: Strong interaction or consequence of failure between component and system Irreplaceability: Few similar components available Essential: Key features in system unavailable if component does not work properly Vendor quality problems: Increased likelihood of a bad component
  82. 82. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 82 Verification & Validation Verification Look for bugs in phase deliverables “Are we building the system right?” Validation Looking for bugs in system, based on phase deliverables “Are we building the right system?”
  83. 83. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 83 IEEE 12207 Standard 1. Scope Purpose, application, tailoring, compliance, limitations 1. Normative references 2. Definitions 3. Application Lifecycle processes, tailoring, and fitting to organization 1. Primary life cycle processes Acquisition, supply, development, operation, maintenance 1. Supporting processes Tech pubs, CM, QA, IV&V, audits, problem resolution 1. Organizational life cycle processes Management, infrastructure, improvement, training
  84. 84. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 84 CMMI Process Maturity Capability Maturity Model Integration: a five-level model from the Software Engineering Institute Initial: unpredictable, poorly controlled, reactive Managed: process established at process level, often reactive Defined: process established across organization, usually proactive Quantitatively managed: process measured and controlled at organization Optimizing: focus on continuous improvement, usually driven by data Testing has been underemphasized in CMM, leading to test-specific model (e.g., Critical Testing Processes, Test Process Improvement, Testing Maturity Model)
  85. 85. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 85 Regardless of the Models… General characteristics of good testing Testing activity for each development activity (e.g., unit test and implementation) Test levels have focused objectives, with coordination to avoid gaps, overlap Test analysis, design begins early, prevents bugs Testers involved in any reviews they are qualified to attend, bringing their unique perspective You can combine or reorganize test levels provided you keep these characteristics in mind
  86. 86. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 86 Terms to Remember Test level Verification Validation V-Model Incremental development model Commercial Off The Shelf (COTS)
  87. 87. Chapter 2: Testing throughout the Software Lifecycle Section 1: Exercise 1 Test Engineering Foundation Essential Knowledge for Test Professionals
  88. 88. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 88 Exercise: Models and Reality Famed quality expert W.E. Deming said, “All models are wrong; some are useful.” Indicate which lifecycle model in this section applied most closely to your past project (or, if there was no organizing model, indicate “code-and-fix”). To what extent do you think the model was useful? To what extent, if any, was it harmful? Discuss.
  89. 89. Chapter 2: Testing throughout the Software Lifecycle Section 1: Exercise 2 Test Engineering Foundation Essential Knowledge for Test Professionals
  90. 90. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 90 Exercise: Omninet Lifecycle Considerations Read the Omninet Marketing Requirements Document. Is a V-model (sequential) or incremental model more appropriate for this project? Why? Are maintenance, integration, or verification and validation important for this project? Why? Discuss.
  91. 91. Chapter 2: Testing throughout the Software Lifecycle Section 2: Test levels or phases Test Engineering Foundation Essential Knowledge for Test Professionals
  92. 92. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 92 Test Phases or Levels Key concepts Major objectives of testing for each level Typical testing objects for each level Typical targets of testing for each level Testing work products for each level Test participants for each level Types of defects and failures for each level Terms to remember
  93. 93. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 93 Component (Unit) Test Objective: Find bugs in the individual pieces of the system under test prior to system integration Basis: Code, database, reqs/design, quality risks Test types: Functionality, resource use, performance, structural Item Under Test (IUT): Varies. Smallest independently testable item (function or class) or a distinct component providing services to others Harnesses and tools: API level (drivers and stubs), freeware and commercial Responsible: Usually programmers, but the level of proficiency and degree of execution varies.
  94. 94. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 94 Unit/Component Test Process Unit/component testing typically… …involves access to the code …is run in a development environment …requires drivers, stubs, and/or harnesses …is done by the programmer who wrote the code Often, bugs are fixed upon being found without any reporting, which reduces the transparency of the development process with respect to quality Test-first/test-driven development Develop a set of unit tests Build and integrate code Run the tests and debug until the tests pass
  95. 95. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 95 Drivers and Stubs During unit, component, and integration testing--and for testing APIs--it’s often necessary to simulate parts of call flow reachable from module(s) under test Data setup sometimes necessary, also Driver: function(s) that call module(s) under test Stub: function(s) called--directly or indirectly--by module(s) under test
  96. 96. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 96 Integration Test Objective: Find bugs in the relationships and interfaces between pairs and groups of components in the system under test as the pieces come together Basis: Design, architecture, schemas, dataflows, quality risks Test types: Functionality, resource use, performance Item Under Test: Builds or backbones Harnesses and tools: API and CLI level, freeware and commercial Responsible: Ideally both testers and programmers, but often no one
  97. 97. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 97 Integration Techniques Big bang Take all tested modules; put them all together; test Quick, but where’s the bug? Why wait until all code is written to start integration? Bottom up Start with bottom layer modules; use appropriate drivers; test Repeat process, replacing drivers with modules, until done Good bug isolation, but what if nasty problems are at the top? Top down Like bottom up, but start from top and use stubs Good bug isolation, but what if nasty problems are at the bottom? Backbone Start with critical modules; build initial backbone; use drivers and stubs; test Repeat process, replacing stubs and drivers with modules in risk order Good bug isolation and finds integration bugs in risk order
  98. 98. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 98 Backbone Integration Technique Backbone 0 (BB0) Consider the following example (based on a real project) We start with a basic backbone Communication APIs Basic networking architecture Test basic functionality, error handling and recovery, reliability, and performance Quality risk: Is the underlying system architecture untenable? Backbone 0: Testing the basic communication software and network architecture
  99. 99. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 99 Backbone Integration Technique Backbone 1 (BB1) Add the modules that implement the most critical core operations and services Again, test basic functionality, error handling and recovery, reliability, and performance Quality risk: Do the core operations and functions integrate with the transport layer? Continue process with next level of quality risk…. Backbone 1: Testing some core operations and services through the communication API and network
  100. 100. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 100 Backbone Integration Technique Backbone N (BBn) As the last step, we are testing (in this case, using automated test drivers) end-to-end through the GUIs on the host systems Quality risk: Does the fully integrated system work? The final backbone for integration testing is also the first fully integrated build for system test Backbone N: Testing the entire, completely integrated system end- to-end. When this works, we’re ready for System Test
  101. 101. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 101 Integration Test Levels There may be more than one level of integration testing on a project Component integration testing: look for bugs in interactions between units or components following unit/component test System integration test: look for bugs in interaction between entire systems following system test System integration testing is complex Multiple organizations controlling the systems’ interfaces, making change dangerous Business processes may span systems Hardware/system compatibility issues can arise
  102. 102. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 102 System Test Objective: Find bugs in the overall and particular behaviors, functions, and responses of the system under test as a whole Basis: Requirements, high-level design, use cases, quality risks, experience, checklists, environments Test types: Functionality, security, performance, reliability, usability, portability, etc. Item Under Test: Whole system, in as realistic-as- possible test environment Harnesses and tools: API, CLI, or GUI, freeware and commercial Responsible: Typically independent testers
  103. 103. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 103 Acceptance Test Objective: Demonstrate that the product is ready for deployment/release Basis: Requirements, contracts, experience Test types: Functional, portability, performance Item Under Test: Whole system, sometimes in the production or customer environment Harnesses and tools: GUI usually Responsible: Often users or customers, but also independent testers
  104. 104. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 104 Variations in Acceptance Testing User acceptance testing: Business users verify fitness for functional purposes. Operational testing: Acceptance by system administrators (e.g., backup-restore, disaster recovery, user management, maintenance, security) Contract and regulation testing: Verification of conformance to contractually-agreed or legally mandated requirements, regulations, or standards. Alpha, Beta, and field testing: Testing and confidence-building by potential or existing customers. Beta testing and field testing are performed in the actual environment(s).
  105. 105. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 105 Pervasive Testing: Early, Cross-Functional Test execution activities for test phases Project timeline
  106. 106. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 106 Why Pervasive Testing? Different participants can test different granularities Different skills for different granularities Different granularities emphasized in each phase Unit testing: primarily structural System testing: primarily behavioral Acceptance testing: primarily live It’s important to be flexible, though Test techniques of various granularities can be useful in all the test execution phases Phase overlap a function of entry and exit criteria Not all test phases occur on all projects
  107. 107. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 107 When Testing Pervades Projects Test tasks occur throughout the development effort Test execution is planned with multiple cycles to allow for fix time Features will be dropped or slipped into later iterations rather than slipping test phase entry dates or entering before ready
  108. 108. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 108 Terms to Remember Alpha testing Beta or field testing Business process-based testing Component testing Contract acceptance testing Daily build Drivers Functional requirements Integration Integration testing
  109. 109. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 109 Terms to Remember Non-functional requirements Operational (acceptance) testing Regulation testing Requirements-based testing Robustness testing Stub System testing Test-driven development Test environment Testware User acceptance testing
  110. 110. Chapter 2: Testing throughout the Software Lifecycle Section 2: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  111. 111. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 111 Exercise: Omninet Test Levels Read the Omninet System Requirements Document. If you were managing all the testing for Omninet, which levels or phases of testing would you plan? Why? What would the major goals of each level or phase of testing be? What kind of acceptance test, if any, would you plan? Why? How do these test levels relate to and affect the lifecycle model you selected in the previous exercise? Discuss.
  112. 112. Chapter 2: Testing throughout the Software Lifecycle Section 3: Test types or targets Test Engineering Foundation Essential Knowledge for Test Professionals
  113. 113. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 113 Test Types or Targets Key concepts Major software test types or targets Functional and non-functional tests Structural tests Confirmation and regression tests Use of functional, non-functional, and structural tests at various levels Terms to remember
  114. 114. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 114 Functionality Reasonable or required action not provided, inaccessible, or seriously impaired No add function on a calculator Add function implemented, “+” key doesn’t work Can only add integers, not real numbers Right action, wrong result Add function: 2+2=5? Right action, right result, wrong side-effect Divide function: 2/2=I (Roman numeral format) System, subsystem or component functionality is described in documents like requirements specification, use cases, or a functional specification (sometimes). However, often some functions remain undocumented, and testers must understand “reasonable behavior”.
  115. 115. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 115 Security Security threats include Virii Cracking into servers Denial of service Common bad assumptions Encryption (HTTPS) on Web server solves security problems Buying a firewall solves problems Unskilled network or system administrators can solve problems Specialized field of test expertise The determined--or bored--cracker might break into any of your servers. One “exploit” can lead the cracker to other vulnerabilities--and to valuable data.
  116. 116. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 116 Performance And Reliability Performance Too slow throughout performance curve Unacceptable “knee” in performance curve Unacceptable performance degradation over time Reliability System fails to complete normal functions System functions normally, but randomly crashes or hangs
  117. 117. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 117 Stress, Capacity, and Volume Stress: extreme conditions cause failure Combinations of error, capacity, and volume tests Capacity: functionality, performance, or reliability problems due to resource depletion Fill hard drive or memory to 80+% Volume: functionality, performance, or reliability problems due to rate of data flows Run 80+% of rated transactions per minute, number of simultaneous users, etc.
  118. 118. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 118 Maintenance And Maintainability Maintenance Update and patch install and deinstall processes don’t work Configurations can’t be changed appropriately (e.g., plug-and-play, hot plugging, adding disk space, etc.) Maintainability Software itself (source code) not maintainable Databases not upgradeable Databases grow monotonically Software not efficiently testable during maintenance; e.g., excessive regression
  119. 119. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 119 Usability and User Interface A system can function properly but be unusable by the intended customer Cumbersome interfaces that do not follow workflows Inaccessible functionality Inappropriately difficult for the users to learn Instructional, help, and error messages that are misleading, confusing, or misspelled
  120. 120. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 120 Configuration, Compatibility, Portability A single platform may be configured in many different ways in software A family of platforms may support various hardware configurations Are configuration changes handled? Add disk space or other storage Add memory Upgrade or add CPU Interoperability with programs, OS, database Portability to various environments
  121. 121. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 121 Other Functional/Non-Functional Tests Localization (user interface) Localization (operational) Standards and regulatory compliance Error handling and recovery Disaster recovery Networked/internet- worked or distributed Timing and coordination Data quality Data conversion Operations Installation De-installation Date and time handling Documentation And many others…
  122. 122. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 122 Structural Tests Tests based on how the system is built Code Data Design Structural (white box) coverage can be measured after functional and non-functional (black box) tests are run to check for omissions This topic will be covered in more depth later…
  123. 123. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 123 Regression and Confirmation Regression testing checks the effects of changes, since even small, localized, isolated changes, don’t always have small, localized, or isolated effects Regression strategies are covered in the next section Confirmation testing confirms that… Changes made to the system are present Bug fixes introduced in the system solve the observed symptoms Repeatability of tests helps with regression and confirmation testing Automation, covered in depth later, is also helpful
  124. 124. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 124 Comparing Test Types Static Structural Func./Non-func. What Test without running Test how system works Test what system does Why Identify bugs before they’re built Identify bugs in functions or interfaces Identify bugs in system results and behavior Who All stakeholders Mostly developers (and knowledgeable testers) Mostly testers (and knowledgeable developers) When During specification and development Mostly during unit, component, integration Mostly integration, system, and acceptance How Analyze reqs, design, data, code, tests, etc. Derive tests from code, data, design Derive tests from reqs, design, quality risks Tools, data, and cases can be shared: encourage cross-pollination of techniques We’ll discuss techniques for each type of testing in upcoming sections
  125. 125. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 125 ISO 9126 Quality Characteristics/Subcharacteristics Functionality: suitability, accuracy, interoperability, security, compliance Reliability: maturity (robustness), fault-tolerance, recoverability, compliance Usability: understandability, learnability, operability, attractiveness, compliance Efficiency: time behavior, resource utilization, compliance Maintainability: analyzability, changeability, stability, testability, compliance Portability: adaptability, installability, co-existence, replaceability, compliance
  126. 126. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 126 Terms to Remember Automation Behavior-based testing Black box or specification-based testing Code-based, white box, or structural testing Code coverage Confirmation testing Functional test Interoperability testing Load testing Maintainability testing Performance testing
  127. 127. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 127 Terms to Remember Portability testing Regression test Reliability testing Security testing Stress testing Test suite Usability testing Use case testing
  128. 128. Chapter 2: Testing throughout the Software Lifecycle Section 3: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  129. 129. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 129 Exercise: Omninet Test Types Referring back to the levels you selected in the previous exercise, list for each level the test targets or types you would include in each level. How does the lifecycle model you selected in a previous exercise affect the amount of regression testing? How does the lifecycle model you selected in a previous exercise affect the amount of confirmation testing? Discuss.
  130. 130. Chapter 2: Testing throughout the Software Lifecycle Section 4: Maintenance testing Test Engineering Foundation Essential Knowledge for Test Professionals
  131. 131. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 131 Maintenance Testing Key concepts Reasons for maintenance testing Maintenance testing versus new application testing Role of regression testing and impact analysis Terms to remember
  132. 132. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 132 Reasons for Maintenance Three typical triggers for maintenance and maintenance testing Modification: enhancements, bug fixes, operational environment changes, patches Migration: a new supported environment Retirement: end-of-life of a subsystem or entire system triggers replacement Maintenance testing addresses the change itself--and what wasn’t changed and shouldn’t change
  133. 133. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 133 Testing Maintenance Releases Regression is the big risk for maintenance releases Some organizations try to put a major release worth of features into a short maintenance release Time to develop new tests is scarce Large project test estimation rules-of-thumb fail
  134. 134. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 134 Regression Regression (misbehavior of a previously correct function, attribute, or feature due to a change) types: Local (fix creates new bug) Exposed (fix reveals existing bug) Remote (fix in one area breaks something in another area) Regression can affect new and existing features
  135. 135. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 135 Regression Strategy 1: Repeat All Tests If our tests are aligned with quality, then repeating all tests should find most important regressions Automation is the only practical means for large complex system We’ll cover automation more thoroughly in a later section
  136. 136. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 136 Regression Strategy 2: Repeat Some Tests Often, full automation is impossible Select some tests to repeat Traceability Change analysis Risk analysis Use cross-functional tests for “accidental regression testing” Can use code coverage to assess level of risk
  137. 137. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 137 Three Other Regression Strategies Release more slowly Release every six months rather than every month Partial or even full repetition can increase coverage on bigger releases Combine emergency patches with slower release process to allow for flexibility while keeping regression risk low Use customer or user testing Beta for mass-market software Pilot, staged or phase, parallel release for IT
  138. 138. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 138 Terms to Remember Maintenance testing Impact analysis Modifications Migration Retirement
  139. 139. Chapter 2: Testing throughout the Software Lifecycle Section 4: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  140. 140. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 140 Exercise: Omninet Maintenance Assume you can fully automate the testing of the Omninet kiosk and call center. If a post-release change is made to the call-center user interface that does not affect functionality, what would you retest? Assume you have not automated the testing of the Omninet kiosk and call center. If the same change is made, what would you retest? What if the change did affect functionality? Discuss.
  141. 141. Chapter 3: Static Techniques Test Engineering Foundation Essential Knowledge for Test Professionals
  142. 142. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 142 3. Static Techniques 1. Reviews and the test process 2. Review process 3. Static analysis by tools
  143. 143. Chapter 3: Static Techniques Section 1: Reviews and the test process Test Engineering Foundation Essential Knowledge for Test Professionals
  144. 144. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 144 Reviews and the Test Process Key concepts Software work products and static techniques The importance and value of static techniques The difference between static and dynamic techniques Terms to remember
  145. 145. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 145 Static Testing Reviews and tools Reviews range from informal to very formal Tools can perform some types of static tests Static techniques can be used for requirements and designs, plus code, database schemas, documentation, tests… Models and prototypes A diagram of a complex system can often reveal design problems that can hide in words An ugly diagram means lots of bugs Test cases and data Test analysis and design based on requirements and design specs is a form of structured review Test analysis and design often reveals problems
  146. 146. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 146 Static Tools Static analysis Problematic wording: Spell/grammar checkers Dangerous programming: J-Test, Safer C, lint… Measurement: Complexity analysis System simulations General Purpose System Simulator Performance modeling/operations research tools Spreadsheets
  147. 147. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 147 Reviews: Costs and Benefits Costs Time required to perform reviews Effort required to gather and analyze metrics Process improvement Benefits Shorter schedules (due to efficient bug removal) Shorter testing periods and lower testing costs Developer productivity Improved quality of product (which reduces downstream costs) Bottom line: Reviews of all kinds are proven, high- return techniques for improving quality
  148. 148. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 148 Relating Static and Dynamic Testing Similarities Seek to identify defects Work best when a broad cross-section of stakeholders are involved Save the company money and time Differences Each technique can find different types of defects more effectively and efficiently Static techniques find defects rather than failures
  149. 149. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 149 Terms to Remember Reviews Dynamic testing Static analysis
  150. 150. Chapter 3: Static Techniques Section 1: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  151. 151. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 151 Exercise: Omninet Static Testing Do you see reviews and static analysis as useful for the Omninet project? If so, what kinds of problems do you think these reviews and static analyses would locate? What kinds of problems might they not locate? Discuss.
  152. 152. Chapter 3: Static Techniques Section 2: Review process Test Engineering Foundation Essential Knowledge for Test Professionals
  153. 153. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 153 Review Process Key concepts Phases, roles and responsibilities of a typical formal review The differences between different types of review The factors for successful reviews Terms to remember
  154. 154. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 154 Types of Reviews Informal: no real process (hallway chats, buddy tests, pair programming), yet useful, cheap, popular Peer: documented and defined defect removal process, involving peers and technical experts but not managers Walkthroughs: author “walks” peers “through” the document or code Inspections: a trained moderator (other than the author) leads the inspection team (with defined roles) through a formal inspection process (rules, checklists, entry and exit criteria), which includes gathering defect removal metrics
  155. 155. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 155 Incompleteness and ambiguity can hide the real meaning of the specifications Agreement and uniform understanding of the specifications Consensus and Understanding Long before any code exists, the specification must be handed to an outside testing group to be scrutinized for completeness and clarity. As [V.A.] Vyssotsky [of Bell Lab’s Safeguard Project] says, the developers themselves cannot do this: “They won’t tell you they don’t understand it; they will happily invent their way through the gaps and obscurities.” – Fred Brooks The Mythical Man-Month 1975
  156. 156. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 156 A Generic Review Process 1. Planning 2. Kick-off 3. Preparation 4. Review meeting 5. Rework/repair 6. Follow-up The details of the review process depend on the specific review type used on the project Includes estimating and planning, training participants, etc. Follow-up includes on individual items as well as overall process improvement analysis, evaluation of defect (bug) removal at phase exit reviews (exit meetings), etc. These steps of the process repeat per each item reviewed. Preparation is usually one to two hours alone. Meeting is one to two hours together. Rework/repair is fixing the bugs found.
  157. 157. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 157 Roles and Responsibilities Moderator: Lead the review meetings Scribe or secretary: Gather information on findings Author: Describe, explain, answer questions on item Reviewer/inspector: Find defects (bugs) in item Manager: Plan, arrange resources and training, support, analyze process metrics In some cases, one person may play multiple roles Authors sometimes act as moderators One of the reviewers can act as the secretary The specifics are determined by the type of review
  158. 158. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 158 Suggestions for Successful Reviews Provide training Review the product, not the producer Set and follow agenda and objectives Limit debate Focusing on finding, not fixing, problems Take written notes Limit and carefully select participants Insist on preparation (e.g., by having people submit notes) Develop a checklist for each type of item that is reviewed Review the reviews Use the right techniques Ensure management support Learn and get better!
  159. 159. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 159 Common Requirements and Design Bugs Ambiguities: What exactly does that mean? E.g.: System shall allow user to read ISP e-mail What ISPs? What size e-mails? Attachments? Incompleteness: Okay, and then what? E.g.: Upon three invalid passwords, system shall lock user’s account… For how long? How to unlock? Who can unlock? Untestability: How can I check this item? E.g.: System shall provide 100% availability No known test technique to demonstrate perfect availability Excessive dependencies, coupling and complexity Look for ugly design diagrams and confusing requirements
  160. 160. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 160 IEEE 1028 Standard for Software Reviews 1. Overview Purpose, scope, conformance, organization, application 1. References 2. Definitions 3. Management reviews Responsibilities, inputs/outputs, entry/exit criteria, procedures 1. Technical reviews Responsibilities, inputs/outputs, entry/exit criteria, procedures
  161. 161. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 161 IEEE 1028 Standard for Software Reviews 6. Inspections Responsibilities, inputs/outputs, entry/exit criteria, procedures, data collection, process improvement 6. Walkthroughs Responsibilities, inputs/outputs, entry/exit criteria, procedures, data collection, process improvement 6. Audits Responsibilities, inputs/outputs, entry/exit criteria, procedures
  162. 162. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 162 Terms to Remember Review process Review meeting Moderator/ inspection leader Reviewer Scribe (or secretary) Inspection Entry criteria Exit criteria Formal review
  163. 163. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 163 Terms to Remember Walkthrough Informal review Peer or technical review Metrics Kick-off
  164. 164. Chapter 3: Static Techniques Section 2: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  165. 165. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 165 Exercise: Omninet Requirements Review Break into teams of three to five people. Select a review technique from among those discussed. If your selected technique involves specific roles, assign the roles. Review the Omninet Marketing Requirements Document. Discuss your team’s findings.
  166. 166. Chapter 3: Static Techniques Section 3: Static analysis by tools Test Engineering Foundation Essential Knowledge for Test Professionals
  167. 167. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 167 Static Analysis by Tools Key concepts The objective of static analysis versus dynamic testing Typical defects and errors identified by static analysis Typical benefits of static analysis List typical code and design defects identified by static analysis tools Terms to remember
  168. 168. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 168 Static Analysis and Dynamic Testing Like dynamic testing, static analysis looks for defects in software source code and software models Unlike dynamic testing, static analysis is performed without actually executing the system Static analysis involves analysis of the system or its components by a tool, while dynamic testing does not always involve tools Static analysis can find defects that are hard to find or isolate in dynamic testing Examples include maintainability issues, unsafe pointer use Isolation is easier because you find the bug, not the symptom
  169. 169. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 169 What Can We Analyze? Program code (e.g. control flow and data flow) Models of the program (e.g., simulations) Generated output such as HTML and XML Requirements and design documents
  170. 170. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 170 Benefits of Static Analysis Early and cheaper detection of bugs (before test execution starts) Warnings about where bug clusters might exist, due to dangerous programming, high complexity, etc. Location of bugs dynamic testing might miss Detection of dependencies and inconsistencies in software models (e.g., such as link problems in Web pages) Improved maintainability of code and design Prevention of defects based on metrics gathered and lessons learned from analysis
  171. 171. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 171 Typical Static Analysis Bugs Referencing a variable with an undefined value Inconsistent interface between modules and components Variables that are never used Unreachable (dead) code Programming standards violations Security vulnerabilities Syntax violations of code and software models
  172. 172. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 172 Using Static Analysis Tools Typical users are… Programmers, often during component and integration testing Designers and system architects during design During initial introduction against an existing system, static analysis tools may produce a large number of warning messages Compilers do some static analysis, but many sophisticated tools are available
  173. 173. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 173 Terms to Remember Static analysis Compiler Complexity Control flow Data flow
  174. 174. Chapter 3: Static Techniques Section 3: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  175. 175. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 175 Exercise: Omninet Static Analysis What kind of static analysis would you suggest for Omninet? Would you use static analysis in areas that would be subject to later testing? Discuss.
  176. 176. Chapter 4: Test Design Techniques Test Engineering Foundation Essential Knowledge for Test Professionals
  177. 177. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 177 4. Test Design Techniques 1. Identifying and designing test cases 2. Categories of test design techniques 3. Specification-based or black-box techniques 4. Structure-based or white-box techniques 5. Experience-based techniques 6. Choosing test techniques
  178. 178. Chapter 4: Test Design Techniques Section 1: Identifying and designing test cases Test Engineering Foundation Essential Knowledge for Test Professionals
  179. 179. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 179 Identifying and Designing Test Cases Key concepts • Base testing on risk analysis • Determine level of risk with likelihood and impact Specify test designs, cases, procedures Write test cases Relate test cases and test procedures Develop test execution schedule Terms to remember
  180. 180. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 180 Phases of Test Development Test development often proceeds in phases (Quality risk) analysis High-level test design Low-level test design (implementation) External inputs used to create internal deliverables (testware)
  181. 181. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 181 Analysis
  182. 182. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 182 High-Level Design These arrows show the relationships between the risk categories and the test suites. We’ll capture this relationship information in the form of what’s called traceability.
  183. 183. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 183 Low-Level Design (Implementation) Each risk to be mitigated via testing will have one or more test cases associated with it. (I’ve not shown all traces, to avoid clutter.)
  184. 184. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 184 What’s a Quality or Product Risk? Risk The possibility of a negative or undesirable outcome The likelihood of a risk becoming an outcome is… >0, <1 in the future… 0 or 1 in the past Quality or product risk The possibility that the system will fail to satisfy customers, users, or other stakeholders A family of possible bugs are behind quality risks
  185. 185. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 185 How Can We Analyze Quality Risks? Informal ISO 9126 Failure Mode and Effect Analysis Start with the classic quality risk categories Start with six main quality characteristics Start with categories, char- acteristics, or subsystems Functionality, states and transactions, capacity and volume, data quality, error handling and recovery, performance, standards and localization, usability, etc. Functionality, Reliability, Usability, Efficiency, Maintainability, Portability (FRUEMP), then decompose into key subcharacteristics for your system Key stakeholders list possible failure modes, predict their effects on system, user, society, etc., assign severity, priority, and likelihood, then calculate risk priority number (RPN) Set priority for testing each quality risk with key stakeholders Set priority for testing each subcharacteristic with key stakeholders Stakeholders use RPN to guide appropriate depth and breadth for testing Regardless of technique the keys are cross-functional stakeholder participation, consensus, and a best-possible-outcome outlook
  186. 186. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 186 Quality Risk Tech. Risk Bus. Risk Risk Pri. # Extent of Testing Tracing Risk Category 1 Risk 1 Risk 2 Risk n Quality risks are potential system problems which could reduce user satisfaction A hierarchy of risk categories can help organize the list and jog your memory. 1 = Very high 2 = High 3 = Medium 4 = Low 5 = Very low Technical risk: Likelihood of the problem Business (operational) risk: Impact of the problem 1-5 = Extensive 6-10 = Broad 11-15 = Cursory 16-20 = Opportunity 21-25 = Report bugs Risk priority number: Aggregate measure of problem risk The product of technical and business risk, from 1-25. Tracing information back to requirements, design, or other risk bases
  187. 187. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 187 Tips for Risk Analysis Use a cross-functional brainstorming team Identify the risk items, then assign the level of risk Only separate risk items when necessary to distinguish between different levels of risk Consider technical and business risk Technical risk: likelihood of problem, impact of problem on system Business risk: likelihood of usage, impact of problem on users Follow up and re-align risk analysis, testing, and the project at key project milestones
  188. 188. Chapter 4: Test Design Techniques Section 1: Exercise 1 Test Engineering Foundation Essential Knowledge for Test Professionals
  189. 189. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 189 Exercise: Omninet Risk Analysis Based on your reading of the Omninet Marketing Requirements Document, the Omninet System Requirements Document, and your experience with testing and bugs, perform a risk analysis for Omninet Discuss.
  190. 190. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 190 IEEE 829 Test Design Specification The test design specification describes a collection of test cases at a high level, and includes the following sections Test design specification identifier Features to be tested (in this test suite) Approach refinements (specific techniques, tools, etc.) Test identification (tracing to test cases in suite) Feature pass/fail criteria (e.g., test oracle, test basis, legacy systems, etc.) This collection of test cases often called a test suite Sequencing (test suites and cases within suites) often driven by risk and business priority and affected by project constraints, resources, and progress
  191. 191. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 191 IEEE 829 Test Case Specification A test case specification describes the details of a test case, and includes the following sections Test case specification identifier Test items (what is to be delivered and tested) Input specifications (user inputs, files, etc.) Output specifications (expected results, including screens, files, timing, etc.) Environmental needs (hardware, software, people, props…) Special procedural requirements (operator intervention, permissions, etc.) Intercase dependencies (if needed to set up preconditions) In practice, test cases vary significantly in effort, duration, and number of test conditions covered
  192. 192. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 192 IEEE 829 Test Procedure Specification A test procedure specification describes how to run one or more test cases, and includes the following sections Test procedure specification identifier Purpose (e.g., which tests are run) Special requirements (skills, permissions, environment, etc.) Procedure steps (logging, set up, start, proceed [steps themselves], measurement of results, shutdown/suspension, restart [if needed], stop, wrap up/tear down, contingencies) Test procedures are often embedded in test cases A test procedure is sometimes referred to as a test script, and may be manual or automated
  193. 193. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 193 Test Case Coverage (Traceability) Measure or correlate tests against areas of concern Used as a way to measure and enhance the tests Practical types Requirements specifications Design specifications Functional areas Quality risks Configurations This is a commonly-used technique to ensure thorough test coverage One technique Use spreadsheet List test cases and coverage area to measure 0: none; 1: indirect; 2: direct Test Case 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 2.1 2.2 2.3 2.4 2.5 Total Spec 1.1 1 1 1 1 2 6 1.2 2 2 1 1 2 8 1.3 0 1.4 2 2 4 1.5 1 1 1 1 4 1.6 2 2 1.7 2 1 2 2 7 1.8 2 1 3 1.9 1 1 2 4 1.10 1 1 1 1 4 Total 5 4 2 5 4 3 3 4 2 2 7 1 0
  194. 194. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 194 Terms to Remember Priority Risk Risk identification Risk analysis Risk control Risk-based testing Product risk Test case Test condition Test data
  195. 195. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 195 Terms to Remember Test case specification Test design specification Test procedure specification Test script Traceability
  196. 196. Chapter 4: Test Design Techniques Section 1: Exercise 2 Test Engineering Foundation Essential Knowledge for Test Professionals
  197. 197. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 197 Exercise: Omninet Test Design Specification Based on your quality risk analysis for Omninet, outline a set of test suites to address the important areas of risk. For each test suite, list briefly: What the test would cover How you would recognize passing or failing tests Establish the relationship between your test suites and the risks they will cover. Based on your risk analysis, how would you sequence the test suites? What other considerations would affect the sequencing of test suites? Discuss.
  198. 198. Chapter 4: Test Design Techniques Section 2: Categories of test design techniques Test Engineering Foundation Essential Knowledge for Test Professionals
  199. 199. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 199 Categories of Test Design Techniques Key concepts Reasons for specification-based (black box), structure-based (white box), and experience-based tests Common black box and white box techniques Characteristics and differences between specification-based testing, structure-based testing, and experience-based testing Terms to remember
  200. 200. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 200 Three Types of Test Design Techniques Specification-based (black box), functional and non- functional Create tests primarily by analysis of the test basis Look for bugs in the way the system behaves Structure-based (white box) Create tests primarily by analysis of the structure of the component or system Look for bugs in the way the system is built Experience-based (attacks, checklists, exploratory) Create tests primarily based on understanding of the system, past experience, and educated guesses about bugs Look for bugs in the places other systems have bugs
  201. 201. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 201 Specification-Based or Black Box Common elements include: Formal or informal models used to specify the problem to be solved, the software or its components Test cases derived systematically from these models Examples include: Equivalence partitioning and boundary value analysis State transition diagrams Decision tables
  202. 202. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 202 Structure-Based or White Box Common elements include: System structure (e.g., code, design, etc.) used to derive the test cases, for example code and design The extent of structural coverage can be measured for existing other test cases Further test cases can be derived systematically to increase coverage Examples include: Statement coverage Branch coverage
  203. 203. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 203 Experience-Based Common elements include: The knowledge and experience used to derive test cases Can consider knowledge and experience of the software, its usage and its environment or… …knowledge about historical and likely defects and their distribution Examples include: Attacks Checklists Exploratory
  204. 204. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 204 Terms to Remember Black box or specification-based techniques Experience-based techniques White box or structure-based techniques
  205. 205. Chapter 4: Test Design Techniques Section 2: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  206. 206. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 206 Exercise: Omninet Test Techniques Refer to your outline of test suites for Omninet. For each test suite, identify whether one, two, or all three of the following categories of test techniques would be useful in designing test cases: Specification-based (black box) Structure-based (white box) Experience-based Discuss.
  207. 207. Chapter 4: Test Design Techniques Section 3: Specification-based or black box techniques Test Engineering Foundation Essential Knowledge for Test Professionals
  208. 208. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 208 Specification-based Techniques Key concepts Writing test cases from given software models using equivalence partitioning, boundary value analysis, decision tables, and state transition diagrams The main purpose of each technique and how coverage may be measured Use case testing Terms to remember
  209. 209. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 209 Equivalence Partitioning Divide the inputs, outputs, behaviors, and environments into classes you think will be handled equivalently Define at least one test case in each partition, or use boundary values in partitions that are ranges Can use marketing info to favor classes Ex: Testing supported printers Physical interface: Parallel, serial, USB 1.1, USB 2.0, infrared, Firewire, SCSI, others? Logical interface: Postscript, HPPL, ASCII, others… Image application: Laser jet, ink jets, bubble jets, dot matrix, line printers, letter quality, pen plotters, others…
  210. 210. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 210 Boundary Value Analysis A refinement of equivalence partitioning that selects the edges or end-points of each partition for testing Equivalence partitioning looks for bugs in the code that handles each equivalent class Boundary values also look for bugs in the definitions of the edges Can only be used when the elements of the equivalence partition are ordered Non-functional boundaries (capacity, volume, etc.) can be used for non-functional testing, too
  211. 211. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 211 Integer “How many items would you like to order?”

×