Test engineering foundation (v2.01)
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Test engineering foundation (v2.01)

  • 1,330 views
Uploaded on

ISTQB ISEB Certification

ISTQB ISEB Certification

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,330
On Slideshare
1,329
From Embeds
1
Number of Embeds
1

Actions

Shares
Downloads
68
Comments
0
Likes
1

Embeds 1

http://www.techgig.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Test Engineering Foundation Essential Knowledge for Test Professionals Rex Black RBCS, Inc. 31520 Beck Road Bulverde, TX 78163 USA Phone: +1 (830) 438-4830 Fax: +1 (830) 438-4831 www.rexblackconsulting.com rex_black@rexblackconsulting.com
  • 2. Introductions, Objectives, and Overview Test Engineering Foundation Essential Knowledge for Test Professionals
  • 3. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 3 The Course Who is this presenter? What is this course? What are the hours? breaks? Must I do exercises? Do I have homework? Can I get certified? May I use my cell phone or read e-mail during the course? Any others???
  • 4. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 4 The Materials These slides following the Foundation Syllabus 2005 Terms come from the ISTQB Glossary (v1.1) In practice terminological variation is wide Not all terms in syllabus are defined in glossary Reference materials, glossary, and study aids for those pursuing certification For use in exercises Omninet Marketing Requirements Document Omninet System Requirements Document Solutions for the exercises A notepad on which to do the exercises
  • 5. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 5 ISTQB Foundation Syllabus 2005 Developed by a team of eight authors spanning seven countries: Thomas Müller (chair), Rex Black, Sigrid Eldh, Dorothy Graham, Klaus Olsen, Maaret Pyhäjärvi, Geoff Thompson and Erik van Veendendaal Forty primary reviewers spanning nine countries Final review and approval by 18 National Boards Distills over 1,000 person-years of experience The ISTQB and the authors are the source of the syllabus (copyright © 2005 by the authors and ISTQB) which is used by permission as the basis for this course
  • 6. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 6 The Main Topics Covered 1. Fundamentals of testing 2. Testing throughout the software lifecycle 3. Static techniques 4. Test design techniques 5. Test management 6. Tool support for testing
  • 7. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 7 The Exercises In many of the exercises, you are working as a tester on the Omninet project Omninet is a project to deploy a network of public access Internet kiosks in places like malls, theaters, and other public places On this realistic project, you will have a chance to apply many of the techniques we discuss Other exercises have been added to illustrate specific points, too
  • 8. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 8 Your Course Objectives This material is designed to help you become a more effective and efficient test professional, and to help you obtain ISTQB certification Please spend the next few minutes writing down what you’d like to get out of the course Use the following page for your objectives Don’t let the class end with an objective unfulfilled!
  • 9. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 9 Course Objectives
  • 10. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 10 It’s Your Course Please join in  Ask questions  Make comments  Share experiences  Second opinions and disagreements welcome Why are you here? What do you want to learn, discuss, and teach? …as well as this. The best sessions have a lot of this...
  • 11. Chapter 1: Fundamentals of Testing Test Engineering Foundation Essential Knowledge for Test Professionals
  • 12. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 12 1. Fundamentals of Testing 1. Why is testing necessary? 2. What is testing? 3. General testing principles 4. Fundamental test process 5. The psychology of testing
  • 13. Chapter 1: Fundamentals of Testing Section 1: Why is testing necessary? Test Engineering Foundation Essential Knowledge for Test Professionals
  • 14. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 14 Why is Testing Necessary? Key concepts How bugs can cause harm Bugs and their effects The necessity of testing The role of testing in quality assurance Terms to remember
  • 15. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 15 The Threat of Bugs Company Damaged reputation for quality High or unpredictable maintenance costs Unexpected delays in release cycles Lack of confidence in system Lawsuits Environment Pollution Waste People, societies, and states Lost jobs Lost lives Lost rights Lost missions Lost wars
  • 16. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 16 Where Bugs Come From and What Bugs Do People put bugs (defects) into the system Requirements and design specifications Code (business logic and user interface) Documentation (electronic and hard copy) When the implementation of these bugs are executed, failures occur If these failures are visible to customers, users, or other stakeholders, dissatisfaction with system quality results
  • 17. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 17 From Whence the Bugs and Failures? Bugs occur due to… programmer, analyst, and other individual contributor (including tester) fallibility time pressure complexity of the code, infrastructure, or problem to be solved changing and meshing technologies many system interactions. Failures occur due to bugs and… environmental conditions misuse (deliberate and accidental)
  • 18. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 18 Testing to Manage Quality Risks Risks and constraints for a software project Features: Right set Schedule: Quickly enough Budget: Acceptably cheap Quality: Ready for customers/release/next step Testing provides the information to guide the project, reduce and manage the risks, and repair the important problems Testing may also address compliance, contractual, and regulatory needs
  • 19. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 19 What Does “Quality” Mean to You? “Fitness for use” vs. “Conformance to requirements” Testing and quality Tests gives confidence where they find few bugs Passing tests reduce the level of quality risk Failing tests provide a chance to improve quality The test set gives an assessment of quality What are the important quality characteristics for your system? Are you testing them (enough)? Testing, quality assurance, and quality improvement Ideally, testing is part of a larger quality assurance strategy for a project For future projects, analyze the root causes of defects found on current projects and take steps to reduce their incidence
  • 20. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 20 Terms to Remember Bug, defect, or fault Error or mistake Failure Quality Risk Software Test Test case
  • 21. Chapter 1: Fundamentals of Testing Section 1: A diversion Test Engineering Foundation Essential Knowledge for Test Professionals
  • 22. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 22 What Happens When Quality is Lacking
  • 23. Chapter 1: Fundamentals of Testing Section 2: What is testing? Test Engineering Foundation Essential Knowledge for Test Professionals
  • 24. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 24 What is Testing? Key concepts Common objectives of testing The purpose of testing… •in software development, maintenance, and operations… •…to find defects, provide confidence and information, and prevent defects Terms to remember
  • 25. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 25 Testing Objectives Typical general testing objectives Finding bugs and providing programmers with the information they need to fix important bugs Gaining confidence about the level of quality of the system Preventing defects (through early involvement in reviews and advanced test design) Provide information about the most important aspects of the quality of the system under test Help management understand system quality Can you think of others for your projects?
  • 26. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 26 Unit/Component Test Find bugs in the individual pieces of the system under test before the pieces are fully integrated into the system Integration/String Test Find bugs in the relationships and interfaces between pairs and groups of components in the system under test as the pieces come together System Test Find bugs in the overall and particular behaviors, functions, and responses of the system under test as a whole Acceptance/Pilot Test Demonstrate that the product is ready for deployment/release or to assess quality and give information on the risk of deployment/release Maintenance Test Check for errors introduced during development of the changes Operational Test Assess non-functional system characteristics such as reliability or availability Test Phases and Objectives
  • 27. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 27 Effectiveness and Efficiency Effective: Producing a decided, decisive, or desired result; impressive To be effective testers, we must select the appropriate objective and desired results Efficient: Productive of desired effect; especially productive without waste To be efficient testers, we must allocate resources (time and money) appropriately These terms are most meaningful in the context of the entire development or maintenance process, not just the testing process.
  • 28. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 28 Testing vs. Debugging Testing finds failures that are caused by bugs Debugging… …identifies the root cause of a bug… …repairs the code… …and checks that the defect is fixed correctly Confirmation testing ensures the fix resolves the observed failure Different responsibilities: Testers test Programmers debug
  • 29. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 29 Find-Debug-Confirm
  • 30. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 30 Beyond Test Execution Testing is not just running tests against a running system Other test activities, before and after execution, include… Planning and control Choosing test conditions Design test cases Checking test results Evaluating exit criteria Test result reporting Closure/end-of-test tasks We will revisit this later
  • 31. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 31 Terms to Remember Code Debugging Software development Review Requirement Test case Test objective Testing Test basis
  • 32. Chapter 1: Fundamentals of Testing Section 2: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 33. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 33 Exercise: The Triangle Tests A program accepts three integers representing the lengths of a triangle’s sides. It outputs “scalene” (no equal sides), “isosceles” (two equal sides), or “equilateral” (three equal sides). Write an effective (finds common bugs) and efficient (as few tests as possible) set of test cases. Usually, a test case consists of tester action, data, and expected result, but here the action is generally “input data”. Discuss.
  • 34. Chapter 1: Fundamentals of Testing Section 3: General testing principles Test Engineering Foundation Essential Knowledge for Test Professionals
  • 35. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 35 General Testing Principles Key concepts Testing reveals the presence of bugs Impossibility of exhaustive testing Benefits of early testing Lumpiness of bugs: defect clustering Pesticide paradox Testing should adapt to specific needs Absence-of-errors fallacy Terms to remember
  • 36. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 36 Testing Reveals Presence of Bugs: A Parable You have a beautiful vegetable garden, but one day you see eaten leaves on the tomatoes “Oh no,” you think, “I have hornworms!” You know you have bugs in your garden If you had not seen the symptoms, could you be sure you had no bugs? Some bugs are easy to spot, others aren’t Testing can reveal the presence of bugs, but cannot prove their absence
  • 37. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 37 Mission Impossible: Exhaustive Testing “Just make sure the software works before we ship it…” This charter is demonstrably impossible The execution paths in non-trivial software are almost infinite Large dataflows separated across space (features) and time (static data) Slight changes can cause regressions which are not linear to the size of the change Myriad usage profiles and field configurations, some unknown and some unknowable
  • 38. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 38 Defusing Exhaustive Testing Expectations Exhaustive testing as a way to prove the software works is a common (mis)expectation Bad expectations create problems for test professionals and test teams Unachievable high demands on test group Perception of incompetence when these demands aren’t met Testers must be ready to communicate (in words the project stakeholders will understand) how testing can contribute
  • 39. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 39 Benefits of Early QA and Testing The cost of a bug tends to increase as the project continues Most of the costs associated with pre-release bugs tend to be associated with the effort required to remove them, so the higher cost means longer schedules The more bugs enter a quality assurance or test activity, the more bugs will escape from that activity
  • 40. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 40 Click here to see the answers
  • 41. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 41 Defect Clustering Studies have long shown and continue to show the unequal distribution of bugs MVS: 38% of field bugs in 4% of modules IMS: 57% of field bugs in 7% of modules Capers Jones reports that the excessive presence of error-prone modules causes a 50% reduction of productivity in software maintenance
  • 42. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 42
  • 43. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 43 Pesticide Paradox Return to your vegetable garden You spray pesticide on your garden, and the hornworms die, but the pesticide is not effective against all bugs Just as pesticides become less effective, so do tests Functional tests can’t find performance bugs Try new test techniques
  • 44. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 44 These two pictures compare efficiencies of the top 11 bug-finding test suites (out of 27). The top graph shows the first time the set of test suites was run. The bottom graph shows the results for the fifth time that same set of test suites was run. The average test suite efficiency in the first pass, 0.4, is used as the axis crossing point in both graphs. With one exception, the same test suites are in the top 11 test suites, which shows bug clustering again. However, all test suites are less effective after five executions.
  • 45. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 45 Absence-of-Errors Fallacy Finding and fixing many bugs does not guarantee user, customer, and/or stakeholder satisfaction Many low-defect products have failed in the market place Successful projects balance competing forces in terms of features, schedule, budget, and quality
  • 46. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 46 Testing Should Adapt to Needs Different projects, organizations, and products have different testing needs Best testing practices exist (and are discussed in this course) but you need to tailor them to your project Failure to adapt the test team and its methods to these needs is a common result of dissolution of test teams
  • 47. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 47 Terms to Remember Exhaustive testing
  • 48. Chapter 1: Fundamentals of Testing Section 3: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 49. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 49 Exercise: Test Principles Observed (and Not) Think back on a recent project. Note which of the principles that you can recall were observed. How about principles that you can recall were not observed (i.e., violated)? Discuss.
  • 50. Chapter 1: Fundamentals of Testing Section 4: Fundamental test process Test Engineering Foundation Essential Knowledge for Test Professionals
  • 51. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 51 Fundamental Test Process Key concepts Plan, prepare, perform, perfect Planning and control Analysis and design Implementation and execution Evaluating test exit criteria and reporting Test closure activities Terms to remember
  • 52. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 52 Test Processes: Critical Testing Processes The test engineer’s role focuses on some of these activities, not all, depending on how roles are defined. However, the effective and efficient test engineer must understand how the test process works and how it fits into the overall project from a big picture perspective.
  • 53. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 53 ISTQB Fundamental Test Process Another way to think of the test process is in terms of the following steps Planning and control Analysis and design Implementation and execution Evaluating test exit criteria and reporting Test closure activities These steps may overlap or take place concurrently
  • 54. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 54 Planning and Control Planning Determine test scope, risks, objectives, strategies Determine required test resources Implement the test strategies Schedule test analysis and design Schedule implementation, execution and evaluation of tests Determine the test exit criteria Control Measure and analyze results Monitor and document progress, coverage and test exit criteria Initiate corrective actions Make decisions
  • 55. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 55 Analysis and Design Analysis Review the test basis (e.g., requirements or design specifications, network/ system architecture, quality risks) Identify test conditions, test requirements, or test objectives and required test data based on analysis of test items, its specification, behavior and structure Design Select specific combinations of test data, actions, and expected results to cover the test basis Evaluate testability of the requirements and system Design the test environment Identify any required infrastructure and tools
  • 56. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 56 Implementation and Execution Implementation Develop and prioritize test cases, create test data, write test procedures Prepare test harnesses and write automated test scripts Organize test suites and sequences of test cases for efficient test execution Verify that the test environment has been set up correctly Execution Execute test cases (manual or automated) Log test results, and the versions of the software under test, test tools and the testware Compare actual and expected results Report and analyze incidents Repeat corrected and/or updated tests Run confirmation and/or regression tests
  • 57. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 57 Exit Criteria, Reporting, and Closure Exit and reporting Check test logs against the test exit criteria specified in test planning Assess if more tests are needed or if the exit criteria specified should be changed Write a test summary report for stakeholders Closure Confirm test deliverables, final resolution or deferral of bug reports, and the acceptance of the system Finalize and archive testware, test environment and test infrastructure Deliver testware to the maintenance organization Perform a retrospective to capture improvements for future releases, projects, and test processes
  • 58. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 58 Terms to Remember Testware Test plan Test strategy Test execution Test basis Test log (Test) exit criteria Test summary report Test coverage Test condition
  • 59. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 59 Terms to Remember Test data Test input Confirmation testing Regression testing Incident
  • 60. Chapter 1: Fundamentals of Testing Section 4: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 61. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 61 Exercise: Test Steps and Tasks Performed Think back on a recent project. Note which of the steps and tasks of the ISTQB test process were carried out. Note which of the steps and tasks of the ISTQB test process were not carried out. Note whether some steps overlapped or were completely parallel. Discuss.
  • 62. Chapter 1: Fundamentals of Testing Section 5: The psychology of testing Test Engineering Foundation Essential Knowledge for Test Professionals
  • 63. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 63 Psychology of Testing Key concepts Psychological factors for testing success Clear objectives Self-testing and independent testing Respectful communication Programmer and tester outlooks for success Terms to remember
  • 64. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 64 Certainty Versus Progress Thorough research can lead to indisputable bug reports, unquestionably correct test cases, etc. Such certainty is intellectually satisfying and reduces rejected and irreproducible bug reports But certainty also can consume too much time and effort for the payoff and delay forward progress Test projects adapt to schedule pressures or fail Testing computers is test engineering Not a search for truth—that’s science Engineering is making useful objects for customers
  • 65. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 65 Test Result Misinterpretation On the one hand we could report correct behavior as a bug assign excessively high severity or priority otherwise overstate bug significance On the other hand we could fail to detect or report incorrect behavior assign excessively low severity or priority otherwise understate bug significance Some tips to avoid these errors 1. Have testers take breaks so they don’t miss important events 2. Automate where practical 3. Define expected results as clearly as possible 4. Assign the right testers to each test execution task 5. Use peer reviews for test execution and bug reports However…recognize that perfect test execution takes too long. Define “good enough” for testing and live with that decision
  • 66. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 66 Professional Pessimism Explore depressing possibilities of failure Anticipate the worst possibilities in order to achieve best obtainable product quality Not adversarial, but a different outlook than the programmers Remember: to assume nothing will fail during testing denies the entire history of computing Caveat: not a license to offend  Don’t target programmers with reports or take glee in failure Challenge: to be positive, pleasant, and the bearer of bad news, all at once Pierre has the pessimism, but perhaps not professionalism?
  • 67. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 67 Bad News == Bad Guy? Testers are sometimes on receiving end of emotions brought on by news of project problems The key is professional pessimism Don’t take on the role of “Lone Champion of Quality” • Product quality is a business decision • Quality risks are weighed against other risks Don’t second-guess development’s bug-fix efforts Never gloat, even if you were right and everyone else was wrong Juan de Mariana: “The greatest of follies is to exert oneself in vain, and to weary oneself without winning anything but hatred.” Grim reaper or friendly guide?
  • 68. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 68 Balanced Curiosity Balance need for thoroughness in any one area with need to cover many areas in a short time Effective and efficient test engineers have a talent for spending time where the bugs are Effective and efficient test engineers can do thorough bug isolation quickly Ineffective and inefficient test engineers Write tests to search for unlikely, low-impact bugs Spend hours researching trivial bugs
  • 69. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 69 Two types of focus problems Pursuing issues narrow-mindedly, losing sight of more important priorities Getting distracted from key tasks Balance and re-evaluate priorities every so often Stay focused on the goals of the test project Focus A seasoned test engineer can find his way towards test project goals, with clear signposts from his test manager
  • 70. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 70 Defining Tester Skills Reading Specifications, e-mails, test cases, etc. Writing Test cases, bug reports, test documentation, etc. Not “native language” dependent Statistics and other mathematics Pertinent technology, project, and testing skills Technology: Programming languages and more, like operating systems, networking, HTML/Web, etc. Application domain: banking, human factors, office applications, etc. Testing: scripting, exploring and attacking the system, automation, performance modeling, etc.
  • 71. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 71 Balancing the Skills Effective and efficient tester have right mix of skills for tasks and activities Application domain expert Understands intended behavior Skilled tester Knows quality risks and test techniques Technical guru Aware of technical issues and limitations What is the right mix for… …Internet appliance testing? …nuclear medicine testing? …your project? The appropriate depth and length of each arrow in the figure depends on the project, process, and product
  • 72. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 72 Terms to Remember Independent testing
  • 73. Chapter 1: Fundamentals of Testing Section 5: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 74. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 74 Exercise: Psychology in Action Reflect on the attitudes and behaviors of the most successful testers you know. To what extent do they display the psychological aspects discussed in this section? What other elements of their personalities and skillsets do you think lead to success for them? Discuss.
  • 75. Chapter 2: Testing throughout the Software Lifecycle Test Engineering Foundation Essential Knowledge for Test Professionals
  • 76. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 76 2. Testing Throughout the Software Lifecycle 1. Software development models 2. Test levels or phases 3. Test types or targets 4. Maintenance testing
  • 77. Chapter 2: Testing throughout the Software Lifecycle Section 1: Software development models Test Engineering Foundation Essential Knowledge for Test Professionals
  • 78. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 78 Software Development Models Key concepts The relationship between development and test activities Adapting software development models to the context of the project and product Reasons for different levels of testing Terms to remember
  • 79. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 79 The “V” Model in Waterfall Projects Usually schedule- and budget-risk-driven Do ever-deeper levels of design, then build, then test Intuitive and familiar model Beats chaos! It’s hard to plan that far in advance! When plans fail, test--at the end--suffers!
  • 80. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 80 Evolutionary and Incremental Models Schedule-risk-driven to hit market window or delivery date Feature set grown around core functionality Can ship (something) any time once the core functionality is ready Becoming a popular approach Ranges in formality from Extreme Programming to RAD and RUP  Still a temptation to ship a system with buggy features  In the “agile” world, the role of testing still evolving
  • 81. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 81 System Integration Many projects involve integrating components Risk-mitigation options Integrate, track, and manage vendor testing in distributed test effort Trust vendor component testing Fix vendor testing/quality Disregard and replace their testing (ouch!) Watch politics in last two options Plan on integration and system testing yourself Coupling: Strong interaction or consequence of failure between component and system Irreplaceability: Few similar components available Essential: Key features in system unavailable if component does not work properly Vendor quality problems: Increased likelihood of a bad component
  • 82. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 82 Verification & Validation Verification Look for bugs in phase deliverables “Are we building the system right?” Validation Looking for bugs in system, based on phase deliverables “Are we building the right system?”
  • 83. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 83 IEEE 12207 Standard 1. Scope Purpose, application, tailoring, compliance, limitations 1. Normative references 2. Definitions 3. Application Lifecycle processes, tailoring, and fitting to organization 1. Primary life cycle processes Acquisition, supply, development, operation, maintenance 1. Supporting processes Tech pubs, CM, QA, IV&V, audits, problem resolution 1. Organizational life cycle processes Management, infrastructure, improvement, training
  • 84. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 84 CMMI Process Maturity Capability Maturity Model Integration: a five-level model from the Software Engineering Institute Initial: unpredictable, poorly controlled, reactive Managed: process established at process level, often reactive Defined: process established across organization, usually proactive Quantitatively managed: process measured and controlled at organization Optimizing: focus on continuous improvement, usually driven by data Testing has been underemphasized in CMM, leading to test-specific model (e.g., Critical Testing Processes, Test Process Improvement, Testing Maturity Model)
  • 85. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 85 Regardless of the Models… General characteristics of good testing Testing activity for each development activity (e.g., unit test and implementation) Test levels have focused objectives, with coordination to avoid gaps, overlap Test analysis, design begins early, prevents bugs Testers involved in any reviews they are qualified to attend, bringing their unique perspective You can combine or reorganize test levels provided you keep these characteristics in mind
  • 86. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 86 Terms to Remember Test level Verification Validation V-Model Incremental development model Commercial Off The Shelf (COTS)
  • 87. Chapter 2: Testing throughout the Software Lifecycle Section 1: Exercise 1 Test Engineering Foundation Essential Knowledge for Test Professionals
  • 88. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 88 Exercise: Models and Reality Famed quality expert W.E. Deming said, “All models are wrong; some are useful.” Indicate which lifecycle model in this section applied most closely to your past project (or, if there was no organizing model, indicate “code-and-fix”). To what extent do you think the model was useful? To what extent, if any, was it harmful? Discuss.
  • 89. Chapter 2: Testing throughout the Software Lifecycle Section 1: Exercise 2 Test Engineering Foundation Essential Knowledge for Test Professionals
  • 90. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 90 Exercise: Omninet Lifecycle Considerations Read the Omninet Marketing Requirements Document. Is a V-model (sequential) or incremental model more appropriate for this project? Why? Are maintenance, integration, or verification and validation important for this project? Why? Discuss.
  • 91. Chapter 2: Testing throughout the Software Lifecycle Section 2: Test levels or phases Test Engineering Foundation Essential Knowledge for Test Professionals
  • 92. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 92 Test Phases or Levels Key concepts Major objectives of testing for each level Typical testing objects for each level Typical targets of testing for each level Testing work products for each level Test participants for each level Types of defects and failures for each level Terms to remember
  • 93. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 93 Component (Unit) Test Objective: Find bugs in the individual pieces of the system under test prior to system integration Basis: Code, database, reqs/design, quality risks Test types: Functionality, resource use, performance, structural Item Under Test (IUT): Varies. Smallest independently testable item (function or class) or a distinct component providing services to others Harnesses and tools: API level (drivers and stubs), freeware and commercial Responsible: Usually programmers, but the level of proficiency and degree of execution varies.
  • 94. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 94 Unit/Component Test Process Unit/component testing typically… …involves access to the code …is run in a development environment …requires drivers, stubs, and/or harnesses …is done by the programmer who wrote the code Often, bugs are fixed upon being found without any reporting, which reduces the transparency of the development process with respect to quality Test-first/test-driven development Develop a set of unit tests Build and integrate code Run the tests and debug until the tests pass
  • 95. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 95 Drivers and Stubs During unit, component, and integration testing--and for testing APIs--it’s often necessary to simulate parts of call flow reachable from module(s) under test Data setup sometimes necessary, also Driver: function(s) that call module(s) under test Stub: function(s) called--directly or indirectly--by module(s) under test
  • 96. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 96 Integration Test Objective: Find bugs in the relationships and interfaces between pairs and groups of components in the system under test as the pieces come together Basis: Design, architecture, schemas, dataflows, quality risks Test types: Functionality, resource use, performance Item Under Test: Builds or backbones Harnesses and tools: API and CLI level, freeware and commercial Responsible: Ideally both testers and programmers, but often no one
  • 97. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 97 Integration Techniques Big bang Take all tested modules; put them all together; test Quick, but where’s the bug? Why wait until all code is written to start integration? Bottom up Start with bottom layer modules; use appropriate drivers; test Repeat process, replacing drivers with modules, until done Good bug isolation, but what if nasty problems are at the top? Top down Like bottom up, but start from top and use stubs Good bug isolation, but what if nasty problems are at the bottom? Backbone Start with critical modules; build initial backbone; use drivers and stubs; test Repeat process, replacing stubs and drivers with modules in risk order Good bug isolation and finds integration bugs in risk order
  • 98. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 98 Backbone Integration Technique Backbone 0 (BB0) Consider the following example (based on a real project) We start with a basic backbone Communication APIs Basic networking architecture Test basic functionality, error handling and recovery, reliability, and performance Quality risk: Is the underlying system architecture untenable? Backbone 0: Testing the basic communication software and network architecture
  • 99. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 99 Backbone Integration Technique Backbone 1 (BB1) Add the modules that implement the most critical core operations and services Again, test basic functionality, error handling and recovery, reliability, and performance Quality risk: Do the core operations and functions integrate with the transport layer? Continue process with next level of quality risk…. Backbone 1: Testing some core operations and services through the communication API and network
  • 100. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 100 Backbone Integration Technique Backbone N (BBn) As the last step, we are testing (in this case, using automated test drivers) end-to-end through the GUIs on the host systems Quality risk: Does the fully integrated system work? The final backbone for integration testing is also the first fully integrated build for system test Backbone N: Testing the entire, completely integrated system end- to-end. When this works, we’re ready for System Test
  • 101. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 101 Integration Test Levels There may be more than one level of integration testing on a project Component integration testing: look for bugs in interactions between units or components following unit/component test System integration test: look for bugs in interaction between entire systems following system test System integration testing is complex Multiple organizations controlling the systems’ interfaces, making change dangerous Business processes may span systems Hardware/system compatibility issues can arise
  • 102. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 102 System Test Objective: Find bugs in the overall and particular behaviors, functions, and responses of the system under test as a whole Basis: Requirements, high-level design, use cases, quality risks, experience, checklists, environments Test types: Functionality, security, performance, reliability, usability, portability, etc. Item Under Test: Whole system, in as realistic-as- possible test environment Harnesses and tools: API, CLI, or GUI, freeware and commercial Responsible: Typically independent testers
  • 103. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 103 Acceptance Test Objective: Demonstrate that the product is ready for deployment/release Basis: Requirements, contracts, experience Test types: Functional, portability, performance Item Under Test: Whole system, sometimes in the production or customer environment Harnesses and tools: GUI usually Responsible: Often users or customers, but also independent testers
  • 104. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 104 Variations in Acceptance Testing User acceptance testing: Business users verify fitness for functional purposes. Operational testing: Acceptance by system administrators (e.g., backup-restore, disaster recovery, user management, maintenance, security) Contract and regulation testing: Verification of conformance to contractually-agreed or legally mandated requirements, regulations, or standards. Alpha, Beta, and field testing: Testing and confidence-building by potential or existing customers. Beta testing and field testing are performed in the actual environment(s).
  • 105. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 105 Pervasive Testing: Early, Cross-Functional Test execution activities for test phases Project timeline
  • 106. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 106 Why Pervasive Testing? Different participants can test different granularities Different skills for different granularities Different granularities emphasized in each phase Unit testing: primarily structural System testing: primarily behavioral Acceptance testing: primarily live It’s important to be flexible, though Test techniques of various granularities can be useful in all the test execution phases Phase overlap a function of entry and exit criteria Not all test phases occur on all projects
  • 107. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 107 When Testing Pervades Projects Test tasks occur throughout the development effort Test execution is planned with multiple cycles to allow for fix time Features will be dropped or slipped into later iterations rather than slipping test phase entry dates or entering before ready
  • 108. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 108 Terms to Remember Alpha testing Beta or field testing Business process-based testing Component testing Contract acceptance testing Daily build Drivers Functional requirements Integration Integration testing
  • 109. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 109 Terms to Remember Non-functional requirements Operational (acceptance) testing Regulation testing Requirements-based testing Robustness testing Stub System testing Test-driven development Test environment Testware User acceptance testing
  • 110. Chapter 2: Testing throughout the Software Lifecycle Section 2: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 111. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 111 Exercise: Omninet Test Levels Read the Omninet System Requirements Document. If you were managing all the testing for Omninet, which levels or phases of testing would you plan? Why? What would the major goals of each level or phase of testing be? What kind of acceptance test, if any, would you plan? Why? How do these test levels relate to and affect the lifecycle model you selected in the previous exercise? Discuss.
  • 112. Chapter 2: Testing throughout the Software Lifecycle Section 3: Test types or targets Test Engineering Foundation Essential Knowledge for Test Professionals
  • 113. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 113 Test Types or Targets Key concepts Major software test types or targets Functional and non-functional tests Structural tests Confirmation and regression tests Use of functional, non-functional, and structural tests at various levels Terms to remember
  • 114. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 114 Functionality Reasonable or required action not provided, inaccessible, or seriously impaired No add function on a calculator Add function implemented, “+” key doesn’t work Can only add integers, not real numbers Right action, wrong result Add function: 2+2=5? Right action, right result, wrong side-effect Divide function: 2/2=I (Roman numeral format) System, subsystem or component functionality is described in documents like requirements specification, use cases, or a functional specification (sometimes). However, often some functions remain undocumented, and testers must understand “reasonable behavior”.
  • 115. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 115 Security Security threats include Virii Cracking into servers Denial of service Common bad assumptions Encryption (HTTPS) on Web server solves security problems Buying a firewall solves problems Unskilled network or system administrators can solve problems Specialized field of test expertise The determined--or bored--cracker might break into any of your servers. One “exploit” can lead the cracker to other vulnerabilities--and to valuable data.
  • 116. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 116 Performance And Reliability Performance Too slow throughout performance curve Unacceptable “knee” in performance curve Unacceptable performance degradation over time Reliability System fails to complete normal functions System functions normally, but randomly crashes or hangs
  • 117. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 117 Stress, Capacity, and Volume Stress: extreme conditions cause failure Combinations of error, capacity, and volume tests Capacity: functionality, performance, or reliability problems due to resource depletion Fill hard drive or memory to 80+% Volume: functionality, performance, or reliability problems due to rate of data flows Run 80+% of rated transactions per minute, number of simultaneous users, etc.
  • 118. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 118 Maintenance And Maintainability Maintenance Update and patch install and deinstall processes don’t work Configurations can’t be changed appropriately (e.g., plug-and-play, hot plugging, adding disk space, etc.) Maintainability Software itself (source code) not maintainable Databases not upgradeable Databases grow monotonically Software not efficiently testable during maintenance; e.g., excessive regression
  • 119. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 119 Usability and User Interface A system can function properly but be unusable by the intended customer Cumbersome interfaces that do not follow workflows Inaccessible functionality Inappropriately difficult for the users to learn Instructional, help, and error messages that are misleading, confusing, or misspelled
  • 120. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 120 Configuration, Compatibility, Portability A single platform may be configured in many different ways in software A family of platforms may support various hardware configurations Are configuration changes handled? Add disk space or other storage Add memory Upgrade or add CPU Interoperability with programs, OS, database Portability to various environments
  • 121. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 121 Other Functional/Non-Functional Tests Localization (user interface) Localization (operational) Standards and regulatory compliance Error handling and recovery Disaster recovery Networked/internet- worked or distributed Timing and coordination Data quality Data conversion Operations Installation De-installation Date and time handling Documentation And many others…
  • 122. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 122 Structural Tests Tests based on how the system is built Code Data Design Structural (white box) coverage can be measured after functional and non-functional (black box) tests are run to check for omissions This topic will be covered in more depth later…
  • 123. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 123 Regression and Confirmation Regression testing checks the effects of changes, since even small, localized, isolated changes, don’t always have small, localized, or isolated effects Regression strategies are covered in the next section Confirmation testing confirms that… Changes made to the system are present Bug fixes introduced in the system solve the observed symptoms Repeatability of tests helps with regression and confirmation testing Automation, covered in depth later, is also helpful
  • 124. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 124 Comparing Test Types Static Structural Func./Non-func. What Test without running Test how system works Test what system does Why Identify bugs before they’re built Identify bugs in functions or interfaces Identify bugs in system results and behavior Who All stakeholders Mostly developers (and knowledgeable testers) Mostly testers (and knowledgeable developers) When During specification and development Mostly during unit, component, integration Mostly integration, system, and acceptance How Analyze reqs, design, data, code, tests, etc. Derive tests from code, data, design Derive tests from reqs, design, quality risks Tools, data, and cases can be shared: encourage cross-pollination of techniques We’ll discuss techniques for each type of testing in upcoming sections
  • 125. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 125 ISO 9126 Quality Characteristics/Subcharacteristics Functionality: suitability, accuracy, interoperability, security, compliance Reliability: maturity (robustness), fault-tolerance, recoverability, compliance Usability: understandability, learnability, operability, attractiveness, compliance Efficiency: time behavior, resource utilization, compliance Maintainability: analyzability, changeability, stability, testability, compliance Portability: adaptability, installability, co-existence, replaceability, compliance
  • 126. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 126 Terms to Remember Automation Behavior-based testing Black box or specification-based testing Code-based, white box, or structural testing Code coverage Confirmation testing Functional test Interoperability testing Load testing Maintainability testing Performance testing
  • 127. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 127 Terms to Remember Portability testing Regression test Reliability testing Security testing Stress testing Test suite Usability testing Use case testing
  • 128. Chapter 2: Testing throughout the Software Lifecycle Section 3: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 129. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 129 Exercise: Omninet Test Types Referring back to the levels you selected in the previous exercise, list for each level the test targets or types you would include in each level. How does the lifecycle model you selected in a previous exercise affect the amount of regression testing? How does the lifecycle model you selected in a previous exercise affect the amount of confirmation testing? Discuss.
  • 130. Chapter 2: Testing throughout the Software Lifecycle Section 4: Maintenance testing Test Engineering Foundation Essential Knowledge for Test Professionals
  • 131. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 131 Maintenance Testing Key concepts Reasons for maintenance testing Maintenance testing versus new application testing Role of regression testing and impact analysis Terms to remember
  • 132. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 132 Reasons for Maintenance Three typical triggers for maintenance and maintenance testing Modification: enhancements, bug fixes, operational environment changes, patches Migration: a new supported environment Retirement: end-of-life of a subsystem or entire system triggers replacement Maintenance testing addresses the change itself--and what wasn’t changed and shouldn’t change
  • 133. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 133 Testing Maintenance Releases Regression is the big risk for maintenance releases Some organizations try to put a major release worth of features into a short maintenance release Time to develop new tests is scarce Large project test estimation rules-of-thumb fail
  • 134. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 134 Regression Regression (misbehavior of a previously correct function, attribute, or feature due to a change) types: Local (fix creates new bug) Exposed (fix reveals existing bug) Remote (fix in one area breaks something in another area) Regression can affect new and existing features
  • 135. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 135 Regression Strategy 1: Repeat All Tests If our tests are aligned with quality, then repeating all tests should find most important regressions Automation is the only practical means for large complex system We’ll cover automation more thoroughly in a later section
  • 136. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 136 Regression Strategy 2: Repeat Some Tests Often, full automation is impossible Select some tests to repeat Traceability Change analysis Risk analysis Use cross-functional tests for “accidental regression testing” Can use code coverage to assess level of risk
  • 137. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 137 Three Other Regression Strategies Release more slowly Release every six months rather than every month Partial or even full repetition can increase coverage on bigger releases Combine emergency patches with slower release process to allow for flexibility while keeping regression risk low Use customer or user testing Beta for mass-market software Pilot, staged or phase, parallel release for IT
  • 138. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 138 Terms to Remember Maintenance testing Impact analysis Modifications Migration Retirement
  • 139. Chapter 2: Testing throughout the Software Lifecycle Section 4: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 140. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 140 Exercise: Omninet Maintenance Assume you can fully automate the testing of the Omninet kiosk and call center. If a post-release change is made to the call-center user interface that does not affect functionality, what would you retest? Assume you have not automated the testing of the Omninet kiosk and call center. If the same change is made, what would you retest? What if the change did affect functionality? Discuss.
  • 141. Chapter 3: Static Techniques Test Engineering Foundation Essential Knowledge for Test Professionals
  • 142. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 142 3. Static Techniques 1. Reviews and the test process 2. Review process 3. Static analysis by tools
  • 143. Chapter 3: Static Techniques Section 1: Reviews and the test process Test Engineering Foundation Essential Knowledge for Test Professionals
  • 144. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 144 Reviews and the Test Process Key concepts Software work products and static techniques The importance and value of static techniques The difference between static and dynamic techniques Terms to remember
  • 145. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 145 Static Testing Reviews and tools Reviews range from informal to very formal Tools can perform some types of static tests Static techniques can be used for requirements and designs, plus code, database schemas, documentation, tests… Models and prototypes A diagram of a complex system can often reveal design problems that can hide in words An ugly diagram means lots of bugs Test cases and data Test analysis and design based on requirements and design specs is a form of structured review Test analysis and design often reveals problems
  • 146. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 146 Static Tools Static analysis Problematic wording: Spell/grammar checkers Dangerous programming: J-Test, Safer C, lint… Measurement: Complexity analysis System simulations General Purpose System Simulator Performance modeling/operations research tools Spreadsheets
  • 147. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 147 Reviews: Costs and Benefits Costs Time required to perform reviews Effort required to gather and analyze metrics Process improvement Benefits Shorter schedules (due to efficient bug removal) Shorter testing periods and lower testing costs Developer productivity Improved quality of product (which reduces downstream costs) Bottom line: Reviews of all kinds are proven, high- return techniques for improving quality
  • 148. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 148 Relating Static and Dynamic Testing Similarities Seek to identify defects Work best when a broad cross-section of stakeholders are involved Save the company money and time Differences Each technique can find different types of defects more effectively and efficiently Static techniques find defects rather than failures
  • 149. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 149 Terms to Remember Reviews Dynamic testing Static analysis
  • 150. Chapter 3: Static Techniques Section 1: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 151. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 151 Exercise: Omninet Static Testing Do you see reviews and static analysis as useful for the Omninet project? If so, what kinds of problems do you think these reviews and static analyses would locate? What kinds of problems might they not locate? Discuss.
  • 152. Chapter 3: Static Techniques Section 2: Review process Test Engineering Foundation Essential Knowledge for Test Professionals
  • 153. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 153 Review Process Key concepts Phases, roles and responsibilities of a typical formal review The differences between different types of review The factors for successful reviews Terms to remember
  • 154. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 154 Types of Reviews Informal: no real process (hallway chats, buddy tests, pair programming), yet useful, cheap, popular Peer: documented and defined defect removal process, involving peers and technical experts but not managers Walkthroughs: author “walks” peers “through” the document or code Inspections: a trained moderator (other than the author) leads the inspection team (with defined roles) through a formal inspection process (rules, checklists, entry and exit criteria), which includes gathering defect removal metrics
  • 155. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 155 Incompleteness and ambiguity can hide the real meaning of the specifications Agreement and uniform understanding of the specifications Consensus and Understanding Long before any code exists, the specification must be handed to an outside testing group to be scrutinized for completeness and clarity. As [V.A.] Vyssotsky [of Bell Lab’s Safeguard Project] says, the developers themselves cannot do this: “They won’t tell you they don’t understand it; they will happily invent their way through the gaps and obscurities.” – Fred Brooks The Mythical Man-Month 1975
  • 156. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 156 A Generic Review Process 1. Planning 2. Kick-off 3. Preparation 4. Review meeting 5. Rework/repair 6. Follow-up The details of the review process depend on the specific review type used on the project Includes estimating and planning, training participants, etc. Follow-up includes on individual items as well as overall process improvement analysis, evaluation of defect (bug) removal at phase exit reviews (exit meetings), etc. These steps of the process repeat per each item reviewed. Preparation is usually one to two hours alone. Meeting is one to two hours together. Rework/repair is fixing the bugs found.
  • 157. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 157 Roles and Responsibilities Moderator: Lead the review meetings Scribe or secretary: Gather information on findings Author: Describe, explain, answer questions on item Reviewer/inspector: Find defects (bugs) in item Manager: Plan, arrange resources and training, support, analyze process metrics In some cases, one person may play multiple roles Authors sometimes act as moderators One of the reviewers can act as the secretary The specifics are determined by the type of review
  • 158. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 158 Suggestions for Successful Reviews Provide training Review the product, not the producer Set and follow agenda and objectives Limit debate Focusing on finding, not fixing, problems Take written notes Limit and carefully select participants Insist on preparation (e.g., by having people submit notes) Develop a checklist for each type of item that is reviewed Review the reviews Use the right techniques Ensure management support Learn and get better!
  • 159. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 159 Common Requirements and Design Bugs Ambiguities: What exactly does that mean? E.g.: System shall allow user to read ISP e-mail What ISPs? What size e-mails? Attachments? Incompleteness: Okay, and then what? E.g.: Upon three invalid passwords, system shall lock user’s account… For how long? How to unlock? Who can unlock? Untestability: How can I check this item? E.g.: System shall provide 100% availability No known test technique to demonstrate perfect availability Excessive dependencies, coupling and complexity Look for ugly design diagrams and confusing requirements
  • 160. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 160 IEEE 1028 Standard for Software Reviews 1. Overview Purpose, scope, conformance, organization, application 1. References 2. Definitions 3. Management reviews Responsibilities, inputs/outputs, entry/exit criteria, procedures 1. Technical reviews Responsibilities, inputs/outputs, entry/exit criteria, procedures
  • 161. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 161 IEEE 1028 Standard for Software Reviews 6. Inspections Responsibilities, inputs/outputs, entry/exit criteria, procedures, data collection, process improvement 6. Walkthroughs Responsibilities, inputs/outputs, entry/exit criteria, procedures, data collection, process improvement 6. Audits Responsibilities, inputs/outputs, entry/exit criteria, procedures
  • 162. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 162 Terms to Remember Review process Review meeting Moderator/ inspection leader Reviewer Scribe (or secretary) Inspection Entry criteria Exit criteria Formal review
  • 163. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 163 Terms to Remember Walkthrough Informal review Peer or technical review Metrics Kick-off
  • 164. Chapter 3: Static Techniques Section 2: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 165. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 165 Exercise: Omninet Requirements Review Break into teams of three to five people. Select a review technique from among those discussed. If your selected technique involves specific roles, assign the roles. Review the Omninet Marketing Requirements Document. Discuss your team’s findings.
  • 166. Chapter 3: Static Techniques Section 3: Static analysis by tools Test Engineering Foundation Essential Knowledge for Test Professionals
  • 167. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 167 Static Analysis by Tools Key concepts The objective of static analysis versus dynamic testing Typical defects and errors identified by static analysis Typical benefits of static analysis List typical code and design defects identified by static analysis tools Terms to remember
  • 168. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 168 Static Analysis and Dynamic Testing Like dynamic testing, static analysis looks for defects in software source code and software models Unlike dynamic testing, static analysis is performed without actually executing the system Static analysis involves analysis of the system or its components by a tool, while dynamic testing does not always involve tools Static analysis can find defects that are hard to find or isolate in dynamic testing Examples include maintainability issues, unsafe pointer use Isolation is easier because you find the bug, not the symptom
  • 169. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 169 What Can We Analyze? Program code (e.g. control flow and data flow) Models of the program (e.g., simulations) Generated output such as HTML and XML Requirements and design documents
  • 170. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 170 Benefits of Static Analysis Early and cheaper detection of bugs (before test execution starts) Warnings about where bug clusters might exist, due to dangerous programming, high complexity, etc. Location of bugs dynamic testing might miss Detection of dependencies and inconsistencies in software models (e.g., such as link problems in Web pages) Improved maintainability of code and design Prevention of defects based on metrics gathered and lessons learned from analysis
  • 171. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 171 Typical Static Analysis Bugs Referencing a variable with an undefined value Inconsistent interface between modules and components Variables that are never used Unreachable (dead) code Programming standards violations Security vulnerabilities Syntax violations of code and software models
  • 172. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 172 Using Static Analysis Tools Typical users are… Programmers, often during component and integration testing Designers and system architects during design During initial introduction against an existing system, static analysis tools may produce a large number of warning messages Compilers do some static analysis, but many sophisticated tools are available
  • 173. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 173 Terms to Remember Static analysis Compiler Complexity Control flow Data flow
  • 174. Chapter 3: Static Techniques Section 3: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 175. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 175 Exercise: Omninet Static Analysis What kind of static analysis would you suggest for Omninet? Would you use static analysis in areas that would be subject to later testing? Discuss.
  • 176. Chapter 4: Test Design Techniques Test Engineering Foundation Essential Knowledge for Test Professionals
  • 177. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 177 4. Test Design Techniques 1. Identifying and designing test cases 2. Categories of test design techniques 3. Specification-based or black-box techniques 4. Structure-based or white-box techniques 5. Experience-based techniques 6. Choosing test techniques
  • 178. Chapter 4: Test Design Techniques Section 1: Identifying and designing test cases Test Engineering Foundation Essential Knowledge for Test Professionals
  • 179. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 179 Identifying and Designing Test Cases Key concepts • Base testing on risk analysis • Determine level of risk with likelihood and impact Specify test designs, cases, procedures Write test cases Relate test cases and test procedures Develop test execution schedule Terms to remember
  • 180. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 180 Phases of Test Development Test development often proceeds in phases (Quality risk) analysis High-level test design Low-level test design (implementation) External inputs used to create internal deliverables (testware)
  • 181. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 181 Analysis
  • 182. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 182 High-Level Design These arrows show the relationships between the risk categories and the test suites. We’ll capture this relationship information in the form of what’s called traceability.
  • 183. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 183 Low-Level Design (Implementation) Each risk to be mitigated via testing will have one or more test cases associated with it. (I’ve not shown all traces, to avoid clutter.)
  • 184. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 184 What’s a Quality or Product Risk? Risk The possibility of a negative or undesirable outcome The likelihood of a risk becoming an outcome is… >0, <1 in the future… 0 or 1 in the past Quality or product risk The possibility that the system will fail to satisfy customers, users, or other stakeholders A family of possible bugs are behind quality risks
  • 185. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 185 How Can We Analyze Quality Risks? Informal ISO 9126 Failure Mode and Effect Analysis Start with the classic quality risk categories Start with six main quality characteristics Start with categories, char- acteristics, or subsystems Functionality, states and transactions, capacity and volume, data quality, error handling and recovery, performance, standards and localization, usability, etc. Functionality, Reliability, Usability, Efficiency, Maintainability, Portability (FRUEMP), then decompose into key subcharacteristics for your system Key stakeholders list possible failure modes, predict their effects on system, user, society, etc., assign severity, priority, and likelihood, then calculate risk priority number (RPN) Set priority for testing each quality risk with key stakeholders Set priority for testing each subcharacteristic with key stakeholders Stakeholders use RPN to guide appropriate depth and breadth for testing Regardless of technique the keys are cross-functional stakeholder participation, consensus, and a best-possible-outcome outlook
  • 186. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 186 Quality Risk Tech. Risk Bus. Risk Risk Pri. # Extent of Testing Tracing Risk Category 1 Risk 1 Risk 2 Risk n Quality risks are potential system problems which could reduce user satisfaction A hierarchy of risk categories can help organize the list and jog your memory. 1 = Very high 2 = High 3 = Medium 4 = Low 5 = Very low Technical risk: Likelihood of the problem Business (operational) risk: Impact of the problem 1-5 = Extensive 6-10 = Broad 11-15 = Cursory 16-20 = Opportunity 21-25 = Report bugs Risk priority number: Aggregate measure of problem risk The product of technical and business risk, from 1-25. Tracing information back to requirements, design, or other risk bases
  • 187. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 187 Tips for Risk Analysis Use a cross-functional brainstorming team Identify the risk items, then assign the level of risk Only separate risk items when necessary to distinguish between different levels of risk Consider technical and business risk Technical risk: likelihood of problem, impact of problem on system Business risk: likelihood of usage, impact of problem on users Follow up and re-align risk analysis, testing, and the project at key project milestones
  • 188. Chapter 4: Test Design Techniques Section 1: Exercise 1 Test Engineering Foundation Essential Knowledge for Test Professionals
  • 189. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 189 Exercise: Omninet Risk Analysis Based on your reading of the Omninet Marketing Requirements Document, the Omninet System Requirements Document, and your experience with testing and bugs, perform a risk analysis for Omninet Discuss.
  • 190. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 190 IEEE 829 Test Design Specification The test design specification describes a collection of test cases at a high level, and includes the following sections Test design specification identifier Features to be tested (in this test suite) Approach refinements (specific techniques, tools, etc.) Test identification (tracing to test cases in suite) Feature pass/fail criteria (e.g., test oracle, test basis, legacy systems, etc.) This collection of test cases often called a test suite Sequencing (test suites and cases within suites) often driven by risk and business priority and affected by project constraints, resources, and progress
  • 191. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 191 IEEE 829 Test Case Specification A test case specification describes the details of a test case, and includes the following sections Test case specification identifier Test items (what is to be delivered and tested) Input specifications (user inputs, files, etc.) Output specifications (expected results, including screens, files, timing, etc.) Environmental needs (hardware, software, people, props…) Special procedural requirements (operator intervention, permissions, etc.) Intercase dependencies (if needed to set up preconditions) In practice, test cases vary significantly in effort, duration, and number of test conditions covered
  • 192. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 192 IEEE 829 Test Procedure Specification A test procedure specification describes how to run one or more test cases, and includes the following sections Test procedure specification identifier Purpose (e.g., which tests are run) Special requirements (skills, permissions, environment, etc.) Procedure steps (logging, set up, start, proceed [steps themselves], measurement of results, shutdown/suspension, restart [if needed], stop, wrap up/tear down, contingencies) Test procedures are often embedded in test cases A test procedure is sometimes referred to as a test script, and may be manual or automated
  • 193. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 193 Test Case Coverage (Traceability) Measure or correlate tests against areas of concern Used as a way to measure and enhance the tests Practical types Requirements specifications Design specifications Functional areas Quality risks Configurations This is a commonly-used technique to ensure thorough test coverage One technique Use spreadsheet List test cases and coverage area to measure 0: none; 1: indirect; 2: direct Test Case 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 2.1 2.2 2.3 2.4 2.5 Total Spec 1.1 1 1 1 1 2 6 1.2 2 2 1 1 2 8 1.3 0 1.4 2 2 4 1.5 1 1 1 1 4 1.6 2 2 1.7 2 1 2 2 7 1.8 2 1 3 1.9 1 1 2 4 1.10 1 1 1 1 4 Total 5 4 2 5 4 3 3 4 2 2 7 1 0
  • 194. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 194 Terms to Remember Priority Risk Risk identification Risk analysis Risk control Risk-based testing Product risk Test case Test condition Test data
  • 195. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 195 Terms to Remember Test case specification Test design specification Test procedure specification Test script Traceability
  • 196. Chapter 4: Test Design Techniques Section 1: Exercise 2 Test Engineering Foundation Essential Knowledge for Test Professionals
  • 197. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 197 Exercise: Omninet Test Design Specification Based on your quality risk analysis for Omninet, outline a set of test suites to address the important areas of risk. For each test suite, list briefly: What the test would cover How you would recognize passing or failing tests Establish the relationship between your test suites and the risks they will cover. Based on your risk analysis, how would you sequence the test suites? What other considerations would affect the sequencing of test suites? Discuss.
  • 198. Chapter 4: Test Design Techniques Section 2: Categories of test design techniques Test Engineering Foundation Essential Knowledge for Test Professionals
  • 199. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 199 Categories of Test Design Techniques Key concepts Reasons for specification-based (black box), structure-based (white box), and experience-based tests Common black box and white box techniques Characteristics and differences between specification-based testing, structure-based testing, and experience-based testing Terms to remember
  • 200. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 200 Three Types of Test Design Techniques Specification-based (black box), functional and non- functional Create tests primarily by analysis of the test basis Look for bugs in the way the system behaves Structure-based (white box) Create tests primarily by analysis of the structure of the component or system Look for bugs in the way the system is built Experience-based (attacks, checklists, exploratory) Create tests primarily based on understanding of the system, past experience, and educated guesses about bugs Look for bugs in the places other systems have bugs
  • 201. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 201 Specification-Based or Black Box Common elements include: Formal or informal models used to specify the problem to be solved, the software or its components Test cases derived systematically from these models Examples include: Equivalence partitioning and boundary value analysis State transition diagrams Decision tables
  • 202. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 202 Structure-Based or White Box Common elements include: System structure (e.g., code, design, etc.) used to derive the test cases, for example code and design The extent of structural coverage can be measured for existing other test cases Further test cases can be derived systematically to increase coverage Examples include: Statement coverage Branch coverage
  • 203. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 203 Experience-Based Common elements include: The knowledge and experience used to derive test cases Can consider knowledge and experience of the software, its usage and its environment or… …knowledge about historical and likely defects and their distribution Examples include: Attacks Checklists Exploratory
  • 204. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 204 Terms to Remember Black box or specification-based techniques Experience-based techniques White box or structure-based techniques
  • 205. Chapter 4: Test Design Techniques Section 2: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 206. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 206 Exercise: Omninet Test Techniques Refer to your outline of test suites for Omninet. For each test suite, identify whether one, two, or all three of the following categories of test techniques would be useful in designing test cases: Specification-based (black box) Structure-based (white box) Experience-based Discuss.
  • 207. Chapter 4: Test Design Techniques Section 3: Specification-based or black box techniques Test Engineering Foundation Essential Knowledge for Test Professionals
  • 208. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 208 Specification-based Techniques Key concepts Writing test cases from given software models using equivalence partitioning, boundary value analysis, decision tables, and state transition diagrams The main purpose of each technique and how coverage may be measured Use case testing Terms to remember
  • 209. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 209 Equivalence Partitioning Divide the inputs, outputs, behaviors, and environments into classes you think will be handled equivalently Define at least one test case in each partition, or use boundary values in partitions that are ranges Can use marketing info to favor classes Ex: Testing supported printers Physical interface: Parallel, serial, USB 1.1, USB 2.0, infrared, Firewire, SCSI, others? Logical interface: Postscript, HPPL, ASCII, others… Image application: Laser jet, ink jets, bubble jets, dot matrix, line printers, letter quality, pen plotters, others…
  • 210. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 210 Boundary Value Analysis A refinement of equivalence partitioning that selects the edges or end-points of each partition for testing Equivalence partitioning looks for bugs in the code that handles each equivalent class Boundary values also look for bugs in the definitions of the edges Can only be used when the elements of the equivalence partition are ordered Non-functional boundaries (capacity, volume, etc.) can be used for non-functional testing, too
  • 211. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 211 Integer “How many items would you like to order?”
  • 212. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 212 Real-Number “What is the average temperature in Duluth?”
  • 213. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 213 Character and String “Password (6-10 character alphanumeric)”
  • 214. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 214 Date “Enter the departure date for your flight”
  • 215. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 215 Time “Enter the departure time for your flight”
  • 216. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 216 Currency “Enter a bid price (under $1000):”
  • 217. Chapter 4: Test Design Techniques Section 3: Exercise 1 Test Engineering Foundation Essential Knowledge for Test Professionals
  • 218. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 218 Exercise: Classes and Boundaries You are testing an e-commerce site that sells Omninet knick- knacks like baseball caps, jackets, etc. Create functional tests for accepting orders The system accepts a five-digit numeric item ID number from 00000 to 99999 Item IDs are sorted by price, with the cheapest items having the lower (close to 00000) item ID numbers and the most expensive items having the higher (close to 99999) item ID numbers The system accepts a quantity to be ordered, from 1 to 99 If the user enters a previously-ordered item ID and a 0 quantity to be ordered, that item is removed from the shopping cart The maximum total order is $999.99 Use boundary value analysis and equivalence class partitioning to create tests in the following template Discuss.
  • 219. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 219 Screen Prototype
  • 220. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 220 Test Template Test Number Inputs, Actions Item ID Quantity ? ? Expected Results Item Price Cart Contents Cart Total ? ? ?
  • 221. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 221 Use Cases/Scenario Tests Design various cases that reflect challenging real- world uses of the product For example, a home equity loan application “John and Jenny Stevens have three kids…” “…a house worth $400K on which they owe $350K…” “… two cars worth $25K on which they owe $17K…” “…incomes of $45K and 75K respectively…” “… one late payment on their Visa card and one late car payment, seventeen months and thirty-five months ago, respectively…” “…and they apply for a 15K home equity loan.” Some object-oriented design methodologies include use cases, so this can be an easy source of tests
  • 222. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 222 Use Cases and Boundaries Revisiting our boundary conditions, do reasonable usage conditions affect boundary conditions? If we have a four-byte unsigned item counter variable, do we want to allow ordering 32K items just because the software supports it? Do we allow departure dates after arrival dates for trips? Is accepting a “ridiculous” input a bug? What do you think?
  • 223. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 223 Decision Tables Business rules can often be specified compactly in decision tables Deciding how to process an order based on size, stock on hand, state to which to ship, and so forth is often in business rules These can be shown as flow-charts or as tables The decision tables can make for instant test cases Let’s take a look at an example
  • 224. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 224 ATM Decision Table Condition 1 2 3 4 5 Valid Card N Y Y Y Y Valid PIN - N N Y Y Invalid PIN=3 - N Y N N Balance OK - - - N Y Action Reject Card Y N N N N Reenter PIN N Y N N N Keep Card N N Y N N Reenter Request N N N Y N Dispense Cash N N N N Y This decision table shows the business logic for an ATM. Notice that the dashes “-” indicate conditions that aren’t reached as part of this rule. The rules are mutually exclusive, in that only one rule can apply at any one moment of time. Notice that the business logic layer is usually under the user interface layer, so at this point the basic sanity checking of the inputs should have been done.
  • 225. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 225 A More Complex Decision Table Some police departments have handheld computers that take credit cards for moving violations A decision table could be used to design and test this system The conditions that determine the actions to take are shown in the decision table on the next page You might want to use boundary value analysis to adequately cover the rules What would you do about interaction of rules?
  • 226. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 226 Police System Decision Table Condition 1 2 3 4 5 6 7 8 License OK No - - - - - - - Warrant - Yes - - - - - - Registration OK - - No - - - - - Vehicle OK - - - No - - - - Excess Speed - - - - 1-10 11-20 21-25 >25 Action Arrest Yes Yes - - - - - Yes Fix-It Ticket - - Yes Yes - - - - Warning - - - - Yes - - - Fine +250 +250 +25 +25 +0 +75 +150 +250
  • 227. Chapter 4: Test Design Techniques Section 3: Exercise 2 Test Engineering Foundation Essential Knowledge for Test Professionals
  • 228. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 228 Exercise: Decision Tables Refer to the payment decision table in the Omninet System Requirements Document. Develop test scenarios which adequately cover the decision table shown. How did you determine the level of coverage required? What implementation assumptions might change the number of tests needed? Discuss.
  • 229. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 229 Finite-State Models Understand the various states the system has, including any which are initial and final Identify transitions, events, conditions, and actions in each state Use a graph or table to model the system and serve as an oracle For each event and condition, verify action and next state
  • 230. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 230 State-Transition Diagram A state-transition diagram for a point-of-sales system, from cashier’s point of view. How would it look from the customer’s point of view? From the system’s? Do you see the bug?
  • 231. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 231 State-Transition Tables Current State Event[Condition] Action New State Logging In Password[invalid] Error Logging In Logging In Password[valid] Open register Waiting Logging In Customer [Undefined] [Undefined] Logging In Scan[any] [Undefined] [Undefined] Logging In Pay[any] [Undefined] [Undefined] Logging In Shift end [Undefined] [Undefined] A state-transition table can represent complex state-transitions that won’t fit on a graph. (However, complexity might indicate bad design.) It can also reveal undefined situation, as does this portion of the table for the graph on the preceding page.
  • 232. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 232 Finite-State Models Let’s look at another example Print server (graph next page) can be: Awaiting job Queuing job Printing job Awaiting user intervention Awaiting operator intervention Printer server responds to events (user or printer inputs) based on state
  • 233. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 233 Print Server State Machine Graph
  • 234. Chapter 4: Test Design Techniques Section 3: Exercise 3 Test Engineering Foundation Essential Knowledge for Test Professionals
  • 235. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 235 Exercise: Kiosk States The Omninet public Internet access kiosks can be in various states, based on receipt of payment, active sessions, and so forth. Refer to the Marketing Requirements Document and the System Requirements Document for Omninet Draw a state diagram for the kiosk. Cover the state diagram with test cases. Discuss.
  • 236. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 236 Terms to Remember Black box (or specification-based) testing Boundary value analysis Decision table testing Equivalence partitioning State transition testing Use case testing
  • 237. Chapter 4: Test Design Techniques Section 4: Structure-based or white box techniques Test Engineering Foundation Essential Knowledge for Test Professionals
  • 238. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 238 Structure-based Techniques Key concepts Code coverage Statement and decision coverage Control-flow test design techniques Terms to remember
  • 239. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 239 Structure-based (White Box) Fundamentals Structure-based tests are based on how the system works inside Determine and achieve a level of coverage of control flows based on code analysis Determine and achieve a level of coverage of data flows based on code and data analysis Determine and achieve a level of coverage of interfaces, classes, call flows, and the like based on analysis of APIs, system design, etc. Coverage of structure is a way to check for gaps in specification- and experience-based tests
  • 240. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 240 Code Coverage Levels of code coverage Statement coverage: every statement executed Branch (decision) coverage: every branch (decision) taken each way, true and false Condition coverage: every combination of true and false conditions evaluated (i.e., the whole truth table) Multicondition decision coverage: only those combinations of conditions that can influence the decision Loop coverage: All loop paths taken zero, once, and multiple (ideally, maximum) times Does statement coverage imply branch (decision) coverage?
  • 241. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 241 Code Coverage Example What test values for n do we need to cover all the statements? n < 0, n > 0 Does that get us branch coverage? No, we also need n=0 Does that get us condition coverage? Yes: no compound conditions How about path coverage? Need to cover n = 1 and n = max 1 #include <stdio.h> 2 main() 3 { 4 int i, n, f; 5 printf(“n = "); 6 scanf("%d", &n); 7 if (n < 0) { 8 printf("Invalid: %dn", n); 9 n = -1; 10 } else { 11 f = 1; 12 for (i = 1; i <= n; i++) { 13 f *= i; 14 } 15 printf("%d! = %dn", n, f); 16 } 17 return n; 18 }
  • 242. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 242 Code Coverage as a Test Design Tool By themselves, black box techniques can leave as much as 75%or more of the statements uncovered Is this a problem? Depends on what is uncovered! Code coverage tools can instrument a program to monitor code coverage during testing Gaps in code coverage can lead to more test cases to achieve higher coverage levels
  • 243. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 243 McCabe Cyclomatic Complexity McCabe’s Cyclomatic Complexity measures control flow complexity Measured by drawing a directed graph Nodes represent entries, exits, decisions Edges represent non-branching statements It has some useful testing implications High-complexity modules are inherently buggy and regression-prone The number of basis paths through the graph is equal to the number of basis tests to cover the graph Let’s see how…
  • 244. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 244 Cyclomatic Complexity for Factorial
  • 245. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 245 Basis Paths and Tests
  • 246. Chapter 4: Test Design Techniques Section 4: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 247. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 247 Exercise: Hexadecimal Converter On the next page you’ll find a simple C program that accepts a string with hexadecimal characters (among other unwanted characters). It ignores the other characters and converts the hexadecimal characters to a numeric representation. If you test with input strings “059”, “ace”, and “ACD” what level of coverage will you achieve? What input strings could you add to achieve statement and branch coverage? Would those be sufficient for testing this program? Discuss.
  • 248. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 248 main() { intc; unsignedlonginthexnum,nhex; hexnum=nhex=0; while((c=getchar())!=EOF){ switch(c){ case'0':case'1':case'2':case'3': case'4':case'5':case'6':case'7': case'8':case'9': /*Convertadecimaldigit*/ nhex++; hexnum*=0x10; hexnum+=(c-'0'); break; case'a':case'b':case'c': case'd':case'e':case'f': /*Convertalowercasehexdigit*/ nhex++; hexnum*=0x10; hexnum+=(c-'a'+0xa); break; case'A':case'B':case'C': case'D':case'E':case'F': /*Convertanuppercasehexdigit*/ nhex++; hexnum*=0x10; hexnum+=(c-'A'+0xA); break; default: /*Skipanynon-hexcharacters*/ break; } } printf("Got%dhexdigits:%xn",nhex,hexnum); return0; } Exercise: Hex Converter Program
  • 249. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 249 Terms to Remember Code coverage Decision coverage Statement coverage Structure-based testing Structural or white box testing
  • 250. Chapter 4: Test Design Techniques Section 5: Experience-based techniques Test Engineering Foundation Essential Knowledge for Test Professionals
  • 251. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 251 Experience-based Techniques Key concepts Reasons for writing test cases based on intuition, experience and knowledge Comparing experience-based techniques with specification-based testing techniques Terms to remember
  • 252. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 252 Experience-based Tests Experienced-based tests are based on the tester’s… …skill and intuition …experience with similar applications …experience with similar technologies Rather than being pre-designed, experience-based tests are often created during test execution (i.e., test strategy is dynamic) Tests are frequently “time-boxed” (i.e., brief periods of testing focused on specific test conditions) Examples include error guessing, bug hunting, “breaking software” based on checklists or bug taxonomies, and exploratory testing
  • 253. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 253 Common Experience-Based Approaches Most professional approaches to experience-based testing do not create tests entirely during execution May be guided by: Checklist Bug taxonomy List of attacks Bug hunt approach Set of test charters These guidelines are prepared in advance to some level of detail Purely on-the-fly testing (ad hoc testing) is common, but usually (ineffective) manual random testing
  • 254. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 254 Dynamic Test Strategies Advantages Effective at finding bugs Resists pesticide paradox due to high variance Efficient (lightweight record-keeping) Good check on prepared tests Fun and creative Disadvantages Gappy coverage, especially under pressure Difficult to estimate No bug prevention Extensive debriefing and discussion doesn’t scale to large teams Not all testers have the necessary level of skill, experience
  • 255. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 255 Exploratory Testing Case Study Staff 7 Technicians 3 Engineers + 1 Mgr. Experience <10 years total > 20 years total Test Type Precise scripts Chartered exploratory Test Hrs/Day 42 6 Bugs Found 928 (78%) 261 (22%) Effectiveness 22 44 This case study shows exploratory testing as about twice as effective at finding bugs on an hour-per-hour basis. It is significantly more efficient, because the scripted tests required extensive effort to create. However, to what extent is the relative level of experience important? What is the value of testing beyond finding bugs? What did the exploratory tests cover? What is the re-use value of the scripts?
  • 256. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 256 Terms to Remember Error guessing Experienced-based testing Exploratory testing
  • 257. Chapter 4: Test Design Techniques Section 5: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 258. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 258 Exercise: Dynamic vs. Pre-designed Tests Major factors or concerns Dynamic Pre-designed Do thorough regression testing Maximize bug finding efficiency Use testers with limited experience Automate half of the test cases Deliver precise, accurate estimates Test without test preparation time Test a rapidly changing UI Use a distributed test effort Put a “+” in the column where the factor or concern motivates towards using the approach and a “-” in the column where the factor or concern motivates away from using the approach.
  • 259. Chapter 4: Test Design Techniques Section 6: Choosing test techniques Test Engineering Foundation Essential Knowledge for Test Professionals
  • 260. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 260 Choosing Test Techniques Key concept The factors that influence the selection of appropriate test design techniques Terms to remember
  • 261. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 261 Factors in Choosing Techniques Type of system Regulatory standards Customer or contractual requirements Level and type of risk Test objectives Documentation available Knowledge of testers Time and budget Development life cycle Previous experiences on types of defects found Others?
  • 262. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 262 The Dynamic/Pre-designed Spectrum
  • 263. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 263 Test Case Detail and Precision Trade-offs on the test documentation spectrum Precise tests allows less skilled testers, but not very flexible Imprecise tests can cover more conditions, but not very reproducible, especially across multiple testers Precise tests provide transparent test criteria, but are hard and expensive to maintain Imprecise tests are quick to write, but coverage can be hard to define and measure The degree to which a test effort is dynamic can be measured by counting the number of words of documentation (in your test cases and test procedures) per test hour of test execution
  • 264. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 264 Terms to Remember None for this section.
  • 265. Chapter 4: Test Design Techniques Section 6: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 266. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 266 Exercise: Omninet Test Techniques List the factors discussed in this section that would affect the choice of test techniques and extent of documentation for Omninet. Discuss.
  • 267. Chapter 5: Test Management Test Engineering Foundation Essential Knowledge for Test Professionals
  • 268. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 268 5. Test Management 1. Test organization 2. Test planning and estimation 3. Test progress monitoring and control 4. Configuration management 5. Risk and testing 6. Incident management
  • 269. Chapter 5: Test Management Section 1: Test organization Test Engineering Foundation Essential Knowledge for Test Professionals
  • 270. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 270 Test Organization Key concepts • The importance of independent testing • The benefits and drawbacks of independent testing • Different team members to be considered for the organization of a test team • Tasks of typical test leader and tester Terms to remember
  • 271. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 271 What Is the Test Team’s Job? Don Quixote: Lone Champion of Quality? By misunderstanding your role in the company, you can do yourself great political harm Test/quality control Risk management Quality assessment Quality assurance Test/QC management Ensuring product quality through process Make sure you’re competent, staffed, and politically supported for your role
  • 272. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 272 Test Team within the Company Test team as developer subgroup Independent test team Part of development team Okay for startups Not independent Hard to advocate quality Completely independent Bureaucratic for small companies Preserves tester independence To preserve independence, the test team must adopt a service- oriented attitude in most companies
  • 273. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 273 Independence Considerations Benefits See more, other and different defects If in doubt, it’s a bug Verify assumptions of specification and implementation Credibility Tester career path Potential traps Isolation from the development team Seen as bottleneck Programmers lose sense of responsibility for quality
  • 274. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 274 Test Leads Devise test strategies, plans Write or review test policy Consult on testing for other project activities Test estimation Test resource acquisition Lead specification, preparation, implementation, and execution of tests Monitor and control the test execution Adapt the test plan based on test results Ensure configuration management of testware Ensure traceability Measure test progress, evaluate the quality of the testing and the product Plan any test automation Select tools and organize any tester training Ensure implementation of the test environment Schedule tests Write test summary reports
  • 275. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 275 Testers Review and contribute to test plans Analyze, review and assess user requirements, specifications Create test suites, cases, data and procedures Set up the test environment Implement tests on all test levels Execute and log the tests, evaluate results and document problems found Monitor testing using the appropriate tools Automate tests Measure performance of components and systems Review each others’ tests
  • 276. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 276 Refining the Tester Position Test engineers Technical peers of programmers Chose testing as a specialty Write test cases, organize test suites Create, customize and use advanced test tools Have unique skills Test technicians Skilled and experienced tester May be an aspiring test engineer Runs tests Reports bugs Updates test status Assists the test engineers Other test team members System and database administrators Release and configuration engineers Test toolsmiths The right positions for your team depends on the skills you need. Be sure to balance skills and positions across the test team—skills are complimentary and the whole can be greater than the sum of the parts.
  • 277. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 277 Using Amateur Testers On many projects, amateur testers (people who do not test for their living) are used as part (or all) of the test team Such people typically include project managers, quality managers, programmers, business and domain experts, or infrastructure or IT operators Such a test team often possesses strong skills in some areas (business domain or technology), but weak skills in others, and no skills or substantial experience with testing Test is a special field, with special skills (of which the basics are covered in this course)
  • 278. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 278 Balancing the Skills Good test teams have right mix of skills based on tasks and activities Application (business) domain expert Understands intended behavior Skilled tester Knows quality risks and test techniques Technical guru Aware of technical issues and limitations What is the right mix for… …Internet appliance testing? …nuclear medicine testing? …your project? The appropriate depth and length of each arrow in the figure depends on the project, process, and product
  • 279. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 279 Case Study: Positions, Organization, Project
  • 280. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 280 Terms to Remember Test leader Test manager Tester
  • 281. Chapter 5: Test Management Section 1: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 282. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 282 Exercise: Omninet Test Team Imagine you are the test manager for the Omninet project, in charge of integration and system testing. Sketch an organizational chart that shows how you would organize the Omninet test team. Where would the test team fit into the Omninet project team? Discuss.
  • 283. Chapter 5: Test Management Section 2: Test planning and estimation Test Engineering Foundation Essential Knowledge for Test Professionals
  • 284. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 284 Test Planning and Estimation Key concepts • Levels and objectives of test planning • Purpose and content of the test plan • Test planning for a project, levels, targets • Test preparation and execution for the plan • Test exit criteria • Estimation via metrics and expertise • Test estimation factors Terms to remember
  • 285. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 285 Developing Test Plans Why write (and update) test plans? Confront challenges, crystallize thinking, adapt to change Communicate plan to testers, peers, managers Consider multiple test plans when tests have… Different time periods (e.g., phases and levels) Different methodologies and tools (e.g., performance and functionality) Different objectives (e.g., system test and beta test) Different audiences (e.g., hardware and software test) …but then you may want a master test plan Circulate one or two drafts Promotes early feedback and discussion Prevents wasted time if you’re on the wrong track
  • 286. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 286 Test Planning Activities Define test approach, test levels Integrate, coordinate testing into the life cycle Decide who, what, when, how of testing Assign resources for test tasks Define the test documentation Set the level of detail for test cases, procedures in order to provide enough information to support reproducible test preparation and execution Select test monitoring, controlling, and reporting metrics, charts, and reports (deliverables)
  • 287. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 287 IEEE 829 Test Plan A test plan is a subproject plan for the testing part of a project, and includes the sections shown in the next slide You can adapt the IEEE 829 outline for use for each detail (e.g., level or phase) test plan as well as the master test plan You can create you own template or outline, too Test planning influences (and is influenced by) test policy of the test organization, the scope of testing, objectives, risks, constraints, criticality, testability, and the availability of resources
  • 288. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 288 IEEE 829 Test Plan Outline Test plan identifier Introduction Test items (i.e., what’s delivered for testing) Features to be tested Features not to be tested Approach (strategies, organization, extent of testing) Item pass/fail criteria Test criteria (e.g., entry, exit, suspension and resumption) Test deliverables (e.g., reports, charts, etc.) Test tasks (or at least key milestones) Environmental needs Responsibilities Staffing and training needs Schedule Risks and contingencies (quality [product] and project risks) Approvals
  • 289. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 289 Transitions: Entry Criteria Entry criteria measure whether the system is ready for a particular test phase Deliverables ready? Lab ready? Teams ready? These tend to become increasingly rigorous as the phases proceed
  • 290. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 290 Sample Entry Criteria System Test can begin when: 1. Bug tracking and test tracking systems are in place. 2. All components are under formal, automated configuration and release management control. 3. The Operations team has configured the System Test server environment, including all target hardware components and subsystems. The Test Team has been provided with appropriate access to these systems. 4. The Development Teams have completed all features and bug fixes scheduled for release. 5. The Development Teams have unit- tested all features and bug fixes scheduled for release. 6. Less than 50 must-fix bugs (per Sales, Marketing, and Customer Service) are open against the release. 7. The Development Teams provide software to the Test Team 3 business days prior to starting System Test. 8. The Test Team completes a 3 day “smoke test” and reports on the results to the System Test Phase Entry meeting. 9. The Project Management Team agrees in a System Test Phase Entry Meeting to proceed. The following topics will be resolved in the meeting: Whether code is complete. Whether unit-testing is complete. Assign a target fix date for any known “must-fix” bugs (no later than 1 week after System Test Phase Entry).
  • 291. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 291 Transitions: Continuation Criteria Continuation criteria measure whether testing can efficiently, effectively proceed Test environment problems Test-blocking bugs in system under test “Continuation criteria” is a polite way of saying “stopping criteria” in the reverse Stopping a test phase is seldom popular
  • 292. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 292 System Test will continue if: 1. All software released to the Test Team is accompanied by Release Notes. 2. No change is made to the system, whether in source code, configuration files, or other setup instructions or processes, without an accompanying bug report. Should a change be made without a bug report, the Test Manager will open an urgent bug report requesting information and escalate to his manager. 3. The open bug backlog (“quality gap”) remains less than 50. The daily and rolling closure periods remain less than 14 days (on average, bugs are fixed within two weekly release cycles). 4. Twice-weekly bug review meetings occur until System Test Phase Exit to manage the open bug backlog and bug closure times. Sample Continuation Criteria
  • 293. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 293 Transitions: Exit Criteria Exit criteria measure whether the test phase can be deemed complete Thoroughness measures, such as coverage of code, functionality or risk Estimates of defect density or reliability measures Cost Residual risks, such as defects not fixed or lack of test coverage in certain areas Schedules such as those based on time to market Remember that these are business decisions
  • 294. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 294 System Test will end when: 1. No changes (design/code/features), except to address System Test defects, occurred in the prior 3 weeks. 2. No panic, crash, halt, wedge, unexpected process termination, or other stoppage of processing has occurred on any server software or hardware for the previous 3 weeks. 3. No client systems have become inoperable due to a failed update during System Test. 4. The Test Team has executed all the planned tests against the GA-candidate software. 5. The Development Teams have resolved all “must-fix” bugs per Sales, Marketing, and Customer Service. 6. The Test Team has checked that all issues in the bug tracking system are either closed or deferred, and, where appropriate, verified by regression and confirmation testing. 7.. The test metrics indicate: product stability and reliability; completion of all planned tests; adequate coverage of the critical quality risks. 8. The Project Management Team agrees that the product, as defined during the final cycle of System Test, will satisfy the customer’s reasonable expectations of quality. 9 .The Project Management Team holds a System Test Phase Exit Meeting and agrees that we have completed System Test. Sample Exit Criteria
  • 295. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 295 Developing a Work-Breakdown-Structure What are the major stages of a testing subproject? Planning Staffing (if applicable) Test environment acquisition and configuration Test development Test execution Break down to discrete tasks within each stage What test suites are required for the critical risks? E.g., functionality, performance, error handling, etc. For each suite, what tasks are required? Tasks should be short in duration (e.g., a few days) “Inch-pebbles” as well as “milestones”
  • 296. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 296 Estimation There are two general approaches for estimation (including test estimation) Estimating individual tasks by the owner of these tasks or by experts (bottom-up via work-breakdown-structure) Estimating the testing effort based on metrics of former or similar projects or based on typical values (top-down or bottom up via parametric models, rules of thumb, etc.) While both are useful, drawing upon team wisdom to create a work-breakdown-structure, then applying models and rules of thumb to check and adjust the estimate, is usually more accurate (and more defensible)
  • 297. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 297 Factors to Consider in Test Estimation Testing is complex, influenced by: Process factors: pervasive testing, change control, process maturity, lifecycle, development and test processes, earlier test phases, (estimated vs. actual) bug levels and fixing Material factors: tools, test system, test and debugging environments, project documentation, similarity People factors: skills, expectations, support, relationships Delaying factors: complexity, many stakeholders, too much newness, geographical distribution, need for detailed test documentation, tricky logistics, fragile test data Understand estimation techniques and these factors Deviation from the test estimate can arise from outside factors and events before or during testing
  • 298. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 298 Test Strategies Analytical: e.g., risk-based and requirement-based Model-based: e.g., stochastic or statistical testing Methodical: following a pre-planned methodology, e.g., failure-based, checklist-based, and quality-based Process/standard-compliant: e.g., IEEE 829 standard or Agile methodology like test-driven development Dynamic: an approach more reactive to events and conditions than pre-planned or pre-designed Consultative or directed: test coverage driven primarily by outside (non-tester) advice, guidance Regression-averse: reuse of existing test material, extensive automation of tests, standard test suites
  • 299. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 299 Terms to Remember Risk analysis Test level Entry criteria Exit criteria Test environment requirements Documentation requirements Test plan Test approach Test strategy Test-driven development
  • 300. Chapter 5: Test Management Section 2: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 301. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 301 Exercise: Omninet Test Plan Outline a test plan for Omninet Using the IEEE 829 template as a starting point, write down two to five bullet items or high-level statements in each section. Discuss.
  • 302. Chapter 5: Test Management Section 3: Test progress monitoring and control Test Engineering Foundation Essential Knowledge for Test Professionals
  • 303. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 303 Test Progress Monitoring and Control Key concepts • Common metrics used for monitoring test preparation and execution • Understand and interpret test metrics The purpose and content of the test report document according to IEEE 829 standard Terms to remember
  • 304. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 304 Common Test Metrics Percentage completion of test case preparation (or percentage of planned test cases prepared) Percentage completion of preparing the test environment Test case execution (e.g. number of test cases run/not run, and number of test cases passed/failed) Defect information (e.g. defect density, defects found and fixed, failure rate, and retest results) Coverage of requirements, risks or code by tests, including pass/fail results Level of confidence (subjective) of testers in the product Dates of test milestones Testing costs, including the cost of finding the next defect or running the next test compared to the benefit
  • 305. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 305 Test Reporting Summarize/analyze the test results Key events (e.g., meeting exit criteria) Analysis (for recommendations, guidance) of… • …remaining defects • …cost/benefit of more testing • …outstanding risks • …level of confidence Metrics gathered to assess: Test objectives adequacy for test level Test approaches adequacy Testing effectiveness per objectives
  • 306. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 306 Test Control Guiding and corrective actions due to test information and metrics, which may affect testing activities—or other software life cycle activities Examples of test control actions: Risk-triggered test re-prioritizing Test schedule adjustments due to availability of a test environment Set an entry criterion requiring re-testing of bug fixes by developers before integration into a build
  • 307. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 307
  • 308. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 308
  • 309. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 309
  • 310. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 310
  • 311. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 311 IEEE 829 Test Summary Report A test summary report describes the results of a given level or phase of testing, and includes the following sections Test summary report identifier Summary (e.g., what was tested, what the conclusions are, etc.) Variances (from plan, cases, procedures) Comprehensive assessment Summary of results (e.g., final metrics, counts) Evaluation (of each test item vis-à-vis pass/fail criteria) Summary of activities (resource use, efficiency, etc.) Approvals May be delivered within or at end of a test level
  • 312. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 312
  • 313. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 313
  • 314. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 314 IEEE 829 Test Log A test log records the relevant details about test execution, and includes the following sections Test log identifier Description (items under test [with version numbers], test environment, etc.) Activity and event entries (test-by-test, event-by-event information on the test execution process, the results of the tests, environmental changes or issues, bugs, incidents, or anomalies observed, testers involved, any suspension or blockage of testing, changes to the plan, impact of change, etc.) Tracking spreadsheets that captures these and many other details are often used for this purpose
  • 315. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 315 Terms to Remember Test monitoring Test control Defect density Test coverage Failure rate Test report or test summary report
  • 316. Chapter 5: Test Management Section 3: Amusement Test Engineering Foundation Essential Knowledge for Test Professionals
  • 317. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 317 Is This Helpful?
  • 318. Chapter 5: Test Management Section 3: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 319. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 319 Exercise: Reporting Bad News On the following slides, you’ll see a text description following by charts illustrating a project in trouble. How would you present these test results to a project team? Discuss.
  • 320. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 320 The Common Themes The project is SpeedyWriter, a browser-hosted word processing application Development ran the Component and Integration Test phase Component Test: December 11 through 29 Integration Test: December 18 through January 12 The project is now in the System Test Phase Cycle one: January 8 through 14 Today is January 15 The System Test Phase Exit Meeting is planned for January 29 (after two more one-week test cycles)
  • 321. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 321 1: Too Many Bugs, Not Enough Time The Test team is working productively and finding a few more bugs than expected Furthermore, a large backlog (about 30) exists Given the current bug find rate—which is not decreasing—and the backlog, the plan to finish testing in two weeks is in jeopardy The high bug find rate forced you to skip a large number of tests in cycle one
  • 322. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 322
  • 323. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 323
  • 324. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 324
  • 325. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 325
  • 326. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 326 2: No Stable Test Environment The Test team is totally blocked The Operations and Development teams were unable to install a working build Monday: Wrong software version on Web server Tuesday: Operating system reinstall required on Web server Wednesday: Five uninstallable builds delivered Thursday: Clients couldn’t see server on net Friday: Application crashed server repeatedly on start Saturday and Sunday: Browser died (GPF) repeatedly. Pages and cell phone calls not returned Testing would start but then stop as the “onion peeling” exercises occurred
  • 327. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 327
  • 328. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 328
  • 329. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 329
  • 330. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 330
  • 331. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 331 3: Inadequate Component/Integration Testing The Test team found many bugs in cycle one Extra testers added to meet plan Problems are myriad and even in basic areas that simple testing would have caught E.g., can’t save a document longer than one page The number and nature of problems blocks many more sophisticated tests The Development team has been warning Test informally that no time was in the schedule for sufficient component and integration testing
  • 332. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 332
  • 333. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 333
  • 334. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 334
  • 335. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 335
  • 336. Chapter 5: Test Management Section 4: Configuration management Test Engineering Foundation Essential Knowledge for Test Professionals
  • 337. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 337 Configuration Management Key concept How configuration management supports testing Terms to remember
  • 338. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 338 Testing and Configuration Management Configuration management establishes, maintains the integrity of the items that make up the software or system through the project and product life cycle  For testing, configuration management: Allows management of testware and results Ensures every item in test can be related back to known system components Supports delivery of a test release into the test lab During project and test planning, the configuration management procedures and infrastructure (tools) should be chosen, documented, and implemented, so that no surprises occur at test execution time
  • 339. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 339 Key Tasks of Configuration Management Store and control access to the items that make up the system (a.k.a., source code control, though it goes beyond code) Identify and document the items under control Allow change to control items through an orderly process (e.g., change control boards) Report on changes pending, underway, and complete Verify completeness of implementation
  • 340. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 340 Test Release Management Release schedule (i.e., weekly? daily? hourly?) Update apply (process to install new build) Update unapply (process to remove bad build) Build naming (revision level); e.g., X.01.017 Interrogation (process to determine rev. level) Synchronizing with databases, other systems, etc. Roles and responsibilities for each step
  • 341. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 341 IEEE 829 Test Item Transmittal Report A test item transmittal report describes the items being delivered for testing, and includes the following sections Test item transmittal report identifier Transmitted items (include item names and revision numbers) Location (where, what media, labeling, etc.) Status (bugs fixed, changes introduced, etc.) Approvals More commonly used are release notes, which include some of this information, usually informally
  • 342. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 342 Terms to Remember Baseline Change control Configuration management Version control
  • 343. Chapter 5: Test Management Section 4: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 344. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 344 Exercise: Omninet Configuration Mgmt Identify the major items that you think would require configuration management for Omninet. Which of these items are important for test release management? Discuss.
  • 345. Chapter 5: Test Management Section 5: Risk and testing Test Engineering Foundation Essential Knowledge for Test Professionals
  • 346. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 346 Risk and Testing Key concepts • Hazards and potential risks • Risk as a possible problem threatening the achievement of stakeholders’ objectives • Project as opposed to product risks Terms to remember
  • 347. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 347 Project Risks A previous chapter introduced the idea of testing as a way of managing product or quality risks Testing is also subject to risk A risk is the possibility of a negative outcome, and that would include events like late test releases, environment problems, etc. To discover risks to the testing effort, ask yourself and other stakeholders: What could go wrong on the project that would delay or invalidate your test plan and/or estimate? What kind of unacceptable testing outcomes do you worry about?
  • 348. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 348 Handling Project Risks For each project risk, you have four options: Mitigation: Reduce the likelihood or impact through preventive steps Contingency: Have a plan in place to reduce the impact Transfer: Get some other party to accept the consequences Ignore: Do nothing about it (best if both likelihood and impact are low!) Another typical risk-management option, buying insurance, is usually not available
  • 349. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 349 Sample Risks, Mitigations and Contingencies Logistics or product quality problems block tests  Careful planning, good bug triage, robust test design Test deliverables won’t install  Smoke testing, nightly builds, uninstall process Excessive change invalidates results, requires test updates  Good change control processes, robust test design, limited test documentation, escalation to management Insufficient or unrealistic test environment(s)  Explain risks to management, outsource performance testing Test environment support unreliable  Good escalation process, sys. admin. skill in team Gaps in test coverage revealed during test execution  Exploratory testing, continuous test improvement Slips in test start dates and/or deliverables to test  Risk priority to drop tests, escalation to management Budget and/or staffing cuts  Risk priority to drop tests, well- rounded team, outsource testing Debugging in the test environment  Escalation to management, risk priority to drop/reschedule tests
  • 350. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 350 Other Project Risks to Consider Organizational factors Skill and staff shortages Personal and training issues Communication problems Lack of follow up on test results, including failure to use such results for process improvement Improper attitude toward or expectations of testing, whether by the testers or by others on the project Complex organization (e.g., distributed) Supplier issues Failure of a third party, including a test tool vendor Contractual issues Technical issues Problems in defining the right requirements Unachievable requirements (leading to blocked/ unrunnable tests) The quality of the design, code, tests and test data Highly complex system
  • 351. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 351 Terms to Remember Complexity Priority Risk Risk-based testing Risk identification Risk analysis Risk control Product risk Project risk
  • 352. Chapter 5: Test Management Section 5: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 353. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 353 Exercise: Omninet Testing Risks Identify three to five project risks for Omninet testing. Assess the likelihood and impact of each risk on a three-point scale (High, Medium, or Low). Out of your four options, pick one or two appropriate steps to take to manage the risk. Discuss.
  • 354. Chapter 5: Test Management Section 6: Bug or incident management Test Engineering Foundation Essential Knowledge for Test Professionals
  • 355. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 355 Bug or Incident Management Key concepts • The content of a bug or incident report • Writing an bug or incident report Terms to remember
  • 356. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 356 Goals and Audience Incident or bug reports often have the following goals: To provide detailed information about the incident or bug to those who need it To be part of the aggregate data to analyze (e.g., bug charts) To lead to development and test process improvement Typical readers of bug reports include: Developers: who need to fix the problem reported Managers: who need to make resource-allocation/ prioritization decisions about the problem Technical support staff: who need to be aware of deferred/ unfixed issues at ship time Testers: who need to know the current state of the system
  • 357. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 357 1. Structure: test carefully 2. Reproduce: test it again 3. Isolate: test it differently 4. Generalize: test it elsewhere 5. Compare: review similar test results 6. Summarize: relate test to customers 7. Condense: trim unnecessary information 8. Disambiguate: use clear words 9. Neutralize: express problem impartially 10. Review: be sure Good bug reports on a problem can differ in style and content Ten Steps to a Good Bug Report
  • 358. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 358 Structure Structured testing is the foundation of good bug reports Use deliberate, careful approach to testing Follow written test cases or run automated ones per written or standardized process Use charters to structure exploratory testing Take careful notes Bug reporting begins when expected and observed results differ Sloppy testing results in sloppy bug reports
  • 359. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 359 Reproduce Always check reproducibility of failure as part of writing bug report  Three times is a good rule of thumb Document a crisp sequence of actions that will reproduce the failure Report intermittent, hard-to-repeat failures Note failure incidence rate (e.g., 1 in 3 tries) Summary should mention intermittence Clean set of steps to reproduce addresses the “irreproducible” issue head-on
  • 360. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 360 Isolate Change variables that may alter symptom Make changes one by one Requires thought and understanding of system under test May not be immediately obvious Can be extensive Match amount of effort to severity of problem Avoid getting into debugging activities Good isolation shows due diligence and gives developers head start on debugging
  • 361. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 361 Generalize Look for related failures in system under test Does the same failure occur in other modules or locations? Are there more severe symptoms of the same fault? Avoid confusing unrelated problems Same symptom can arise from different bugs  Generalizing reduces duplicate bug reports and refines understanding of failure
  • 362. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 362 Compare Examine results for similar tests Same test run against earlier versions Similar conditions, other tests, same version Is failure a regression? Change introduces defect not in earlier versions Usually found when previously passed tests fail Not always possible Test previously blocked, reinstall impractical Tested feature unavailable in earlier versions
  • 363. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 363 Summarize The summary is often the only part of the bug report read Put a short “tag line” on each report Capture failure and impact on customer ≅ Analogy: Newspaper headline Harder than it seems Testers must invest time in summary  Advantages of good summaries Get management attention Help set priorities Name bug report for developers
  • 364. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 364 Condense Eliminate extraneous words or steps Reread report carefully Avoid both cryptic commentary and droning on Are any details or actions irrelevant? Everyone’s time is precious… …so don’t waste any of it on unnecessary verbiage… …but don’t cut any meat, either
  • 365. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 365 Disambiguate Remove, rephrase, or expand vague, misleading, or subjective words and statements Make sure report is not subject to misinterpretation  Goal: Clear, indisputable statements of fact ∴ Lead developer by the hand to bug
  • 366. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 366 Neutralize Deliver bad news gently Be fair-minded in wording and implications Avoid: Attacking developers Criticizing the underlying error Attempting humor Using sarcasm Temper tantrums? Confine bug reports to statements of fact You never know who’ll read the reports
  • 367. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 367 Review Each tester should submit each bug report to one or more test peers for a review Reviewing peers should: Make suggestions to improve report Ask clarifying questions Even challenge “bugginess” if appropriate Test team should only submit best possible bug reports, given time constraints appropriate to priority of bug
  • 368. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 368 IEEE 829 Test Incident (Bug) Report A test incident (or bug) report describes a test event needing further investigation, especially a bug, and includes the following sections Test incident report identifier Summary (one line, especially impact on stakeholders) Incident description (inputs, expected results, actual results, anomalies, date and time seen, environment, test in progress, reproducibility, reported by, and other details) Impact (on testing, project, product, stakeholders, etc.) Strictly, incident reports describe any questionable behavior, while bug reports describe behavior known due to bugs rather than bad tests or test data, tester errors, test environment problems, and the like
  • 369. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 369 Bug reports move through a series of states (lifecycle or workflow) to resolution In each non-terminal state, a manager or the bug triage committee specifies an owner who is to move the bug to the next state Bug tracking systems can and should implement and automate these workflows, but project team and management support make the workflow to work! Bug Report Lifecycle or Workflow
  • 370. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 370 Other Information to Include Additional information about the configuration of the software or system The phase of introduction, detection, and removal of the bug Urgency/priority to fix Conclusions and recommendations Risks, costs, opportunities, and benefits associated with fixing/not fixing this bug Change history, especially for each state change
  • 371. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 371 Terms to Remember Incident logging Tracking and analysis Incident management tools
  • 372. Chapter 5: Test Management Section 6: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 373. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 373 Exercise: Omninet Bug Report The next page shows a poorly-written bug report for the Omninet project. Try to improve it, making some reasonable assumptions about the testing done. Try to cover all points addressed in the IEEE 829 incident report standard. You may assume the instructor is the tester who wrote the bug report and ask questions to help fill in any missing data. Discuss.
  • 374. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 374 Summary: Session lasts too long. Steps to Reproduce: 1. Swipe credit card in payment subsystem. 2. Buy some blocks of time. 3. Start a stopwatch when surfing starts. 4. Session will last between one and five minutes too long.
  • 375. Chapter 6: Tool Support for Testing Test Engineering Foundation Essential Knowledge for Test Professionals
  • 376. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 376 6. Tool Support for Testing 1. Types of test tools 2. Effective use of tools, potential benefits and risks 3. Introducing a tool into an organization
  • 377. Chapter 6: Tool Support for Testing Section 1: Types of test tools Test Engineering Foundation Essential Knowledge for Test Professionals
  • 378. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 378 Types of Test Tools Key concepts Different types of test tools • Programmers’ test tools Terms to remember
  • 379. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 379 Test Tool Classification Tools can be classified by the primary testing activities that they support For most classifications of tool, there are both commercial and freeware options available  Tools automate repetitive tasks, make testing more efficient Tools improve the reliability of testing; e.g., automating large data comparisons, generating load Tools are either intrusive or non-intrusive Intrusive tools have probe effects Non-intrusive tools are generally more expensive Some tools are primarily for developers
  • 380. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 380 Test Management Tools Provide traceability of tests, test results and incidents to the test basis Allow logging of test results, and generation of progress reports Help manage of tests and testing Interface to test execution, bug tracking, and requirement management tools Provide version control or interface with an external configuration management tool Perform quantitative analysis (metrics) related to the tests
  • 381. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 381 Requirements Management Tools Store requirement statements Check for consistency and undefined (missing) requirements Allow requirements to be prioritized Enable individual tests to be traceable to (and reported in terms of) requirements, functions, and/or features
  • 382. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 382 Bug (or Defect or Incident) Tracking Tools Store and manage bug reports Facilitate bug prioritization/ classification Provide state-based workflow, including assignment of actions to people (e.g., fix, confirmation test, etc.), Enable monitoring a project’s bugs and bug status over time, especially vis-à-vis trends Provide support for statistical analysis (often through export; e.g., to Excel) Create reports and graphs (though often of limited usefulness due to the variation in needs)
  • 383. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 383 Configuration Management Tools Store information about versions and builds of software and testware Enable traceability between testware and software work products and product variants Help with developing and testing on multiple configurations of hardware/ software environments
  • 384. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 384 Review Process Support Tools Store information about review processes Store and communicate review comments Report on defects and effort Manage references to review rules and/or checklists Provide aid for online reviews Track traceability between documents and source code
  • 385. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 385 Static Analysis Tools Primarily used by developers Find defects before dynamic testing Enforce of coding standards Analyze structures and dependencies Aid in understanding source code Calculate metrics from the code
  • 386. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 386 Modeling and Design Tools Primarily used by developers Help create models of the system Validate models Aid in generating some test cases based on the model
  • 387. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 387 Test Design Tools Generate test inputs or the actual tests from: Requirements Graphical user interface Design models Code Generate expected results (though the reliability of such test oracles is often limited) Generate test frameworks, templates, and stubs
  • 388. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 388 Test Data Tools Manipulate or create databases, files or data for use during test execution Create large volumes of useful test data Validate test data according to specific rules Analyze the data for frequency of conditions, etc. Scramble or anonymize live or customer data
  • 389. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 389 Test Execution Tools Run tests (i.e., submit inputs per an automated script and compare actual and expected results) Properly used, the script is an input-independent test procedure that allows you to repeat tests with different data, run similar tests, and easily maintain the tests Improperly used--e.g, typical capture-playback--the scripts are intertwined with the data and/or expected results, have hard-coded delays in them, and are quite fragile Capture-playback can be useful during exploratory testing Include comparators Produce analyzable logs Work at GUI, API, or CLI
  • 390. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 390 Test Frameworks, Harnesses, Simulators Replace or substitute for missing and/or potentially troublesome pieces of hardware Facilitate testing of unit(s) by generating and/or supporting drivers, stubs, and/or mock objects that replace portions of the system, which are unavailable or removed to isolate the unit Provide execution frameworks in middleware to test languages, operating systems or hardware
  • 391. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 391 Test Comparators Check files, databases, or test results against expectations, either during test execution or afterward May include an automated test oracle rather than comparing against static baselines Can be made smart by programming it to handle dynamic variables likes dates and time, or to ignore the order of records when sort-orders in reports or queries are ambiguous
  • 392. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 392 Coverage Measurement Tools Primarily used by developers Either intrusive or non-intrusive Measure the percentage of specific types of code structure that have been exercised Statements Branches or decisions Objects Function calls Check how thoroughly a set of tests has executed the measured type of structure
  • 393. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 393 Security Tools Check for computer viruses Simulate various kinds of attacks Simulate various security conditions
  • 394. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 394 Performance, Monitoring, Dynamic Analysis Primarily used by developers, though many testers use performance and monitoring tools Dynamic analysis tools often are used to check for time dependencies or memory leaks Monitor and report on how a system behaves under simulated usage conditions Generate various (hopefully realistic) load conditions for the application, a database, network or server, often per some script or programmed procedure Monitor, analyze, verify and report on usage of specific system resources, and give warnings of possible problems
  • 395. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 395 Other Tools Some tools are focused on particular applications Web-based performance testing tools Language-specific static analysis tools Security test tools Some target specific application areas like embedded systems Of course, testers also use spreadsheets (lots of them!) and databases Many developers use debugging tools
  • 396. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 396 Terms to Remember Configuration management tool Coverage measurement tool Debugging tool Driver Dynamic analysis tool Incident management tool Load testing tool Modelling tool Monitoring tool Performance testing tool
  • 397. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 397 Terms to Remember Probe effect Requirements management tool Review process support tool Security tool Static analysis tool Stress testing tool Stub Test comparator Test data preparation tool Test design tool Test harness
  • 398. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 398 Terms to Remember Test execution tool Test management tool Unit test framework tool
  • 399. Chapter 6: Tool Support for Testing Section 1: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 400. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 400 Exercise: Omninet Test Tools List the types tools discussed in this section which you think would be useful for testing Omninet. Discuss.
  • 401. Chapter 6: Tool Support for Testing Section 2: Effective use of tools, potential benefits and risks Test Engineering Foundation Essential Knowledge for Test Professionals
  • 402. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 402 Effective Use of Tools Key concepts Different scripting techniques for test execution tools, including data driven and keyword driven Potential benefits and risks of test automation Terms to remember
  • 403. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 403 Automation Opportunities and Risks Opportunities: Reduced repetitive work Improved consistency and repeatability Objective assessment (especially coverage, performance, etc.) Simplified reporting Risks: Unrealistic expectations Underestimating the time, cost and effort of development, execution, and maintenance Use of the tool for unsuitable tests
  • 404. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 404 Automated Testing Theory and Practice Test automation is typically software development A non-trivial process requiring significant time, skill, money Automation ROI typically takes longer than one project • Exception: simple API-based unit, component, integration tests • Exception: Some non-functional tests Usually, what’s automated is only test execution, not results analysis Automation can complicate analysis (and increase number) of failed tests If you focus just on test execution, test automation looks very efficient, but don’t forget that the automated tests had to be automated by someone to be automated. Intelligent automation of appropriate testing tasks is a good idea.
  • 405. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 405 Test Automation ROI A return on most test automation efforts requires: Sufficiently high regression risks to justify repeating tests A test system design that keeps the costs of running and maintaining the automated tests well below the cost of equivalent manual tests A relatively stable system Must recoup high up-front test development costs over multiple projects 0 10 20 30 40 50 60 70 80 90 100 Dev Exec Maint Manual Automated If maintenance is not required, we can recoup the investment in on the 4th execution. If maintenance is required for each execution, it will take 18 executions.
  • 406. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 406 Choosing Manual or Automated Testing Tests well-suited for manual testing Operations and maintenance Configuration and compatibility Error handling and recovery Localization Usability Installation and setup Documentation and help Manual, automated, or combined Functional Use cases (user scenarios) User interface Date and time handling Tests well-suited for automated testing Regression and confirmation Monkey (or random) Load, volume, and capacity Performance and reliability Standards compliance white-box, especially API-based unit, component, integration Static complexity and code analysis Inappropriate manual testing misleads people about test coverage Inappropriate automation usually fails Some tests mix techniques
  • 407. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 407 Tips for Success with Test Execution Tools Capture-replay seems to lower development costs, but execution and maintenance costs for large numbers of tests are high Data-driven tests separate inputs (the data) from procedures (scripts) that use the data Non-automation experts can create test data Keyword-driven tests use keywords (or action words) in a file read by the script Non-automation experts can create keyword files Automation experts create the scripts
  • 408. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 408 Tips for Success with Other Tools Performance testing tools require expertise in performance to design the tests and interpret the results Static analysis tools applied to source code can enforce coding standards, but a phased roll-out for existing code is needed to manage the volume of errors found Test management tools often have poor reporting facilities, so exporting data to a spreadsheet is usually the best way to produce the desired reports
  • 409. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 409 Static and White-Box Case Study Programmers code, test units in their local test area, then check in Each check-in triggers a static test, a build, and a white-box test While this approach is not nearly as glamorous or widely-marketing as the GUI techniques, it is cheap and quick We did this in six weeks with open-source tools See “Mission Made Possible,” www.stickyminds.com. I thank my colleague and coauthor on this article, Greg Kubaczkowski
  • 410. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 410 Terms to Remember Data-driven testing Keyword-driven testing Scripting language
  • 411. Chapter 6: Tool Support for Testing Section 2: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 412. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 412 Exercise: Omninet Test Tools Based on the discussion in this section, what kinds of test automation (if any) do you think would be successful for testing the kiosk? Based on the discussion in this section, what kinds of test automation (if any) do you think would be successful for testing the call center? Would you suggest automating the end-to-end testing (kiosk to call center/call center to kiosk), and, if so, how? Discuss.
  • 413. Chapter 6: Tool Support for Testing Section 3: Introducing a tool into an organization Test Engineering Foundation Essential Knowledge for Test Professionals
  • 414. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 414 Introducing a Tool Key concepts Introducing a tool into an organization The goals of a proof-of-concept for tool evaluation • Factors required for good tool support Terms to remember
  • 415. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 415 Knocking Down Barriers to Automation The following problems should be rectified before starting an automation effort Chaotic, reactive, time-crunched test process…get the manual testing process under control first, then automate Insufficiently skilled staff…hire or train Unstable, rapidly changing system…stabilize the system— especially at tested interfaces like GUI and APIs—first Unrealistic management expectations…effectively communicate automation benefits, limitations, business case (ROI), and costs No appropriate tool or practical oracle available for the important quality risks…build own tool or oracle—or wait When these problems remain, automation usually fails, creating organizational resistance to subsequent automation projects
  • 416. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 416 Objectives of an Automation Pilot Project Learn more about the tool and how to use it Adapt the tool and/or processes and practices to fit organization and systems Standardize ways of using, managing, storing and maintaining the tool and the test assets Assess the return on investment
  • 417. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 417 Success Factors for Tool Deployment Deploy the tool to the rest of the organization incrementally Adapt and improve the test processes to fit use of the tool Provide training, coaching , and mentoring for new users Define tool usage guidelines Learn ways to improve use of the tool continuously Monitor tool use and benefits
  • 418. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 418 Craig’s Test Tool Traps No clear strategy Unrealistic expectations Lack of stakeholder buy-in Inadequate or poor quality tool training Automating wrong thing (e.g., wrong tests) Choosing wrong tool Usability problems with tool Choosing wrong vendor Unstable system under test Doing too much, too soon Underestimating time and resources needed Inadequate or unique test environment Right approach, wrong time Excessive cost From Rick Craig and Stefan Jaskiel’s Systematic Software Testing, Chapter 6
  • 419. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 419 Terms to Remember None for this section
  • 420. Chapter 6: Tool Support for Testing Section 3: Exercise Test Engineering Foundation Essential Knowledge for Test Professionals
  • 421. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 421 Exercise: Omninet Test Tool Introduction Would you use the call center or the kiosk as a pilot for automation? What barriers would you need to overcome to pilot the tool against the call center or kiosk? What test tool traps would you need to avoid? Discuss.
  • 422. Questions, Comments, and Discussion about the Course? Test Engineering Foundation Essential Knowledge for Test Professionals
  • 423. Bibliography Test Engineering Foundation Essential Knowledge for Test Professionals
  • 424. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 424 Beizer, Boris: Software Testing Techniques, 2e, Van Nostrand Reinhold, 1990. Black, Rex: Managing the Testing Process, 2e, John Wiley & Sons, 2001 Buwalda, Hans et al.: Integrated Test Design and Automation, Addison Wesley, 2001. Chrissis, Mary-Beth et al.: CMMI: Guidelines for Process Integration and Product Improvement, Addison Wesley, 2004. Copeland, Lee: A Practitioner’s Guide to Software Test Design, Artech House, 2004. Fewster, Mark and Graham, Dorothy: Software Test Automation, Addison Wesley, 1999. Hetzel, Bill: Complete Guide to Software Testing, QED, 1998. Kaner, Cem at al.: Lessons Learned in Software Testing, Wiley, 2002. Myers, Glenford: The Art of Software Testing, John Wiley & Sons, 1979. van Veenendaal, Erik, ed.: The Testing Practitioner, UTN Publishers, 2004. Syllabus Referenced Books
  • 425. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 425 ISTQB Glossary of testing terms (version 1.1): Referenced throughout syllabus. CMMI: Guidelines for Process Integration and Product Improvement: See section 2.1 IEEE Std 829-1998 - IEEE Standard for Software Test Documentation: See sections 2.3, 2.4, 4.1, 5.2, 5.3, 5.5, 5.6 IEEE Std 1028-1997 - IEEE Standard for Software Reviews: See section 3.2 ISO/IEC 9126-1:2001. Software Engineering – Software Product Quality: See section 2.3 IEEE 12207 / ISO/IEC 12207-1996. Software life cycle processes: See section 2.1 Syllabus Referenced Standards
  • 426. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 426 I am the Principal Consultant in Rex Black Consulting Services, Inc. My associates and I provide expert services in the following areas: Test team “bootstrapping” Turnkey test project execution Test automation Test execution and management tools Test training Test project and process consulting Distributed test coordination Test staff augmentation These services are available throughout the world. Should you have questions about these materials, or an interest in discussing ways in which RBCS can help your organizations, please contact me. Address: Rex Black Consulting Services, Inc. 31520 Beck Road Bulverde, TX 78163-3911 USA Phone: +1 (830) 438-4830 Fax: +1 (830) 438-4831 E-mail: Rex_Black@RexBlackConsulting.com Web: www.RexBlackConsulting.com Contacting Me
  • 427. Thank you for your attention and participation! I hope the material in this class and my books proves helpful for you in your career as a test professional! Test Engineering Foundation Essential Knowledge for Test Professionals
  • 428. Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 428 Change History Rev #. Date Released Change 1.0 Apr 29, 2004 Reworked, from “Test Fundamentals,” to fix the order, focus on exercises, cover the ISQTB Foundation syllabus, and improve timing. 1.01 Feb 8, 2005 Added figures for equivalence partitioning and boundary values 1.98 May 15, 2005 Revised to address the new Foundation syllabus. 1.99 Sep 21, 2005 Finalized for Foundation accreditation 2.0 Oct 13, 2005 Minor final changes to fully comply with Syllabus upon submission. 2.01 Nov 27, 2005 Minor corrections to achieve ISTQB accreditation.