PowerPoint presentation (785KB)


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

PowerPoint presentation (785KB)

  1. 1. Quality Assurance Institute’s (QAI) Effective Methods of Software Testing A Perspective Seattle Area Software Quality Assurance Group John Napier, CSQE, CSTE BEST Consulting
  2. 2. Purpose <ul><li>Value of Professional Associations </li></ul><ul><li>Value of Software Certification programs </li></ul><ul><li>QAI’s “Effective Methods of Software Testing” </li></ul>
  3. 3. Disclaimers <ul><li>The opinions expressed in this presentation are my own, and not those of: </li></ul><ul><ul><li>BEST Consulting </li></ul></ul><ul><ul><li>The Boeing Company </li></ul></ul><ul><ul><li>Quality Assurance Institute </li></ul></ul><ul><ul><li>SASQAG </li></ul></ul>
  4. 4. What its like in the trenches... <ul><li>No common language </li></ul><ul><li>“In my last job…” </li></ul><ul><li>No individual professional context </li></ul><ul><li>No organizational professional context </li></ul><ul><li>Can’t see the forest... </li></ul>
  5. 5. What’s a person to do? <ul><li>Establish a personal professional context </li></ul><ul><li>Get to know “the forest” </li></ul><ul><li>Help create a common language </li></ul><ul><li>Help bring best practices into your organization </li></ul>
  6. 6. The Quality Assurance Institute What’s QAI?
  7. 7. The Quality Assurance Institute <ul><li>An international association of 1000 organizations banded together to share best practices, benchmark data and experiences </li></ul><ul><li>Software quality focus </li></ul><ul><li>Certification Programs </li></ul><ul><ul><li>Certified Quality Analyst </li></ul></ul><ul><ul><li>Certified Software Quality Engineer </li></ul></ul>
  8. 8. QAI Certified Software Test Engineer (CSTE) <ul><li>Mandatory Skills </li></ul><ul><ul><li>Test Planning </li></ul></ul><ul><ul><li>Use of Test Tools </li></ul></ul><ul><ul><li>Test Performance </li></ul></ul><ul><ul><li>Defect Management </li></ul></ul><ul><li>Optional Skills </li></ul><ul><ul><li>Risk Analysis </li></ul></ul><ul><ul><li>Measurement </li></ul></ul><ul><ul><li>Test Case Design </li></ul></ul><ul><ul><li>Establishing and Maintaining a Test Environment </li></ul></ul>
  9. 9. ASQC* Certified Quality Software Engineer (CQSE) <ul><li>Experience </li></ul><ul><li>Professional Association </li></ul><ul><li>Software Engineering Body of Knowledge </li></ul><ul><ul><li>Exam </li></ul></ul><ul><li>Many other ASQC Certification Programs </li></ul>* American Society for Quality Control
  10. 10. Effective Methods of Software Testing Carolyn Fairbanks Q-Star Consultants 10-14 March 1997 Seattle
  11. 11. Effective Methods of Software Testing <ul><li>There are really only two issues </li></ul><ul><li>in software testing: </li></ul><ul><li>Requirements </li></ul><ul><li>Organizational Culture </li></ul>
  12. 12. QAI Test Materials Organization <ul><li>Test Strategy </li></ul><ul><li>Test Plan </li></ul><ul><li>Supporting Materials </li></ul><ul><ul><li>Metrics </li></ul></ul><ul><ul><li>Test Toolbox - Techniques </li></ul></ul><ul><ul><li>Test Conditions - Techniques </li></ul></ul><ul><ul><li>Test Case Design - Techniques </li></ul></ul>
  13. 13. Test Strategy Definition <ul><li>1. Type of Project </li></ul><ul><li>2. Type of Software </li></ul><ul><li>3. When Testing will Occur </li></ul><ul><li>4. Critical Success Factors </li></ul><ul><li>5. Who will Conduct the Testing </li></ul><ul><li>6. Tradeoffs </li></ul>
  14. 14. 1. Types of Project <ul><li>Traditional </li></ul><ul><li>Client/Server </li></ul><ul><li>Prototyping </li></ul><ul><li>Object Oriented </li></ul><ul><li>Purchased / Contracted </li></ul>
  15. 15. 2. Types of Software Batch Process Control Graphic User Interface Object Oriented Procedure Control Expert System Multi-Media Advanced Math Models Diagnostic Software Sensor and Signal Simulation DB Management Data Presentation Decision Analysis Pattern & Image Computer Operations Software Development Tools
  16. 16. 3. When will Testing Occur? <ul><li>Requirements Phase Activities </li></ul><ul><li>Design Phase Activities </li></ul><ul><li>Build Phase Activities </li></ul><ul><li>Test Phase Activities </li></ul><ul><li>Installation Phase Activities </li></ul><ul><li>Maintenance Phase Activities </li></ul>
  17. 17. 4. Critical Success Factors Correctness Authorization Data Integrity Audit Trail Continuity of Processing Service Levels Access Control Methodology Reliability Ease of Use Maintainable Portable Coupling Performance Ease of Operations
  18. 18. Testing Concerns Matrix Critical Success Factors Life Cycle Stages Correctness Requirements Complete, Clear, Correct, Testable, Feasible Design Etc
  19. 19. 7. Tradeoffs (What can be given up?) <ul><li>Schedule </li></ul><ul><li>Cost / Resources </li></ul><ul><li>Scope </li></ul><ul><li>Technology </li></ul><ul><li>Quality / Critical Success Factors </li></ul>
  20. 20. Some QAI Philosophy Test Life Cycle, Test Workbench
  21. 21. Testing Relationship Definitions - Testing Lifecycle Concept Operational or Business Need Define Requirements Design System Build System Unit Test Integration Test System Test Acceptance Test
  22. 22. Testing Relationship Definitions - Continuous Life Cycle Testing Operational or Business Need Define Requirements Design System Build System Unit Test Integration Test System Test Acceptance Test Verify Verify Verify Verify Validate Validate Validate Validate
  23. 23. Testing Relationship Definitions - Test Support Activities Operational or Business Need Define Requirements Design System Build System Unit Test Integration Test System Test Acceptance Test Verify Verify Verify Verify Validate Validate Validate Validate TEST OBJECTIVES TEST PLANNING TEST EXECUTION (STATIC) TEST EXECUTION (DYNAMIC) & TEST REPORTING
  24. 24. QAI Definitions - “Test Workbench” Concept Test Processes Test Toolbox Test Measures Validated Test Results Problems Products for Test Test Quality Control
  25. 25. The QAI Test Process 1. Set Test Objectives 2. Develop Test Plan 3. Execute Tests 4. Summarize and Report Results
  26. 26. Step 1 - Set Test Objectives <ul><li>Task 1 - Organize Test Team </li></ul><ul><li>Task 2 - Perform Risk Assessment </li></ul><ul><li>Task 3 - Set Test Objectives </li></ul><ul><li>Task 4 - Quality Control </li></ul>
  27. 27. Set Test Objectives Task 1 - Organize Test Team <ul><li>Define Test Responsibilities / Tasks </li></ul><ul><li>Determine People to Perform Tasks </li></ul>Worksheets 4,5
  28. 28. Set Test Objectives 2. Perform Risk Assessment <ul><li>Dimensions of Risk </li></ul><ul><ul><li>Criticality / Mission Impact </li></ul></ul><ul><ul><li>System Size, Scale and Complexity </li></ul></ul><ul><ul><li>System Environment and Stability </li></ul></ul><ul><ul><li>Reliability and Integrity </li></ul></ul><ul><ul><li>Technology Integration </li></ul></ul>Impact of Failure Likelihood of Failure Simple or Complex Scoring Approach
  29. 29. Set Test Objectives 3. Set Test Objectives <ul><li>Using: </li></ul><ul><ul><li>System Objectives </li></ul></ul><ul><ul><li>Test Strategy </li></ul></ul><ul><ul><li>Risk Score Analysis </li></ul></ul><ul><ul><li>Team Assignments </li></ul></ul><ul><li>Set Objectives </li></ul><ul><ul><li>Brainstorming </li></ul></ul><ul><ul><li>Relate Objectives to Function / Output </li></ul></ul><ul><ul><li>Relate Objectives to Events / System Input </li></ul></ul>Prioritize Each- Low Medium High Worksheet 6
  30. 30. Set Test Objectives Quality Control <ul><li>Objectives QC Checklist </li></ul><ul><li>Objectives Incident Report (Change Request) </li></ul>… lots of checklists!
  31. 31. Develop Test Plan <ul><li>1. Define Business Processes </li></ul><ul><li>2. Define Structural Attributes </li></ul><ul><li>3. Define Tests </li></ul><ul><li>4. Create Test / Function Matrix </li></ul><ul><li>5. Define Admin Requirements </li></ul><ul><li>6. Formulate System Test Plan </li></ul><ul><li>7. QC System Test Plan </li></ul><ul><li>8. Develop Unit Test Plan </li></ul><ul><li>9. QC Unit Test Plan </li></ul>
  32. 32. Develop Test Plan 1. Define Business Functions <ul><li>Describe what the system should do </li></ul><ul><ul><li>…stands alone as an activity </li></ul></ul><ul><ul><li>…is controllable </li></ul></ul><ul><ul><li>…can be a transition </li></ul></ul><ul><ul><li>…can be described by a process </li></ul></ul>
  33. 33. Business Function Matrix Function Description Evaluation Criteria Worksheet 8
  34. 34. Develop Test Plan 2. Define Structural Attributes <ul><li>Reliability </li></ul><ul><li>Efficiency </li></ul><ul><li>Integrity </li></ul><ul><li>Usability </li></ul><ul><li>Maintainability </li></ul><ul><li>Testability </li></ul><ul><li>Flexibility </li></ul><ul><li>Portability </li></ul><ul><li>Reusability </li></ul><ul><li>Interoperability </li></ul>Worksheet 9
  35. 35. Structural Attribute Matrix Attribute Priority Evaluation Criteria Description
  36. 36. Develop Test Plan 3. Define Tests <ul><li>Using </li></ul><ul><ul><li>Test Objectives Worksheet </li></ul></ul><ul><ul><li>Business Functions Worksheet </li></ul></ul><ul><ul><li>Structural Attributes Worksheet </li></ul></ul><ul><li>Define Tests </li></ul><ul><ul><li>Associate test factors, techniques and test types </li></ul></ul><ul><ul><li>Associate evaluation criteria with techniques and types of tests </li></ul></ul><ul><ul><li>Determine type of test which satisfies each objective </li></ul></ul>
  37. 37. Type of Test Techniques <ul><li>Functional </li></ul><ul><ul><li>Requirements </li></ul></ul><ul><ul><li>Regression </li></ul></ul><ul><ul><li>Error Handling </li></ul></ul><ul><ul><li>Manual Support </li></ul></ul><ul><ul><li>Intersystems </li></ul></ul><ul><ul><li>Control </li></ul></ul><ul><ul><li>Parallel </li></ul></ul><ul><li>Structural </li></ul><ul><ul><li>Stress </li></ul></ul><ul><ul><li>Execution </li></ul></ul><ul><ul><li>Recovery </li></ul></ul><ul><ul><li>Operations </li></ul></ul><ul><ul><li>Compliance (to process) </li></ul></ul><ul><ul><li>Security </li></ul></ul>
  38. 38. Test Data Sources <ul><li>Test Data / Test Desks </li></ul><ul><li>Production Data </li></ul><ul><li>Simulation / Prototyping </li></ul><ul><li>Review / Confirmation </li></ul><ul><li>Instrumentation / Tracing </li></ul><ul><li>Test Script </li></ul>
  39. 39. Test Definition <ul><li>Each Test is defined in terms of: </li></ul><ul><ul><li>Controls </li></ul></ul><ul><ul><li>Inputs </li></ul></ul><ul><ul><li>Outputs </li></ul></ul><ul><ul><li>Procedures </li></ul></ul>...At a high level
  40. 40. Develop Test Plan 4. Create Function / Test Matrix Tests Function Test 1 Test 2 Ect. Function #1 Function #2 25 17,30 5,7 45
  41. 41. Develop Test Plan 5. Define Admin Test Requirements <ul><li>Schedule </li></ul><ul><li>Budget </li></ul><ul><li>Resources </li></ul><ul><li>Testing Materials </li></ul><ul><li>Test Training </li></ul><ul><li>Tests to be Conducted </li></ul><ul><li>Defect Estimation </li></ul>Worksheets 13,14
  42. 42. Develop Test Plan 6. Formalize Test Plan <ul><li>Section I - General Information </li></ul><ul><li>Section 2 - Plan </li></ul><ul><li>Section 3 - Specifications and Evaluation </li></ul><ul><li>Section 4 - Test Description </li></ul>See Test Plan Example
  43. 43. Develop Test Plan 7. Test Plan Quality Control <ul><li>Quality Control Checklist </li></ul><ul><li>Test Plan Incident Report (Change Request) </li></ul>… even more checklists!
  44. 44. Develop Test Plan 8. Develop Unit Test Plan <ul><li>Using: </li></ul><ul><ul><li>System Test Plan </li></ul></ul><ul><ul><li>Program Initiation Document </li></ul></ul><ul><ul><li>Program Design / Specification Documents </li></ul></ul><ul><li>Develop </li></ul><ul><ul><li>Test Plan Administration </li></ul></ul><ul><ul><li>Subfunction / Test Matrix </li></ul></ul><ul><ul><li>Interface / Test Matrix </li></ul></ul>Worksheets 20,21,22
  45. 45. Develop Test Plan 9. Unit Test Plan Quality Control <ul><li>Checklists </li></ul>
  46. 46. Step 3 - Execute Tests <ul><li>Task 1 - Select Test Tools </li></ul><ul><li>Task 2 - Develop Test Cases </li></ul><ul><li>Task 3 - Execute Tests </li></ul><ul><li>Task 4 - Quality Control </li></ul>
  47. 47. QAI Tool Definition <ul><li>What’s a Test Tool? </li></ul><ul><ul><li>…any vehicle which assists in performing a test. </li></ul></ul><ul><ul><li>Examples- </li></ul></ul><ul><ul><ul><li>Decision Tables </li></ul></ul></ul><ul><ul><ul><li>Automated capture / playback </li></ul></ul></ul><ul><ul><ul><li>Checklists </li></ul></ul></ul><ul><ul><ul><li>Program Analyzer </li></ul></ul></ul>
  48. 48. Execute Tests Step 1 - Select Test Tools <ul><li>Using: </li></ul><ul><ul><li>System Test Plan </li></ul></ul><ul><ul><li>Unit Test Plan </li></ul></ul><ul><ul><li>Tool Selection Methodology </li></ul></ul>
  49. 49. Execute Tests Step 2 - Develop Test Cases Worksheet 23 Expected Result Test Condition Test Procedures
  50. 50. Execute Tests Step 3 - Execute Tests <ul><li>Test Activities Log </li></ul><ul><li>Regression Testing </li></ul><ul><ul><li>Full </li></ul></ul><ul><ul><li>Partial </li></ul></ul>Worksheet 24
  51. 51. Step 4 - Summarize and Report Test Results <ul><li>Task 1 - Record Defects </li></ul><ul><li>Task 2 - Perform Data Reduction </li></ul><ul><li>Task 3 - Develop Findings and Recommendations </li></ul><ul><li>Task 4 - Finalize Test Report </li></ul><ul><li>Task 5 - Test Report Quality Control </li></ul><ul><ul><li>Test Defect Report </li></ul></ul><ul><ul><li>Test Defect List </li></ul></ul><ul><ul><li>Test Defect Log </li></ul></ul>Worksheets 25,26,27,28
  52. 52. Supporting Material <ul><li>Metrics </li></ul><ul><li>Toolbox Techniques </li></ul><ul><li>Test Condition Analysis </li></ul><ul><li>Test Design Analysis </li></ul>
  53. 53. Test Metrics Method - Goal - Question - Metric
  54. 54. QAI Recommend Test Metrics <ul><li>Defect Reports </li></ul><ul><li>Requirements Implemented </li></ul><ul><li>Backlog of Defects (Not Corrected) </li></ul><ul><li>Status of - </li></ul><ul><ul><li>Project, Tests, Schedule, Budget </li></ul></ul><ul><li>Selected Test Metrics </li></ul>
  55. 55. QAI Test Metrics Selection Types General Subject 1 Involvement in Testing 4 Extent of Testing 3 Resources Consumed in Testing 20 Effectiveness of Testing 2 Assessment of Testing
  56. 56. The QAI Test Toolbox <ul><li>Data Modeling </li></ul><ul><li>Boundary Value Analysis </li></ul><ul><li>Capture/Playback </li></ul><ul><li>Cause-Effect Graphing </li></ul><ul><li>Change Control Trackers </li></ul><ul><li>Checklists </li></ul><ul><li>Checkpoint Review </li></ul><ul><li>Code Comparison </li></ul><ul><li>Compiler-based Analysis </li></ul><ul><li>Complexity-based Analysis </li></ul><ul><li>Compliance Checkers </li></ul><ul><li>Control Flow Analysis </li></ul><ul><li>Correctness Proofs </li></ul><ul><li>Coverage Based Analysis </li></ul><ul><li>Data Dictionary </li></ul><ul><li>Decision Tables </li></ul><ul><li>Defect Trackers </li></ul><ul><li>Desk Checking </li></ul><ul><li>Equivalence Partitioning </li></ul><ul><li>Error Guessing </li></ul>
  57. 57. The QAI Test Toolbox <ul><li>File Comparison </li></ul><ul><li>Flowcharts </li></ul><ul><li>Help Desk Trackers </li></ul><ul><li>Inspections </li></ul><ul><li>Instrumentation </li></ul><ul><li>Integrated Test Facility </li></ul><ul><li>Mapping </li></ul><ul><li>Model Animation </li></ul><ul><li>Model Balancing </li></ul><ul><li>Network Analyzers </li></ul><ul><li>Performance / Timing Analyzers </li></ul><ul><li>Population Analysis </li></ul><ul><li>Risk Matrix </li></ul><ul><li>Scoring </li></ul><ul><li>Snapshots </li></ul><ul><li>Symbolic Execution </li></ul><ul><li>System Logs </li></ul><ul><li>Test Case Generator </li></ul><ul><li>Test Data Generator </li></ul><ul><li>Test Scripts </li></ul>
  58. 58. The QAI Test Toolbox <ul><li>Test Stubs and Harnesses </li></ul><ul><li>Test Suite Managers </li></ul><ul><li>Tracing </li></ul><ul><li>Walkthroughs </li></ul>
  59. 59. The QAI Test Toolbox Test Tools Boundary Analysis Types of Functional and Structural Tests Population Analysis Checklists X X X X X
  60. 60. Test Condition Analysis <ul><li>Sources - </li></ul><ul><ul><li>1. Specification Decomposition </li></ul></ul><ul><ul><li>2. Population Analysis </li></ul></ul><ul><ul><li>3. Business Process Analysis </li></ul></ul><ul><ul><li>4. Structural Analysis </li></ul></ul><ul><ul><li>5. Test Transactions </li></ul></ul>Test Condition - Expected Result - Test Process
  61. 61. Identify Test Conditions <ul><li>1. System Specifications or Production Environment </li></ul><ul><li>2. Population Analysis </li></ul><ul><ul><li>Files and Tables </li></ul></ul><ul><ul><li>Screens </li></ul></ul><ul><ul><li>Field/Data Element </li></ul></ul>Worksheet 4 Test Condition - Expected Result - Test Process
  62. 62. Identify Test Conditions <ul><li>Field </li></ul><ul><li>Record </li></ul><ul><li>File </li></ul><ul><li>Relationships </li></ul><ul><li>Error </li></ul><ul><li>Use (Outputs) </li></ul><ul><li>Search </li></ul><ul><li>Match / Merge </li></ul><ul><li>Stress </li></ul><ul><li>Control </li></ul><ul><li>Attributes </li></ul><ul><li>States </li></ul><ul><li>Procedures </li></ul><ul><li>Control Flow </li></ul>3. Test Transaction Types Test Condition - Expected Result - Test Process
  63. 63. Identify Test Conditions <ul><li>4. Business Cases </li></ul><ul><ul><li>…business scenarios… </li></ul></ul><ul><li>5. Structural Analysis </li></ul><ul><ul><li>…decision points... </li></ul></ul>Test Condition - Expected Result - Test Process
  64. 64. Design Test Cases <ul><li>Design Test Cases </li></ul><ul><li>Develop Scripts / Sets </li></ul>Expand Test Condition s
  65. 65. Design Test Cases <ul><li>Expand Conditions into Test Cases using - </li></ul><ul><ul><li>Boundary Value Analysis </li></ul></ul><ul><ul><li>Equivalence Partitioning </li></ul></ul><ul><ul><li>Decision Tables </li></ul></ul><ul><ul><li>Code Coverage </li></ul></ul><ul><ul><ul><li>Statement Coverage </li></ul></ul></ul><ul><ul><ul><li>Condition Coverage </li></ul></ul></ul>Expand Test Condition s
  66. 66. Design Test Cases Test Scripts Actor/Scene Dialog Detailed Test Script Descriptions Script Reuse … Sam scowls at the report... “ Where did this come from?” Expand Test Condition s
  67. 67. QAI Summary - Ten Key Test Concepts <ul><li>Use strategic test questions to set direction </li></ul><ul><li>Develop Test Objectives </li></ul><ul><li>Use a cross-functional “test team” </li></ul><ul><li>Prepare a test plan, then execute it </li></ul><ul><li>Utilize both verification and validation methods </li></ul><ul><li>Collect and analyze defects </li></ul><ul><li>Prepare test management reports </li></ul>
  68. 68. QAI Summary - Ten Key Test Concepts <ul><li>Make test efforts reusable </li></ul><ul><li>Feed the continuous improvement process with information from testing </li></ul><ul><li>Continuously improve the process </li></ul>
  69. 69. Incorporating the Process Models within Models …Make them your own
  70. 70. Questions?