PowerPoint presentation (785KB)

  • 5,511 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
5,511
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
273
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Quality Assurance Institute’s (QAI) Effective Methods of Software Testing A Perspective Seattle Area Software Quality Assurance Group John Napier, CSQE, CSTE BEST Consulting
  • 2. Purpose
    • Value of Professional Associations
    • Value of Software Certification programs
    • QAI’s “Effective Methods of Software Testing”
  • 3. Disclaimers
    • The opinions expressed in this presentation are my own, and not those of:
      • BEST Consulting
      • The Boeing Company
      • Quality Assurance Institute
      • SASQAG
  • 4. What its like in the trenches...
    • No common language
    • “In my last job…”
    • No individual professional context
    • No organizational professional context
    • Can’t see the forest...
  • 5. What’s a person to do?
    • Establish a personal professional context
    • Get to know “the forest”
    • Help create a common language
    • Help bring best practices into your organization
  • 6. The Quality Assurance Institute What’s QAI?
  • 7. The Quality Assurance Institute
    • An international association of 1000 organizations banded together to share best practices, benchmark data and experiences
    • Software quality focus
    • Certification Programs
      • Certified Quality Analyst
      • Certified Software Quality Engineer
  • 8. QAI Certified Software Test Engineer (CSTE)
    • Mandatory Skills
      • Test Planning
      • Use of Test Tools
      • Test Performance
      • Defect Management
    • Optional Skills
      • Risk Analysis
      • Measurement
      • Test Case Design
      • Establishing and Maintaining a Test Environment
  • 9. ASQC* Certified Quality Software Engineer (CQSE)
    • Experience
    • Professional Association
    • Software Engineering Body of Knowledge
      • Exam
    • Many other ASQC Certification Programs
    * American Society for Quality Control
  • 10. Effective Methods of Software Testing Carolyn Fairbanks Q-Star Consultants 10-14 March 1997 Seattle
  • 11. Effective Methods of Software Testing
    • There are really only two issues
    • in software testing:
    • Requirements
    • Organizational Culture
  • 12. QAI Test Materials Organization
    • Test Strategy
    • Test Plan
    • Supporting Materials
      • Metrics
      • Test Toolbox - Techniques
      • Test Conditions - Techniques
      • Test Case Design - Techniques
  • 13. Test Strategy Definition
    • 1. Type of Project
    • 2. Type of Software
    • 3. When Testing will Occur
    • 4. Critical Success Factors
    • 5. Who will Conduct the Testing
    • 6. Tradeoffs
  • 14. 1. Types of Project
    • Traditional
    • Client/Server
    • Prototyping
    • Object Oriented
    • Purchased / Contracted
  • 15. 2. Types of Software Batch Process Control Graphic User Interface Object Oriented Procedure Control Expert System Multi-Media Advanced Math Models Diagnostic Software Sensor and Signal Simulation DB Management Data Presentation Decision Analysis Pattern & Image Computer Operations Software Development Tools
  • 16. 3. When will Testing Occur?
    • Requirements Phase Activities
    • Design Phase Activities
    • Build Phase Activities
    • Test Phase Activities
    • Installation Phase Activities
    • Maintenance Phase Activities
  • 17. 4. Critical Success Factors Correctness Authorization Data Integrity Audit Trail Continuity of Processing Service Levels Access Control Methodology Reliability Ease of Use Maintainable Portable Coupling Performance Ease of Operations
  • 18. Testing Concerns Matrix Critical Success Factors Life Cycle Stages Correctness Requirements Complete, Clear, Correct, Testable, Feasible Design Etc
  • 19. 7. Tradeoffs (What can be given up?)
    • Schedule
    • Cost / Resources
    • Scope
    • Technology
    • Quality / Critical Success Factors
  • 20. Some QAI Philosophy Test Life Cycle, Test Workbench
  • 21. Testing Relationship Definitions - Testing Lifecycle Concept Operational or Business Need Define Requirements Design System Build System Unit Test Integration Test System Test Acceptance Test
  • 22. Testing Relationship Definitions - Continuous Life Cycle Testing Operational or Business Need Define Requirements Design System Build System Unit Test Integration Test System Test Acceptance Test Verify Verify Verify Verify Validate Validate Validate Validate
  • 23. Testing Relationship Definitions - Test Support Activities Operational or Business Need Define Requirements Design System Build System Unit Test Integration Test System Test Acceptance Test Verify Verify Verify Verify Validate Validate Validate Validate TEST OBJECTIVES TEST PLANNING TEST EXECUTION (STATIC) TEST EXECUTION (DYNAMIC) & TEST REPORTING
  • 24. QAI Definitions - “Test Workbench” Concept Test Processes Test Toolbox Test Measures Validated Test Results Problems Products for Test Test Quality Control
  • 25. The QAI Test Process 1. Set Test Objectives 2. Develop Test Plan 3. Execute Tests 4. Summarize and Report Results
  • 26. Step 1 - Set Test Objectives
    • Task 1 - Organize Test Team
    • Task 2 - Perform Risk Assessment
    • Task 3 - Set Test Objectives
    • Task 4 - Quality Control
  • 27. Set Test Objectives Task 1 - Organize Test Team
    • Define Test Responsibilities / Tasks
    • Determine People to Perform Tasks
    Worksheets 4,5
  • 28. Set Test Objectives 2. Perform Risk Assessment
    • Dimensions of Risk
      • Criticality / Mission Impact
      • System Size, Scale and Complexity
      • System Environment and Stability
      • Reliability and Integrity
      • Technology Integration
    Impact of Failure Likelihood of Failure Simple or Complex Scoring Approach
  • 29. Set Test Objectives 3. Set Test Objectives
    • Using:
      • System Objectives
      • Test Strategy
      • Risk Score Analysis
      • Team Assignments
    • Set Objectives
      • Brainstorming
      • Relate Objectives to Function / Output
      • Relate Objectives to Events / System Input
    Prioritize Each- Low Medium High Worksheet 6
  • 30. Set Test Objectives Quality Control
    • Objectives QC Checklist
    • Objectives Incident Report (Change Request)
    … lots of checklists!
  • 31. Develop Test Plan
    • 1. Define Business Processes
    • 2. Define Structural Attributes
    • 3. Define Tests
    • 4. Create Test / Function Matrix
    • 5. Define Admin Requirements
    • 6. Formulate System Test Plan
    • 7. QC System Test Plan
    • 8. Develop Unit Test Plan
    • 9. QC Unit Test Plan
  • 32. Develop Test Plan 1. Define Business Functions
    • Describe what the system should do
      • …stands alone as an activity
      • …is controllable
      • …can be a transition
      • …can be described by a process
  • 33. Business Function Matrix Function Description Evaluation Criteria Worksheet 8
  • 34. Develop Test Plan 2. Define Structural Attributes
    • Reliability
    • Efficiency
    • Integrity
    • Usability
    • Maintainability
    • Testability
    • Flexibility
    • Portability
    • Reusability
    • Interoperability
    Worksheet 9
  • 35. Structural Attribute Matrix Attribute Priority Evaluation Criteria Description
  • 36. Develop Test Plan 3. Define Tests
    • Using
      • Test Objectives Worksheet
      • Business Functions Worksheet
      • Structural Attributes Worksheet
    • Define Tests
      • Associate test factors, techniques and test types
      • Associate evaluation criteria with techniques and types of tests
      • Determine type of test which satisfies each objective
  • 37. Type of Test Techniques
    • Functional
      • Requirements
      • Regression
      • Error Handling
      • Manual Support
      • Intersystems
      • Control
      • Parallel
    • Structural
      • Stress
      • Execution
      • Recovery
      • Operations
      • Compliance (to process)
      • Security
  • 38. Test Data Sources
    • Test Data / Test Desks
    • Production Data
    • Simulation / Prototyping
    • Review / Confirmation
    • Instrumentation / Tracing
    • Test Script
  • 39. Test Definition
    • Each Test is defined in terms of:
      • Controls
      • Inputs
      • Outputs
      • Procedures
    ...At a high level
  • 40. Develop Test Plan 4. Create Function / Test Matrix Tests Function Test 1 Test 2 Ect. Function #1 Function #2 25 17,30 5,7 45
  • 41. Develop Test Plan 5. Define Admin Test Requirements
    • Schedule
    • Budget
    • Resources
    • Testing Materials
    • Test Training
    • Tests to be Conducted
    • Defect Estimation
    Worksheets 13,14
  • 42. Develop Test Plan 6. Formalize Test Plan
    • Section I - General Information
    • Section 2 - Plan
    • Section 3 - Specifications and Evaluation
    • Section 4 - Test Description
    See Test Plan Example
  • 43. Develop Test Plan 7. Test Plan Quality Control
    • Quality Control Checklist
    • Test Plan Incident Report (Change Request)
    … even more checklists!
  • 44. Develop Test Plan 8. Develop Unit Test Plan
    • Using:
      • System Test Plan
      • Program Initiation Document
      • Program Design / Specification Documents
    • Develop
      • Test Plan Administration
      • Subfunction / Test Matrix
      • Interface / Test Matrix
    Worksheets 20,21,22
  • 45. Develop Test Plan 9. Unit Test Plan Quality Control
    • Checklists
  • 46. Step 3 - Execute Tests
    • Task 1 - Select Test Tools
    • Task 2 - Develop Test Cases
    • Task 3 - Execute Tests
    • Task 4 - Quality Control
  • 47. QAI Tool Definition
    • What’s a Test Tool?
      • …any vehicle which assists in performing a test.
      • Examples-
        • Decision Tables
        • Automated capture / playback
        • Checklists
        • Program Analyzer
  • 48. Execute Tests Step 1 - Select Test Tools
    • Using:
      • System Test Plan
      • Unit Test Plan
      • Tool Selection Methodology
  • 49. Execute Tests Step 2 - Develop Test Cases Worksheet 23 Expected Result Test Condition Test Procedures
  • 50. Execute Tests Step 3 - Execute Tests
    • Test Activities Log
    • Regression Testing
      • Full
      • Partial
    Worksheet 24
  • 51. Step 4 - Summarize and Report Test Results
    • Task 1 - Record Defects
    • Task 2 - Perform Data Reduction
    • Task 3 - Develop Findings and Recommendations
    • Task 4 - Finalize Test Report
    • Task 5 - Test Report Quality Control
      • Test Defect Report
      • Test Defect List
      • Test Defect Log
    Worksheets 25,26,27,28
  • 52. Supporting Material
    • Metrics
    • Toolbox Techniques
    • Test Condition Analysis
    • Test Design Analysis
  • 53. Test Metrics Method - Goal - Question - Metric
  • 54. QAI Recommend Test Metrics
    • Defect Reports
    • Requirements Implemented
    • Backlog of Defects (Not Corrected)
    • Status of -
      • Project, Tests, Schedule, Budget
    • Selected Test Metrics
  • 55. QAI Test Metrics Selection Types General Subject 1 Involvement in Testing 4 Extent of Testing 3 Resources Consumed in Testing 20 Effectiveness of Testing 2 Assessment of Testing
  • 56. The QAI Test Toolbox
    • Data Modeling
    • Boundary Value Analysis
    • Capture/Playback
    • Cause-Effect Graphing
    • Change Control Trackers
    • Checklists
    • Checkpoint Review
    • Code Comparison
    • Compiler-based Analysis
    • Complexity-based Analysis
    • Compliance Checkers
    • Control Flow Analysis
    • Correctness Proofs
    • Coverage Based Analysis
    • Data Dictionary
    • Decision Tables
    • Defect Trackers
    • Desk Checking
    • Equivalence Partitioning
    • Error Guessing
  • 57. The QAI Test Toolbox
    • File Comparison
    • Flowcharts
    • Help Desk Trackers
    • Inspections
    • Instrumentation
    • Integrated Test Facility
    • Mapping
    • Model Animation
    • Model Balancing
    • Network Analyzers
    • Performance / Timing Analyzers
    • Population Analysis
    • Risk Matrix
    • Scoring
    • Snapshots
    • Symbolic Execution
    • System Logs
    • Test Case Generator
    • Test Data Generator
    • Test Scripts
  • 58. The QAI Test Toolbox
    • Test Stubs and Harnesses
    • Test Suite Managers
    • Tracing
    • Walkthroughs
  • 59. The QAI Test Toolbox Test Tools Boundary Analysis Types of Functional and Structural Tests Population Analysis Checklists X X X X X
  • 60. Test Condition Analysis
    • Sources -
      • 1. Specification Decomposition
      • 2. Population Analysis
      • 3. Business Process Analysis
      • 4. Structural Analysis
      • 5. Test Transactions
    Test Condition - Expected Result - Test Process
  • 61. Identify Test Conditions
    • 1. System Specifications or Production Environment
    • 2. Population Analysis
      • Files and Tables
      • Screens
      • Field/Data Element
    Worksheet 4 Test Condition - Expected Result - Test Process
  • 62. Identify Test Conditions
    • Field
    • Record
    • File
    • Relationships
    • Error
    • Use (Outputs)
    • Search
    • Match / Merge
    • Stress
    • Control
    • Attributes
    • States
    • Procedures
    • Control Flow
    3. Test Transaction Types Test Condition - Expected Result - Test Process
  • 63. Identify Test Conditions
    • 4. Business Cases
      • …business scenarios…
    • 5. Structural Analysis
      • …decision points...
    Test Condition - Expected Result - Test Process
  • 64. Design Test Cases
    • Design Test Cases
    • Develop Scripts / Sets
    Expand Test Condition s
  • 65. Design Test Cases
    • Expand Conditions into Test Cases using -
      • Boundary Value Analysis
      • Equivalence Partitioning
      • Decision Tables
      • Code Coverage
        • Statement Coverage
        • Condition Coverage
    Expand Test Condition s
  • 66. Design Test Cases Test Scripts Actor/Scene Dialog Detailed Test Script Descriptions Script Reuse … Sam scowls at the report... “ Where did this come from?” Expand Test Condition s
  • 67. QAI Summary - Ten Key Test Concepts
    • Use strategic test questions to set direction
    • Develop Test Objectives
    • Use a cross-functional “test team”
    • Prepare a test plan, then execute it
    • Utilize both verification and validation methods
    • Collect and analyze defects
    • Prepare test management reports
  • 68. QAI Summary - Ten Key Test Concepts
    • Make test efforts reusable
    • Feed the continuous improvement process with information from testing
    • Continuously improve the process
  • 69. Incorporating the Process Models within Models …Make them your own
  • 70. Questions?