Testing Types and Paradigms - 2015-07-13 - V11
•Download as PPTX, PDF•
6 likes•3,455 views
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Testing Types and Paradigms - 2015-07-13 - V11
Similar to Testing Types and Paradigms - 2015-07-13 - V11 (20)
Testing Types and Paradigms - 2015-07-13 - V11
- 1. © 2015 Carnegie Mellon University Testing Types and Testing Paradigms Next Gen Testing Conference London England Software Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213 Donald G. Firesmith, Principle Engineer 24 June 2015 (updated 1 July 2015)
- 2. 2 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 Copyright 2015 Carnegie Mellon University This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense. NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. This material has been approved for public release and unlimited distribution except as restricted below. This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu. TBD.
- 3. 3 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 Topics Goals of Presentation What is Testing? Presentation Scope Relevant Testing Challenges Testing Types Testing Paradigms Conclusion
- 4. 4 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 Goals of the Presentation Make it clear that: • There are many different types of testing. • Testing is a complex discipline with its own technical jargon. • There is a lot of overlap between different classes of testing types. – Think multiple classification (object-oriented design) or multiple inheritance (object-oriented programming). Become aware that there multiple testing paradigms. Get you to take a look at your: • Test strategies and ask yourselves “Are they sufficiently complete?” • Testers and other testing stakeholders and ask yourselves “Do they need additional training in testing types and paradigms.
- 5. 5 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 What is Testing? Testing The execution of an Object Under Test (OUT) under specific preconditions with specific stimuli so that its actual behavior can be compared with its expected or required behavior – Preconditions: pretest mode, states, stored data, or external conditions – Stimuli: • Calls, commands, and messages (control flows) • Data inputs (data flows) • Trigger events such as state changes and temporal events – Actual Behavior: • During Test: – Calls, commands, and messages (control flows) – Data outputs (data flows) • Postconditions: post-test mode, states, stored data, or external conditions
- 6. 6 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 Presentation Scope by Object Under Test (OUT) by Reason by Level of Automation by Role by Quality Characteristic by Phase by Built-In-Test (BIT) by Technique by Level of Scripting by Organizational Distribution by Collaboration Test and Evaluation (T&E) Quality Control (QC) Verification and Validation (V&V) Methods Testing Evaluation Walk-Through Inspection Analysis (Static + Dynamic) Review Warantee Reuse Certification Interview Observation Demonstration ← In Scope ← Out Of Scope How How How How Well What When When Why Who Who Where by Physical DistributionWhere
- 7. 7 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 Relevant Testing Challenges Types of Testing: • Many testers are only aware of a minority of types of testing, let alone know how to perform them. • Test managers and developers are aware of even fewer testing types. • The test strategies, project test plans, and test sections of system/software development plans tend to identify a very small number of types of testing (e.g., unit, integration, system, and acceptance testing). – Not planned → Not performed Test Paradigms (overall approaches): • Many testing stakeholders are unaware that there are general testing paradigms. • Many project and testing managers are not aware of the incompatibilities between these different paradigms. • They may verbally back a transition to a new testing paradigm while mandating past practices that are incompatible with the new paradigm. – Paradigm Shift – Paradigm Paralysis
- 8. 8 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 Types of Testing – Overview A type of testing is: • A specific way to perform testing • A class or subclass of testing • Much narrower in scope than a testing paradigm There are relationships between the various types of testing. Most testers know: • A lot about a few types of testing • A little about some additional types of testing • Very little about a sizable number of testing types
- 9. 9 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 Types of Testing – Answering 5W+H Questions How: • by Technique vs. by Level of Automation vs. by Level of Scripting How Well: • by Quality Characteristic What: • by Object Under Test (OUT) When: • by Phase vs. Built-In-Test (BIT) Why: • by Reason Who: • by Role vs. by Collaboration Where: • By Physical Location vs. by Organizational Location
- 10. 10 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 How: by Technique Blackbox Testing Whitebox Testing Experience- Based Testing Graybox Testing Cause and Effect Testing Classification Tree Testing Combinatorial Testing Decision Table Testing Fuzz Testing Requirements Testing Risk-Based Testing Scenario Testing Smoke Testing State Transition Testing Syntax Testing Boundary Value Testing Equivalence Class Testing Control Flow Testing Data Flow Testing Branch Testing Condition Testing Statement Testing All Definitions Testing All Uses Testing Error Guessing Testing Exploratory Testing Procedure Testing Bug Hunt Testing User Interface Navigation Testing Random Testing Testing by Technique Galumphing Patterns- Based Testing Monkey Testing Fuzz Testing Cat-on-the- Keyboard Testing
- 11. 11 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 How: by Level of Automation Testing by Level of Automation Manual Testing Automated Testing - Test Script Generation - Test Data Generation - Test Script Execution - Test Report Generation Data-Driven Testing Action-Keyword Testing Script-Based Testing Record-Playback Testing Model-Based Testing
- 12. 12 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 How: by Level of Scripting Testing by Level of Scripting Unscripted TestingScripted Testing
- 13. 13 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 How Well: by Quality Characteristic Based on the associated quality characteristic and its associated quality attributes: • Uncover related defects • Determine level of quality A/B Testing Capacity Testing Compatibility Testing Consistency Testing Data Migration Testing Functionality Testing Performance Testing Configuration Testing Reliability Testing Robustness Testing Safety Testing Security Testing Usability Testing Load Testing Stress Testing Volume Testing Endurance (Stability) Testing Reliability Mechanism Testing Encryption Testing Testing by Quality Characteristic EMSEC Testing Accessibility Testing Alpha Testing Content Usage Testing Correctness Testing Interoperability Testing Blue Team Testing Red Team Testing Access Control Testing Infrastructure Testing Penetration Testing Scalability Testing Backwards Compatibility Testing Infrastructure Compatibility Testing Anti-Tamper Testing Anti-Spoofing Testing Flexibility Testing Internationalization Testing PersonalizationTesting Beta Testing (Closed/Open) (Flash) Mob Testing
- 14. 14 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 What: by Object Under Test (OUT) Testing by Object Under Test (OUT) Model Testing Requirements Model Testing Design Model Testing Architecture Model Testing Software Testing System Testing System Integration Testing SoS Integration Testing SoS Testing Subsystem Testing System Testing Hardware Testing HW Qualification Testing Software-in-the-Loop (SIL) Testing Hardware-in-the-Loop (HIL) Testing Processor-in-the-Loop (PIL) Testing Tool/Environment Testing SW Unit Component Testing SW Application Testing SW Integration Testing Human-in-the-Loop (HIL) Testing Data Center Testing Configuration Testing Network Traffic Testing Failover and Restore Testing Test Environment Testing Test Tool Testing
- 15. 15 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 System Lifecycle Phases and Environments DT (unit) Development Environment Build Environment Test Environment Test Environment Staging Environment Production Environment(s) Low Fidelity High FidelityHigh Fidelity DT OTAT + OTDT ••• Continuous Testing Continuous Integration ← Shift Left Testing Continuous Delivery / Fielding ← Less Fidelity – More Fidelity → Development OperationAcceptance
- 16. 16 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 When: by Phase Beware of Synonyms! Testing by Phase Developmental Testing (DT) Acceptance Testing (AT) Business Acceptance Testing (BAT) Contract(ual) Acceptance Testing (CAT) Operational Acceptance Testing (OAT) Regulation Acceptance Testing (RAT) Customer Acceptance Testing (CAT) Site Acceptance Testing (SAT) Factory Acceptance Testing (FAT) Production Acceptance Testing (PAT) Operational Testing (OT) Initial Operational Testing (IOT) Follow-on Operational Testing (FOT) Qualification Operational Testing (QOT) Many Types of Developmental Testing Operational Effectiveness Testing Operational Suitability Testing Data Center Testing Configuration Testing Failover and Restore Testing Network Traffic Testing User Acceptance Testing (UAT)
- 17. 17 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 When: by Built-In Testing (BIT) Testing by Built-In Testing (BIT) Execution Periodic Built-In Testing (PBIT) Power-Up Built-In Testing (PupBIT) Interrupt-driven Built-In Testing (IBIT) User-initiated Built-In Testing (UBIT) Shutdown Built-In Testing (SBIT)
- 18. 18 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 Why: by Reason Smoke Testing: Is it good enough to test? Initial Testing: Does it have defects? Retesting: Have defects been fixed? Regression Testing: Have fixes introduced new defects? Error-Seeding: How many defects remain? Testing by Reason Initial Testing Regression TestingRetesting Partial Regression Testing Complete Regression Testing Smoke Testing Error Seeding
- 19. 19 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 Who: by Role Testing by Role Security Testing Safety Testing Developer performs Safety Engineer performs Usability Testing Human Factors Engineer performs User Testing Tester Testing User performs Tester performs Security Engineer Code Testing Programmer performs Architecture Model Testing Architect performs Requirements Model Testing Requirements Engineer performs Developer Testing Design Model Testing performs
- 20. 20 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 Who: by Collaboration Testing by Collaboration Individual Testing Pair TestingBuddy Testing Group Testing
- 21. 21 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 Where: by Organizational Location Outsourced Testing Insourced Testing Testing by Organizational Location
- 22. 22 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 Where: by Physical Location Testing by Physical Location Distributed Testing Cloud Testing Centralized Testing
- 23. 23 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 Testing Paradigms – Overview A Testing Paradigm is: • One of a small number of very general, fundamentally different ways of approaching testing • Based on a cohesive set of general principles • Much broader in scope than individual types of testing Most testers fall into one – or at most two – of these camps and many know little about the others. Trends in development (e.g., Agile, DevOps) often imply corresponding trends in test paradigms.
- 24. 24 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 Spectrum of Testing Paradigms Test paradigms lie along a spectrum from very informal (light-weight) to quite formal (heavy-weight). Test paradigms overlap. Location along spectrum is controversial. Testing paradigms can be like: • Philosophies • Armed camps • Religious sects Testing Paradigms Ad Hoc Testing Best-Practices- Based Testing Agile Testing Process- Driven Testing Standards- Driven Testing Model-Based Testing (MBT) Context- Driven Testing DevOps Testing Risk-Based Testing More Formal (Heavy-Weight) →← More Informal (Light-Weight)
- 25. 25 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 Conclusion Testing Types: • Most testers are not aware of all of the different types of testing. • If you are not aware that it exists, then you don’t know whether you need it. • Most systems require multiple types of testing. Testing Paradigms: • There are 8 major testing paradigms. • They divide into two major categories: – More complete, rigorous, and standardized – More incomplete, informal, and flexible • Many testers follow more than one related paradigms. • Some types of testing are more popular in some paradigms than others.
- 26. 26 Testing Paradigms and Testing Types Donald G. Firesmith, 1 JUly 2015 Contact Information Slide Format Donald G. Firesmith Principal Engineer Software Solutions Division Telephone: +1 412-268-6874 Email: dgf@sei.cmu.edu U.S. Mail Software Engineering Institute Customer Relations 4500 Fifth Avenue Pittsburgh, PA 15213-2612 USA Web www.sei.cmu.edu www.sei.cmu.edu/contact.cfm Customer Relations Email: info@sei.cmu.edu Telephone: +1 412-268-5800 SEI Phone: +1 412-268-5800 SEI Fax: +1 412-268-6257
Editor's Notes
- 7/13/2015
- 7/13/2015
- 7/13/2015
- An OUT can be: A test tool or environment An executable requirements, architecture, or design model A system or subsystem (containing HW, SW, data, personnel, facilities, etc.) A software application, component, or unit Testing requires: Controllability to set up preconditions and provide stimuli Observability to verify preconditions, outputs, and postconditions
- The traditional verification techniques are analysis, demonstration, inspection, and test. The term static testing (e.g., static analysis, reviews) is a misnomer caused by the realization within the testing community that testing by itself is not adequate.
- Smoke Testing : Is it good enough to test? Initial Testing: Does it have defects? Retesting: Have defects been fixed? Regression Testing: Have fixes introduced new defects?
- 7/13/2015