Recommended
More Related Content
Similar to Introducing the New Software Testing Standard
Similar to Introducing the New Software Testing Standard (20)
More from TechWell
More from TechWell (20)
Recently uploaded
Recently uploaded (20)
Introducing the New Software Testing Standard
- 1. T24 Concurrent Class 10/3/2013 3:00:00 PM "Introducing the New Software Testing Standard" Presented by: Jon Hagar Consultant Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
- 2. Jon Hagar Grand Software Testing Jon Hagar is a systems-software engineer and tester consultant supporting software product integrity and verification and validation, with a specialization in embedded and mobile software systems. For more than thirty years Jon has worked in software engineering, particularly testing, supporting projects including control system (avionics and auto), spacecraft, mobile-smart devices, IT, and attack testing of smart phones.
- 3. 9/19/2013 Special Topics T24 Introducing the New Software Testing Standard: ISO/IEEE/IEC 29119 1 Jon Hagar embedded@ecentral.com Grand Software Testing (GST), LLC 303-903-5536 Copyright 2013 Jon Hagar Grand Software Testing GST Scope Purpose of standards Types of standards Development of standards Overview of ISO29119 Applicability Recent developments Timeline Future work Copyright 2013 Jon Hagar Grand Software Testing GST 1
- 4. 9/19/2013 Some Preliminaries Under disclosure, I am the IEEE Project Editor for ISO/IEEE/IEC 29119 Software Testing Standard Before you decide to roast me . . .please recognize Standards are a fact of life for many industry professionals not just testers Standards form part of the business for many companies , customers, and countries Many of us end up in a mixed relationship with standards Standards can be a basis for research and change Industry has not had a comprehensive software testing standard Copyright 2013 Jon Hagar Grand Software Testing GST What are standards? “Guideline documentation that reflects agreements on products, practices, or operations by nationally or internationally recognized industrial, professional, trade associations or governmental bodies” - ISO Guideline documents—not compulsory unless mandated Agreements because they often reflect a certain level of consensus within a community Copyright 2013 Jon Hagar Grand Software Testing GST 2
- 5. 9/19/2013 How can a standard “help?” Standards describe a current ‘body of knowledge’ that provides the basis for a professional discipline Standards are a basis for: Communication – common terminology Professionalism and criticism Conformance/compliance schemes Benchmark of ‘industry practice’ (not best, may not be good for all, etc.) Contracts and regulations Interoperability and consistency. . . Copyright 2013 Jon Hagar Grand Software Testing GST ISO29119 Standard - My Viewpoint Any software standard is based on limited history (rear looking) and developed based on consensus of many voices Yet still may represent only some of viewpoints of industry But does not represent all states of the art Standards need to be inclusive and ethical As with many “schools,” views and opinions but with NO “best” Not claimed to be perfect Most projects/professionals will use “tailoring” Copyright 2013 Jon Hagar Grand Software Testing GST 3
- 6. 9/19/2013 Standardization Bodies Copyright 2013 Jon Hagar Grand Software Testing GST Motivation for ISO29119 Conflicts in definitions, processes and procedures “One ring to rule them all” — standards to be replaced by one e.g., IEEE 829, IEEE 1008, BS7925-1/-2, IEEE 1028 Users do not know which standard to follow Lacking in current standards or incomplete: Organizational areas e.g., Test Policy and Organizational Test Strategy Project Test Management BS7925 only covers unit testing General processes Common functional techniques missing Coverage of non-functional testing Copyright 2013 Jon Hagar Grand Software Testing GST 4
- 7. 9/19/2013 ISO/IEC29119 –Structure and Flow Part 1 ISO 12207 ISO 15288 Directives Concepts & Vocabulary BS7925-1 Part 4 Part 2 Part 3 Testing Techniques Processes Documentation IEEE 1008 BS7925-2 IEEE 829 Part 5 Copyright 2013 ISO/IEC 33063 Keyword-Driven Testing Process Assessment Jon Hagar Grand Software Testing GST A Brief Diversion– Schools of Testing Initially proposed by Pettichord (workshop notes) et. al. Labels used here with some liberty (to be positive) Context driven is one, but there are others, possibly Agile (the rage for the last 10 years) Academic (researchers) Quality assurance (whatever that is) Process based “None” (not an official name, but I see these testers all the time) Some “schools” may not recognize themselves (by the names) Schools are common in art, music, sciences, and other professions But many “artists” cross the school lines of art ISO29119 aims to be “school independent”, but will have a process centric view but with liberal ability to tailor Copyright 2013 Jon Hagar Grand Software Testing GST 5
- 8. 9/19/2013 Do Testers Need Standards? – Yes, maybe, but Standards support common communication within the topic Common reference points Starting point for research, usage (pro & con), critic Maturity is an issue but a baseline serves as sounding board and common reference point for “Scientific” methods An international benchmark Thinkers and researchers can prove/disprove benchmark(s) Part of being in a profession (but only part) Standards have dangers Copyright 2013 Jon Hagar Grand Software Testing GST My View – Potential Dangers Not every book, school, paper, ideal will be represented There must be a vetting, upgrade, and maintenance effort Will not make everyone happy, so in most cases must be tailored by anyone using them Will be subject to “misuse” by various groups Unthinking audits/reviews May stop some people from thinking for themselves (but many do that anyway) Will (always) be imperfect Copyright 2013 Jon Hagar Grand Software Testing GST 6
- 9. 9/19/2013 Part 1: Concepts & Vocabulary DEFINITIONS SOFTWARE TESTING CONCEPTS TESTING IN DIFFERENT LIFE CYCLE MODELS ROLES AND RESPONSIBILITIES IN TESTING Test: Approach, Basis, Methods - Risk Based Testing Scope, Conformance, Normative References ANNEXES – Metrics, Examples, Bibliography Copyright 2013 Jon Hagar Grand Software Testing GST Part 2: Testing Processes ORGANIZATIONAL TEST PROCESS TEST MANAGEMENT PROCESSES DYNAMIC TEST PROCESSES Copyright 2013 Jon Hagar Grand Software Testing GST 7
- 10. 9/19/2013 Instantiating Testing Processes ORGANIZATIONAL TEST PROCESS TEST MANAGEMENT PROCESSES DYNAMIC TEST PROCESSES Ref: S. Reid Copyright 2013 Jon Hagar GST Grand Software Testing Organizational Test Processes Develop Organizational Test Specification (OT1) Organizational Test Specification Monitor and Control Use of Organizational Test Specification (OT2) Controlled Organizational Test Specification Update Organizational Test Specification (OT3) Updated Organizational Test Specification Copyright 2013 Jon Hagar Grand Software Testing GST 8
- 11. 9/19/2013 Test Management Processes ORGANIZATIONAL TEST PROCESS ORGANIZATIONAL TEST DOCUMENTATION FEEDBACK ON ORGANIZATIONAL TEST DOCUMENTATION TEST MANAGEMENT PROCESSES TEST PLAN UPDATES TEST PLAN TEST PLANNING TEST PLAN, CONTROL DIRECTIVES TEST MONITORING & CONTROL TEST COMPLETION TEST PLAN, TEST COMPLETION REPORT, TEST MEASURES TEST PLAN, CONTROL DIRECTIVES TEST MANAGEMENT PROCESSES Copyright 2013 TEST COMPLETION REPORT TEST MEASURES DYNAMIC TEST PROCESSES Jon Hagar Grand Software Testing GST Test Planning Processes Understand Context Scope Organize Test Plan Development Identify & Estimate Risks Analyzed Risks Identify Risk Treatment Approaches Schedule, Staffing Profile Draft Test Plan Approved Test Plan Test Plan Treatment Approaches Design Test Strategy Determine Staffing and Scheduling Test Strategy Document Test Plan Gain Consensus on Test Plan Publish Test Plan Copyright 2013 Jon Hagar Grand Software Testing GST 9
- 12. 9/19/2013 Standard Primarily has a Risk-based Process Flavor Risk based testing is a primarily “good” practice Other bases/approaches (Agile, exploratory, model-based, mathbased, etc.) are included All parts may need “tailoring” (few efforts would be “full” conformance) Use of the standard implies skills, thinking, dependence on context (“it depends”), etc. Copyright 2013 Jon Hagar Grand Software Testing GST Test Monitoring & Control Process test status report test progress info test plan Report test control info [testing incomplete] [testing complete] test measures Set-Up Monitor test progress info measures Control control directives ...TEST PROCESSES... Dynamic/Management Copyright 2013 Jon Hagar Grand Software Testing GST 10
- 13. 9/19/2013 Dynamic Test Processes (LEVEL) TEST MANAGEMENT PROCESS (LEVEL) TEST PLAN CONTROL DIRECTIVES TEST MEASURES DYNAMIC TEST PROCESSES TEST DESIGN & IMPLEMENTATION TEST SPECIFICATION TEST EXECUTION TEST RESULTS [NO ISSUES NOTICED] [ISSUE NOTICED or RETEST RESULT] TEST ENVIRONMENT REQUIREMENTS Copyright 2013 TEST ENVIRONMENT SET-UP Jon Hagar TEST ENVIRONMENT READINESS REPORT TEST INCIDENT REPORTING Grand Software Testing INCIDENT REPORT GST Part 3 – Test Documentation Select a subset of docs Scope, Conformance, Normative References TEST DOCUMENTATION ANNEXES - EXAMPLES Copyright 2013 Jon Hagar Grand Software Testing GST 11
- 14. 9/19/2013 Part 3: Test Documentation Organizational test documentation Test policy Test strategy Project test documentation Project test plan Test project completion report Test Level documentation Test plan Test specification Test results Anomaly reports Level test status report Test environment report Test level completion report Appendices Examples of documents at each level of testing Copyright 2013 Jon Hagar Grand Software Testing GST Part 4 – Test Techniques Scope, Conformance, Normative References TEST DESIGN TECHNIQUES Functional Structural TEST COVERAGE MEASUREMENT ANNEXE – TESTING OF QUALITY CHARACTERISTICS ANNEXE – SELECTION OF TECHNIQUES ANNEXE – TEST TECHNIQUE EFFECTIVENESS Copyright 2013 Jon Hagar Grand Software Testing GST 12
- 15. 9/19/2013 Part 5- Keyword-Driven Testing Part 5 will address: Concept Applicability Interfaces Approach Part 5 WD was sent out in May and next draft due in Nov 2013 Copyright 2013 Jon Hagar Grand Software Testing GST Applicability of ISO/IEEE 29119 It is intended to be a generic standard to support all domains Some domains are more standards oriented such as: safety-related telecoms Some domains will become more regulated such as: financial – banks, stock markets, etc. Needs trails in all domains and project types Applicability to sequential, iterative and agile Applicability to new as well as maintenance projects Applicability to IT and embedded Copyright 2013 Jon Hagar Grand Software Testing GST 13
- 16. 9/19/2013 Discussion Point: Thoughts? Do you think we need standards? If we use a standard does this mean we do not think? What should the test community do? Copyright 2013 Jon Hagar Grand Software Testing GST My Thoughts Thinking testers with a context-driven/based viewpoint Use standards when a customer or project requires them e.g., in regulated environments Understand where standards are “good” (worthwhile) and bad (not worthwhile) Use standards as a professional reference work, but know when and how to break the “rules” via tailoring How should the test community react? Engage to identify and address both the good and bad Do not use standards in all situations (test projects) Use materials for constructive criticism They will have many concepts which need to be “over turned” Drive the scientific review and healthy debated Copyright 2013 Jon Hagar Grand Software Testing GST 14
- 17. 9/19/2013 Timelines – Current Status Parts 1, 2, 3 Working Draft (WD) Committee Draft (CD) Draft International Standard (DIS) Final Draft International Standard (FDIS) Final International Standard (FIS) Part 4 Part 5 WD FIS WD CD1 CD1 WD … May 2010 Copyright 2013 CD2 CD3 FIS CD2 CD3 FDIS DIS May 2012 May 2011 Jon Hagar DIS FDIS Grand Software Testing 4 May 2013 GST For more information jon.d.hagar@gmail.com For more information and details http://softwaretestingstandard.org/ WG26 website http://www.jtc1-sc7.org/ Access to official documents released by WG 26 Copyright 2013 Jon Hagar Grand Software Testing GST 15
- 18. 9/19/2013 Summary First attempt at a comprehensive software test standard is being released Even in the presence of standards, a thinking tester is always needed What will you do? Copyright 2013 Jon Hagar Grand Software Testing GST Major Reused Reference Point: Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500 Fax: 0207 623 8459 www.testing-solutions.com Copyright 2013 Jon Hagar Grand Software Testing GST 16