The document summarizes ISO/IEC 20246, which provides guidance on work product reviews. It describes the generic review process, including planning, individual reviews, issue collation, and reporting. It also covers different review techniques like checklist-based and scenario-based reviewing. Finally, it discusses factors to consider when choosing a review approach and provides an overview of the ISO 20246 standard development process and opportunities to get involved.

1. Stuart Reid PhD, FBCS
(stuart@sta.co.kr)
© 2015 Stuart Reid
Making Your Reviews
More Effective
Using the New ISO Standard
(NIPA – 2nd July 2015)

2. • Context - ISO/IEC 20246 & the ISO 29119 standards
• The generic review process
• Individual review techniques
• Review types vs review attributes
• Current status
• Question?
Scope

3. Putting ISO 20246
in Context

4. ISO 29119/33063/20246 – Structure
BS 7925-1
BS 7925-2 IEEE 829
Concepts & Vocabulary
Part 1
Testing
Techniques
Part 4
Documentation
Part 3Part 2
Processes
Keyword-Driven
Testing
Part 5
Process
Assessment
ISO/IEC 33063
Reviews
ISO/IEC 20246
IEEE 1028

5. ISO 29119 Part 2: Testing Processes
TEST MANAGEMENT PROCESSES
ORGANIZATIONAL TEST PROCESS
DYNAMIC TEST PROCESSES

6. TEST MANAGEMENT PROCESSES
STATIC TEST
PROCESSES
ORGANIZATIONAL TEST PROCESS
DYNAMIC TEST
PROCESSES
Testing Processes – the intention
REVIEWS
STATIC
ANALYSIS

7. V&V Taxonomy – ISO/DoD Washington, 2010
Verification &
Validation
Formal
Methods
Model Checking
Proof of
Correctness
Specification-
Based
Structure-Based
Experience-
Based
V&V Analysis
Simulation
Evaluation Quality Metrics
Model
Verification
Static Analysis
Reviews
Static Testing
Dynamic
Testing
Testing

8. ISO/IEC/IEEE 29119
TEST MANAGEMENT
PROCESSES
ORGANIZATIONAL TEST
PROCESS
DYNAMIC TEST
PROCESSES
Testing Processes – the end result
ISO/IEC/IEEE 20246
REVIEW PROCESS

9. ISO/IEC/IEEE 29119
TEST
MANAGEMENT
PROCESSES
ORGANIZATIONAL
TEST PROCESS
DYNAMIC TEST
PROCESSES
Testing Processes – the next step?
ISO/IEC/IEEE 20246
REVIEW PROCESS
ISO/IEC/IEEE ?????
STATIC ANALYSIS

10. ORGANIZATIONAL TEST PROCESS
TEST MANAGEMENT PROCESSES
TEST
PLANNING
TEST
MONITORING
& CONTROL
TEST
COMPLETION
ORGANIZATIONAL
TEST
DOCUMENTATION
FEEDBACK ON
ORGANIZATIONAL TEST
DOCUMENTATION
TEST PLAN UPDATES
TEST
PLAN
TEST
COMPLETION
REPORT
DYNAMIC TEST
PROCESSES
TEST
MANAGEMENT
PROCESSES
TEST PLAN,
TEST COMPLETION
REPORT,
TEST MEASURES
TEST
MEASURES
TEST PLAN,
CONTROL
DIRECTIVES
TEST PLAN,
CONTROL
DIRECTIVES
Overall Test Management Processes
WORK PRODUCT
REVIEW
PROCESS
REVIEW
MEASURES
TEST PLAN,
CONTROL
DIRECTIVES

11. Organise
Test Plan
Development
Identify &
Estimate Risks
Design Test
Strategy
Determine
Staffing and
Scheduling
Document
Test Plan
Schedule, Staffing
Profile
Test
Strategy
Estimated
Risks
Scope
Identify Risk
Mitigation
Approaches
Gain
Consensus on
Test Plan
Approved
Test Plan
Draft
Test Plan
Test
Plan Publish
Test Plan
Understand
Context
Treatment
Approaches
ISO 29119 Part 2 - Test Planning Process
Understand
Context
MUST INCLUDE
REVIEWS

12. ISO/IEC 20246
Work Product
Reviews

13. • Anything can be reviewed
– in parts or the whole thing
• For example:
– Contracts
– Requirements Specification
– Design Specification
– Code
– Test Specification
– User Manual
– User Procedures
– Project Plan
– Test Plan
– Development Plan
Why ‘Work Product’ Reviews?

14. • What do you think?
• Find defects
• Measure quality
• Educate reviewers
• Gain consensus
• Generate new ideas
• Motivate authors to improve their practices
Software Reviews – Main Purpose

15. Generic
Review Process

16. Work Product Reviews - Generic Process
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation &
Analysis
Fixing &
Reporting

17. • Identify review scope
– what is being reviewed
– the purpose of the review (there may be more than one)
– relevant supporting information, such as standards
– the timeframes for the review
• Identify review characteristics
– review activities
– individual review techniques
– checklists
• Distribute review materials (as early as possible)
– product to be reviewed
– baseline specifications
– checklists
Planning & Preparation
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation &
Analysis
Fixing &
Reporting

18. • Choose appropriate reviewers
– agreed with Project Manager
– assign specific roles
• Agree with each reviewer
– availability (and a substitute, just in case)
– the review role and focus
Planning & Preparation – Reviewers
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation &
Analysis
Fixing &
Reporting

19. • Optional
• Review Leader
– presents the scope of the review to the review
participants
– presents related documents to reviewers
– explains the role, responsibilities and focus of each review
participant
– details the timeline for the review
• Reviewers (and author)
– formally commit to the review
Overview Meeting
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation &
Analysis
Fixing &
Reporting

20. • Each reviewer identifies issues with the work product
• Identified issues shall be communicated to the review
leader / author
Individual Review
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation &
Analysis
Fixing &
Reporting

21. • The author collates the Issues identified by the reviewers
• Collated issues are analysed and a decision made on what
to do with each issue
• Issue Review Status
– ‘rejected’
– ‘issue to be noted but no action’
– ‘issue to be addressed’
– ‘issue updated due to analysis’
– etc.
• Issues are then assigned based on their review status.
– to authors
– to other responsible individuals (e.g. to update earlier
specifications, organization standards)
Issue Collation & Analysis – 1 (of 2)
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation &
Analysis
Fixing &
Reporting

22. • Finally the review decision is made on the reviewed
product:
– used as is
– updated based on the identified issues and used
– reworked and re-reviewed
– discarded
• The BIG Question – is this analysis and decision-making
done by the author or done by a group of reviewers as
part of a Review Meeting???
Issue Collation & Analysis – 2 (of 2)
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation &
Analysis
Fixing &
Reporting

23. • The author
– addresses the outstanding issues
– organizes for others to fix their outstanding issues in related
documents
• in ‘earlier’ specifications
• in organizational standards
– checks that all issues have been addressed
– either:
• publishes the updated work product
• submits the updated work product for re-review
• The reviewers
– sign off on the final version (confirming that issues addressed)
• A Review Report is generated
Fixing & Reporting
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation &
Analysis
Fixing &
Reporting

24. Review Timelines
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation &
Analysis
Fixing &
Reporting
3 days
5-10 days
3 days
5-10 days
2 days
They vary!!
But need to be set and agreed.

25. Generic Review - Roles
Reviewer
Author
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation &
Analysis
Fixing &
Reporting
Review Leader
Reader Recorder/Scribe
Moderator
QA
SME
Customer
Management
Technical Lead

26. Individual Reviews
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation &
Analysis
Fixing &
Reporting

27. • Ad Hoc Reviewing
• Checklist-Based Reviewing
• Scenario-Based Reviewing
• Perspective-Based Reading (PBR)
Individual Review Techniques

28. • This is the most common approach (completely
unstructured)
• Each reviewer is expected to find as many defects as
possible of any type
• Little or no guidance on how to review is provided
• This approach is highly-dependent on reviewer skills
• Often the same issues are identified by different
reviewers
Ad Hoc Individual Review
“60% of reviewers don’t
prepare at all”

29. Individual Reviews -
Checklist-based
Reviewing
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation &
Analysis
Fixing &
Reporting

30. • A systematic approach to identifying defects
• Typically review checklists take the form of a set of
questions based on potential defects (and risk)
• Assign different reviewers to different checklists to
– get wider coverage
– prevent the duplication inherent in the ad hoc approach
• There is a danger that some reviewers limit themselves to
only considering the checklist entries
• Will normally take longer than using an ad hoc approach
– often multiple passes are needed to cover all questions
(especially if checklist is long)
– Requires a systematic approach
Checklist-based Review
“50% OF REVIEWERS
USE CHECKLISTS”

31. • Checklist defects are derived from experience
– within the project
– within the organization
– across the industry as a whole
• Checklists should be specific to
– the product under review (be wary of generic questions)
– the methodology used to develop the work product (e.g. there may
be different checklist questions for requirements in the form of plain
text to those in the form of use cases or user stories)
– the application domain of the work product (e. g. a checklist for a
banking work product may be based on banking regulations while a
checklist for an avionics work product would be based on avionics
standards)
• Checklists should be based on risk
– to ensure that the most important (and the most frequently-
occurring) risks are reviewed in more depth
Creating Checklists

32. • Many checklists are too long and never change
• The ideal checklist should be constrained to about 10
entries and regularly updated
– as entries become stale and find fewer issues (hopefully
because the authors have learned and improved)
– new entries should reflect issues missed in the recent past
Managing Checklists

33. Individual Reviews –
Scenario-based
Reviewing
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation &
Analysis
Fixing &
Reporting

34. • Used where multiple scenarios can be identified for different
users
• Reviewers perform ‘dry runs’ on the work product to check
functionality and handling of error conditions
– following scenarios rather than reading the document from top to
bottom
• Assign different reviewers to different scenarios to:
– get wider coverage
– prevent the duplication inherent in the ad hoc approach
• There is a danger that some reviewers limit themselves to only
considering the provided scenarios
– and will miss defects of omission, where required functionality is not
included in the work product under review
• Research indicates that scenario-based reviews
– can be more cost effective than checklist-based reviews
– but they do take longer
Scenario-based Reviewing

35. • Where requirements, designs (or tests) are documented in a scenario-
type format (e.g. use cases) then the existing specifications are a good
starting point
– the review can then use the modelled scenarios to drive the review
• However, scenarios typically also need to be created (we need
multiple scenarios to cover everything):
– overview scenario
– most common scenarios
– alternative scenarios
– error condition scenarios
– growth scenarios (e.g. for an architecture review)
– input handling scenarios
– output generating scenario
– enquiry scenarios (getting specific results)
• Scenarios should be based on risk
– to ensure that the most important (and the most frequently-used)
scenarios are reviewed in more depth
Identifying Scenarios

36. Individual Reviews -
Perspective-based
Reading
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation &
Analysis
Fixing &
Reporting

37. Perspectives and Roles
PRODUCT
UNDER REVIEW
View 2
View 5
View 7

38. Perspective-based Reading (PBR)
Identify
Stakeholders
Create/Identify
PBR Scenarios
Perform ReviewPerform ReviewPerform ReviewPerform Review
• Understand Stakeholder Perspective
• Attempt to Create a Work Product
• Use Checklist on the Work Product
Describes the stakeholder role
the reviewer is taking for this
review – and their interest in
the work product under review
Describes the high
level work product
that the stakeholder
would be expected to
develop
A checklist of questions
specific to the high-level
work product developed

39. Choosing your Review

40. • Milestone Reviews
• Inspections
• Technical Reviews
• Peer Reviews
• Walkthroughs
• Informal Reviews
• Author Check
• Buddy Check
• Pair Review
• Peer Deskcheck
Work Product Reviews – Types or Bespoke
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation &
Analysis
Fixing &
Reporting
Chosen
Attributes
OR

41. • Purpose
• Roles
• Individual review techniques
• Optional activities
• Number of reviewers
• Planned number of reviews
• Work product type
• Work product format
• Formal reporting
• Training required
• Review improvement
• Entry and exit criteria
• Geographic distribution of reviewers
Work Product Reviews - Attributes

42. • Factors
– purpose
– product type
– product format
– risks
– reviewer availability
– reviewer skills (and availability of training)
– life cycle model
– budget (time and cost)
Choosing your Review

43. • Foreword
• Introduction
• 1 Scope
• 2 Conformance
• 3 Normative References
• 4 Terms and Definitions
• 5 Work Product Reviews - Introduction
• 6 Software Review Process
• 7 Review Techniques
• Annex A (informative) Review Documentation
• Annex B (informative) Review Roles
• Annex C (informative) Review Attributes
• Annex D (informative) Review Types (and attribute mapping)
• Annex E (informative) Mapping to IEEE 1028
• Annex F (informative) Reviews - Life Cycle Mapping
• Annex G (informative) Review Measurement & Improvement
• Annex H (informative) Tool Support
• Annex I (informative) Bibliography
ISO/IEC 20246 - Work Product Reviews -
Contents

44. ISO/IEC 20246 Development
Working Draft (WD)
Committee Draft (CD)
Draft International Standard (DIS)
Final Draft International Standard (FDIS)
Final International Standard (FIS)
May
2014
May
2015
May
2016
WD1
DIS
FDIS
WD2
DIS2

45. ISO 29119/30363/20246 – Current Status
Testing Concepts & Vocabulary
29119 Part 1
Testing
Techniques
29119 Part 4
Test
Documentation
29119 Part 329119 Part 2
Keyword-Driven
Testing
29119 Part 5
Process
Assessment
ISO/IEC 33063
AUG ‘13
AUG ‘13AUG ‘13@ FDIS
@ DIS2 @ FDIS
Work Product Reviews
ISO/IEC 20246
 DIS
Test
Processes

46. • Context - ISO/IEC 20246 & the ISO 29119 standards
• The generic review process
• Individual review techniques
• Review types vs review attributes
• Current status
• Question?
Summary

47. Thank you for listening
Any Questions?

48. • Join ISO Working Group 26
– representing your national standards body
– 6 day meetings, every 6 months
– contribute between meetings
• Join a WG26 mirror group
– for your national standards body
• Contribute materials
• Review drafts
• Trial the standards on real projects
Do you want to be involved?

49. • stuart@sta.co.kr
– if you have any questions on the standards
– if you are interested in trialling the standard on a project,
reviewing drafts or writing examples
• http://softwaretestingstandard.org/
– WG26 website
• http://www.jtc1-sc7.org/
– access to official documents released by WG 26
Finally…