This document summarizes Ken Jenks' presentation on NASA's product peer review process. It discusses that product peer reviews are used to discover defects, validate products, and prepare for formal reviews. The presentation provides an overview of NASA's requirements for peer reviews according to various directives and standards. It also describes the different types of peer reviews NASA uses and demonstrates the flow and expectations of a product peer review through a live example, with introductions of team members and a background of the product being reviewed.

1. NASA Product Peer Review
Process
Ken Jenks, Johnson Space Center
2009-02-24
2009-02

2. Inspection/Peer Review at NASA
Inspection/Peer Review at NASA
• Product Peer Reviews are used to discover defects, as a
validation technique, and to prepare for formal reviews.
• Peer reviews/Inspections are part of the program/project
management process and system engineering process
• Product Peer Reviews can be used on many different
products at any phase in a project life cycle.
Peer reviews are often held prior to formal reviews on completed
Peer reviews are often held prior to formal reviews on completed
products.
products.
The results of peer reviews may be addressed at formal document
The results of peer reviews may be addressed at formal document
reviews.
reviews.
August 08

3. Reasons for product peer reviews
Reasons for product peer reviews
• It is difficult for an individual Author or development team
to spot every mistake or flaw in a complicated piece of
work.
• This is not necessarily a reflection on the individuals
concerned, but because with a new and perhaps eclectic
subject, an opportunity for improvement may be more
obvious to someone with special expertise or who simply
looks at it with a fresh eye.
• Showing work to others increases the probability that
weaknesses will be identified and improved.
August 08

4. Different types of peer reviews
Different types of peer reviews
• NASA uses two different types of peer review: scientific
peer review (also known as refereeing) and Product Peer
Review. This presentation describes only Product Peer
Reviews.
• The JSC Engineering Directorate Product Peer Review
process has been used for informal pilot studies, but has
not been baselined yet.
Similar to the Goddard Space Flight Center’s Engineering Peer
Similar to the Goddard Space Flight Center’s Engineering Peer
Review process
Review process
August 08

5. The NASA requirements for peer reviews
The NASA requirements for peer reviews
• Peer reviews/Inspections are part of the program/project
management process and system engineering process
NPR 7120.5D, NASA Space Flight Program and Project
NPR 7120.5D, NASA Space Flight Program and Project
Management Requirements
Management Requirements
NPR 7123.1A NASA Systems Engineering Processes and
NPR 7123.1A NASA Systems Engineering Processes and
Requirements, Appendix G.20
Requirements, Appendix G.20
NASA/SP-2007-6105, NASA Systems Engineering Handbook,
NASA/SP-2007-6105, NASA Systems Engineering Handbook,
Appendix N
Appendix N
NPR 7150.2, NASA Software Engineering Requirements,
NPR 7150.2, NASA Software Engineering Requirements,
[SWE-087], [SWE-088], [SWE-089], [SWE-119]
[SWE-087], [SWE-088], [SWE-089], [SWE-119]
August 08

6. Live Demonstration of a Product Peer Review
Live Demonstration of a Product Peer Review
• The rest of this presentation gives a live demo
• Green slides are presented at the Kick-Off Meeting
• Red slides are presented at the Product Peer Review
Panel Meeting
• Blue slides are part of the NASA PM Challenge
presentation
• Volunteers from the audience will form our panel of
Reviewers
August 08

7. Magic Cookie Caper
Product Peer Review
Kick-Off Meeting
Ken Jenks Moderator
2009-02-24
7

8. Agenda
Purpose of today’s meeting
Purpose of this Product Peer Review
Product Peer Review team introductions
Expectations for the review meeting and process
Product overview
Procedures and tools used in this peer review
Reviewer homework
Defects
Technical risks
Closing the meeting
8

9. Kick-off Meeting Purposes
The meeting today is a Kick-off Meeting
intended to
Introduce members of the review team
Familiarize Reviewers with the product
Train the Reviewers on the Product Peer
Review process
9

10. Purpose of this review
This Product Peer Review is intended to improve the product
Compare the product against its requirements, standards
and specifications, finding defects
Specific suggestions for product improvements
Often used to prepare for formal reviews
Used as a validation technique
Support the evolving design and development of the
product
Provide technical insight needed to ensure product and
process quality
Provide insight into technical risks
The product under review:
Product title, revision level
Stage in the project life cycle
The Author will correct all major defects
Minor and trivial defects are addressed as time permits
At the end of this peer review process, the Moderator will prepare 10
and distribute the Product Peer Review Report

11. Procedure flow
Planning Kick-off Meeting Preparation Review Meeting Rework Follow-Up
Moderator Moderator
Moderator Moderator Author Moderator
Author
Author Author Reviewers Author
Recorder
Reviewers
Reader
Reviewers
Individual Third Hour
Panel Meeting Preparation Logs
Announcement Moderator Meeting Report
and Package
Author
Trivial Individual Reviewers
Defects Log Defect List
Others
Legend
Process Stage
Master Disposition
Optional Stage Record
Person
Stage Transition
Form Based on NASA/SP-2007-6105
Appendix N 11

12. Peer Review Team
Introductions
Moderator
Moderator name, organization, contact info
Author
Author name, organization, contact info
Reviewers
Reviewer 1, organization, expertise
Reviewer 2, organization, expertise
Reader
Reader name, organization
Recorder
Recorder name, organization
12

13. Expectations
You are expected to do your homework
You are expected to find defects
Any complicated technical product will have defects
Finding zero defects is not acceptable
You are expected to record basic metrics (effort,
defect lists)
You are expected to attend the Product Peer
Review Panel Meeting
Charge codes for this peer review effort
This meeting 1 hour, homework 3 hours, next meeting 2
hours
Disclosure of vested interests, conflicts of interest
Author’s Line Management stays out of the room
Helps Reviewers be more objective
13

14. Expectations: Social
Aspects
Social dynamics between the Reviewers and the
Author are a sensitive issue
Reviewers
Not here to show that you’re smarter than the Author
Not here to show that you’re smarter than each other
Thoughtfully select the words you use to raise an issue
Comment about the product and not about the Author
Author
Not here to justify every bit of the work product
Not here to rationalize away problems
Accept the comments graciously, even if you disagree
We’re all here to improve the product
Nobody walks away mad
14

15. Product Background
Product Background
• The Reader presents the background
• These charts are usually supplied by the Author
• The product background includes
Who are the product stakeholders
Who are the product stakeholders
How will the product be used
How will the product be used
How does the product fit into its product architecture
How does the product fit into its product architecture
When is the product needed
When is the product needed
August 08

16. Product Background
“The Magic Cookie Caper” is a movie
coming out next year. Our company is
bidding to produce cookies to be sold
in theaters as a movie tie-in.
This batch of cookies is a prototype.
Our company gives its pitch to the film
distribution company next month.
16

17. Requirements, Standards and
Requirements, Standards and
Specifications
Specifications
• The Reader presents these charts
• These charts are usually supplied by the Author
• Compliance checklists and traceability matrices are
helpful
• It’s common for Reviewers to find problems in the
product’s requirements, standards and specifications
Don’t report those defects as product defects
Don’t report those defects as product defects
Moderator should include a CR form in Data Package
Moderator should include a CR form in Data Package
August 08

18. Requirements, Standards
and Specifications
[MCP-00100] There shall be 12 cookies in each batch.
[MCP-00200] At least 5 cookies shall be sugar cookies.
[MCP-00300] At least 5 cookies shall be chocolate chip.
[MCP-00400] At least 4 cookies shall have pink sprinkles.
[MCP-00500] At least 3 cookies shall have blue sprinkles.
[MCP-00600] The cookies shall be round.
[MCP-00700] The cookies shall be between 2 inches and 3 inches in
diameter.
18

19. Product Information
Product Information
• The Reader presents the product information
• These charts are usually supplied by the Author
August 08

20. Product Information
This batch of cookies was prepared by our product
development team, not a commercial kitchen
We added the Oreos because we found that we can
get a profitable cross-promotion deal with Nabisco
20

21. Product Discussion
Reviewers are invited to ask question
of the Author
About the product
About requirements
About the standards
About the specifications
About technical risks
21

22. Procedures and tools
EA-WI-038 describes the procedure for this
Product Peer Review
Moderator distributes forms to Reviewers
During the Product Peer Review Panel Meeting,
the Recorder will record any additional defects
discovered
Moderator will use EA Action Item Database
for action items
Track actions identified in the reviews until they
are resolved
Moderator will use Microsoft Exchange
“Shared Tasks” for issue tracking
22

23. Reviewer Homework
Inspect the product carefully
Compare the product against its requirements,
standards and specifications
Record major and minor defects that you detect on the
Individual Defect List
Use the Trivial Defects Log for typos, grammatical
changes, spelling errors, formatting problems or other
trivia
Take notes about questions you have or risks that
you’ve identified
Keep track of the effort spent performing these tasks
Fill out the Individual Preparation Log
Send your Individual Defect List and your Individual
Preparation Log to the Moderator by the due date
Bring your notes to the Product Peer Review Panel 23
Meeting

24. Defects
Defect: discrepancy or nonconformity to a
requirement or specification
Reviewers will identify defects in the product
Include assumptions made in determining defects
Severity Description
Major An error that would cause a malfunction or prevents attainment of an
expected or specified result.
Any error that would in the future result in an approved change request or
failure report.
Minor A violation of standards, guidelines, or rules that would not result in a
deviation from requirements if not corrected but could result in minor
difficulties in terms of operations, maintenance, or future
development.
Trivial Editorial errors such as spelling, punctuation, and grammar that do not
cause errors or change requests.
Recorded as redlines or in the electronic Trivial Defects Log. Presented
directly to Author at the end of the meeting.
24

25. Technical Risks
Discussions of defects and approaches
will often reveal technical risks
Author will record technical risks in the
project risk list
Although a Product Peer Review may reveal
risks, risk management is not part of the
Product Peer Review process
25

26. Closing the Meeting
Assignment for Reviewers
Review the product against its
requirements
Location of product and requirements
Individual Preparation Log
Trivial Defects Log
Due date: yyyy/mm/dd
Schedule for the Product Peer Review
Panel meeting
yyyy/mm/dd, hh:mm, location 26

27. And then a week goes by
And then a week goes by
• A week between the Kick-Off Meeting and the Product
Peer Review Panel Meeting gives the Reviewers time to
do their homework and turn in their defect lists to the
Moderator
• The Moderator collects the defects together, combining
duplicates
• The Moderator and the Author can correct some defects
• Normally, the Product Peer Review Panel Meeting should
be held in the same conference room or workstation
cluster a week later
August 08

28. Magic Cookie Caper
Product Peer Review
Panel Meeting
TBD Moderator
yyyy/mm/dd
28

29. Agenda
Purpose of this Product Peer Review
Product Peer Review team
Expectations
Success criteria
Closing the meeting
29

30. Purpose of this review
Product Peer Reviews are used
to discover defects
as a validation technique
to prepare for formal reviews
The product under review:
Product title, revision level
Stage in the project life cycle
30

31. Peer Review Team
Moderator
Moderator name, organization, contact info
Author
Author name, organization, contact info
Reviewers
Reviewer 1, organization, expertise
Reviewer 2, organization, expertise
Reader
Reader name, organization
Recorder
Recorder name, organization
31

32. Expectations
This Product Peer Review is intended to improve
the product
Compare the product against its requirements,
standards and specifications, finding defects
Specific suggestions for product improvements
Gain insight into the technical risks
Record basic metrics (effort, defect lists)
Charge codes for this peer review
Disclosure of vested interests, conflicts of
interest
Author’s Line Management stays out of the room
Helps Reviewers be more objective
32

33. Procedure flow
Planning Kick-off Meeting Preparation Review Meeting Rework Follow-Up
Moderator Moderator
Moderator Moderator Author Moderator
Author
Author Author Reviewers Author
Recorder
Reviewers
Reader
Reviewers
Individual Third Hour
Panel Meeting Preparation Logs
Announcement Moderator Meeting Report
and Package
Author
Trivial Individual Reviewers
Defects Log Defect List
Others
Legend
Process Stage
Master Disposition
Optional Stage Record
Person
Stage Transition
Form Based on NASA/SP-2007-6105
Appendix N 33

34. Expectations: Social Aspects
Social dynamics between the Reviewers and the Author
are a sensitive issue
Reviewers
Not here to show that you’re smarter than the Author
Not here to show that you’re smarter than each other
Thoughtfully select the words you use to raise an issue
Comment about the product and not about the Author
Author
Not here to justify every bit of the work product
Not here to rationalize away problems
Accept the comments graciously, even if you disagree
We’re all here to improve the product
Nobody walks away mad
34

35. Defect List
Defect List
• The Reader presents the major defects
• These charts are usually supplied by the Moderator
August 08

36. Procedures and tools
EA-WI-038 describes the procedure for
this Product Peer Review
Moderator will use EA Action Item
Database for action items
Moderator will use Microsoft Exchange
“Shared Tasks” for issue tracking
36

37. Success criteria
Y N Criteria
Is the product is complete?
Does the product conform to the applicable regulations, standards, guidelines, plans, and procedures?
standards,
Have changes to the product been properly implemented so that they affect only the specified areas?
they
Is the product is suitable for its intended use?
Is the product is ready for the next activity?
Do any defects or discrepancies still exist?
Are the defects and their recommended resolutions clearly identified?
identified?
Has a list of action items been generated to resolve the defects?
defects?
Do the Product Peer Review Panel members agree to the recommended resolutions of identified
recommended
anomalies?
Has the meeting been adequately documented? 37

38. Closing the Meeting
Did the product pass its peer review?
Determining the need for a Third Hour
Schedule for the Product Peer Review Report
38

39. After the meeting
After the meeting
• The Author corrects all Major defects
Author may correct some or all Minor and Trivial defects
Author may correct some or all Minor and Trivial defects
• The Moderator prepares the Product Peer Review Report
From a template
From a template
August 08

40. To obtain the EA Product Peer Review
To obtain the EA Product Peer Review
materials
materials
• Contact Ken.Jenks@nasa.gov
August 08