Slide title: QPI/020/11-10 Slide Number: 1
SM003Rev1
QPI Unit http://www.zenkara.com
Secrets to effective peer reviews
Jam...
Slide title: QPI/020/11-10 Slide Number: 2
SM003Rev1
QPI Unit http://www.zenkara.com
Overview Part 1
• This presentation d...
Slide title: QPI/020/11-10 Slide Number: 3
SM003Rev1
QPI Unit http://www.zenkara.com
Overview Part 2
• Deployments of revi...
Slide title: QPI/020/11-10 Slide Number: 4
SM003Rev1
QPI Unit http://www.zenkara.com
Scope
The study involved software int...
Slide title: QPI/020/11-10 Slide Number: 5
SM003Rev1
QPI Unit http://www.zenkara.com
General Situations at T0
Some documen...
Slide title: QPI/020/11-10 Slide Number: 6
SM003Rev1
QPI Unit http://www.zenkara.com
Typical Implementation Strategy
Big b...
Slide title: QPI/020/11-10 Slide Number: 7
SM003Rev1
QPI Unit http://www.zenkara.com
Typical Implementation Motivation
Cri...
Slide title: QPI/020/11-10 Slide Number: 8
SM003Rev1
QPI Unit http://www.zenkara.com
Typical Implementation Method
1. Iden...
Slide title: QPI/020/11-10 Slide Number: 9
SM003Rev1
QPI Unit http://www.zenkara.com
Typical Review process
Ready for revi...
Slide title: QPI/020/11-10 Slide Number: 10
SM003Rev1
QPI Unit http://www.zenkara.com
Review process - Types of review
1. ...
Slide title: QPI/020/11-10 Slide Number: 11
SM003Rev1
QPI Unit http://www.zenkara.com
Review Process - Roles
Moderator - a...
Slide title: QPI/020/11-10 Slide Number: 12
SM003Rev1
QPI Unit http://www.zenkara.com
Forms/Workflow
Call For Review
•basi...
Slide title: QPI/020/11-10 Slide Number: 13
SM003Rev1
QPI Unit http://www.zenkara.com
Issues with Forms/Workflow
Forms wer...
Slide title: QPI/020/11-10 Slide Number: 14
SM003Rev1
QPI Unit http://www.zenkara.com
Use of Checklists
Seen as useful by ...
Slide title: QPI/020/11-10 Slide Number: 15
SM003Rev1
QPI Unit http://www.zenkara.com
Use of Checklists - Improvement
Spec...
Slide title: QPI/020/11-10 Slide Number: 16
SM003Rev1
QPI Unit http://www.zenkara.com
Role of the S/EPG or SQAG
A distingu...
Slide title: QPI/020/11-10 Slide Number: 17
SM003Rev1
QPI Unit http://www.zenkara.com
Management Commitment
Managers provi...
Slide title: QPI/020/11-10 Slide Number: 18
SM003Rev1
QPI Unit http://www.zenkara.com
Management Commitment
Another form o...
Slide title: QPI/020/11-10 Slide Number: 19
SM003Rev1
QPI Unit http://www.zenkara.com
Use of metrics
Metrics were used to ...
Slide title: QPI/020/11-10 Slide Number: 20
SM003Rev1
QPI Unit http://www.zenkara.com
Graphs
A large number of graphs were...
Slide title: QPI/020/11-10 Slide Number: 21
SM003Rev1
QPI Unit http://www.zenkara.com
Graphs - Initial
Num of reviews per ...
Slide title: QPI/020/11-10 Slide Number: 22
SM003Rev1
QPI Unit http://www.zenkara.com
Graphs - Initial
Defects identifed i...
Slide title: QPI/020/11-10 Slide Number: 23
SM003Rev1
QPI Unit http://www.zenkara.com
Analysis
• Initial over-focus in cer...
Slide title: QPI/020/11-10 Slide Number: 24
SM003Rev1
QPI Unit http://www.zenkara.com
Graphs - Subsequent
Subsequent use w...
Slide title: QPI/020/11-10 Slide Number: 25
SM003Rev1
QPI Unit http://www.zenkara.com
Graphs - Impressions Management
Init...
Slide title: QPI/020/11-10 Slide Number: 26
SM003Rev1
QPI Unit http://www.zenkara.com
Graphs – Impressions Staff
General c...
Slide title: QPI/020/11-10 Slide Number: 27
SM003Rev1
QPI Unit http://www.zenkara.com
Initial analysis - Findings
A lot of...
Slide title: QPI/020/11-10 Slide Number: 28
SM003Rev1
QPI Unit http://www.zenkara.com
Subsequent analysis of review proces...
Slide title: QPI/020/11-10 Slide Number: 29
SM003Rev1
QPI Unit http://www.zenkara.com
Training
Training was generally cond...
Slide title: QPI/020/11-10 Slide Number: 30
SM003Rev1
QPI Unit http://www.zenkara.com
Followup and refresher courses
Just ...
Slide title: QPI/020/11-10 Slide Number: 31
SM003Rev1
QPI Unit http://www.zenkara.com
Benefits of the reviews
ROI of betwe...
Slide title: QPI/020/11-10 Slide Number: 32
SM003Rev1
QPI Unit http://www.zenkara.com
Benefits of the reviews
Improved qua...
Slide title: QPI/020/11-10 Slide Number: 33
SM003Rev1
QPI Unit http://www.zenkara.com
Side effects of the reviews
Particip...
Slide title: QPI/020/11-10 Slide Number: 34
SM003Rev1
QPI Unit http://www.zenkara.com
Keeping the ball rolling
Set coverag...
Slide title: QPI/020/11-10 Slide Number: 35
SM003Rev1
QPI Unit http://www.zenkara.com
Keeping the ball rolling
Ensure some...
Slide title: QPI/020/11-10 Slide Number: 36
SM003Rev1
QPI Unit http://www.zenkara.com
Strategy and method
Begin with the e...
Slide title: QPI/020/11-10 Slide Number: 37
SM003Rev1
QPI Unit http://www.zenkara.com
Outsourcing and Cultural Issues
Need...
Slide title: QPI/020/11-10 Slide Number: 38
SM003Rev1
QPI Unit http://www.zenkara.com
Metrics
Start collecting metrics as ...
Slide title: QPI/020/11-10 Slide Number: 39
SM003Rev1
QPI Unit http://www.zenkara.com
About – James Kelly
James Kelly is C...
Slide title: QPI/020/11-10 Slide Number: 40
SM003Rev1
QPI Unit http://www.zenkara.com
Who are we?
Zenkara provides advance...
Upcoming SlideShare
Loading in …5
×

Secrets to effective peer reviews

639 views

Published on

Describes a three year long initiative which studied the implementation of walkthroughs, peer reviews and inspections. It gives guidance on how to implement and run effective reviews.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
639
On SlideShare
0
From Embeds
0
Number of Embeds
14
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Secrets to effective peer reviews

  1. 1. Slide title: QPI/020/11-10 Slide Number: 1 SM003Rev1 QPI Unit http://www.zenkara.com Secrets to effective peer reviews James Kelly CEO, Zenkara
  2. 2. Slide title: QPI/020/11-10 Slide Number: 2 SM003Rev1 QPI Unit http://www.zenkara.com Overview Part 1 • This presentation describes a three year long initiative which studied the implementation of walkthroughs, reviews and inspections. • Strategies for effective implementation and running of peer reviews is given together with analysis and lessons learned. • Obstacles arose out of the implementation. The primary challenges are described together with what was done to address them.
  3. 3. Slide title: QPI/020/11-10 Slide Number: 3 SM003Rev1 QPI Unit http://www.zenkara.com Overview Part 2 • Deployments of reviews are described and analysed in detail – what worked, what didn’t, and what was redone. • And finally some lessons learned are presented together with a discussion on formalising the detection & correction of defects with metrics embedded in the process.
  4. 4. Slide title: QPI/020/11-10 Slide Number: 4 SM003Rev1 QPI Unit http://www.zenkara.com Scope The study involved software intensive organisations providing industry-specific software both within Australia and internationally. The reviews were implemented across the entire system project life cycle and across projects of between 5 and within a department of 50 staff.
  5. 5. Slide title: QPI/020/11-10 Slide Number: 5 SM003Rev1 QPI Unit http://www.zenkara.com General Situations at T0 Some documents were reviewed informally. Little metrics data was collected, other than basic staff costs per team. Few staff had formal review training (other than at university). Few staff had some experience with formal reviews.
  6. 6. Slide title: QPI/020/11-10 Slide Number: 6 SM003Rev1 QPI Unit http://www.zenkara.com Typical Implementation Strategy Big bang approach Incremental scope “Hot Spots” Priority: project or team Incremental formality Walkthroughs, Reviews, then inspections Decision on strategy usually made by Dev/Eng Managers
  7. 7. Slide title: QPI/020/11-10 Slide Number: 7 SM003Rev1 QPI Unit http://www.zenkara.com Typical Implementation Motivation Critical to explain rationale to development/engineering staff To reduce defects/problems in documents, code, tests and other objects – NOT to measure performance Keep other key stakeholders in the loop
  8. 8. Slide title: QPI/020/11-10 Slide Number: 8 SM003Rev1 QPI Unit http://www.zenkara.com Typical Implementation Method 1. Identified key documents/deliverables to be reviewed 2. Determined audiences and established level of formality required (walkthroughs or reviews or inspections” 3. Basic metrics identified – effort, defects (inc type) 4. Formalised existing reviews 5. Added informal followup 6. Created Call-for-Review and Review Log 7. Training of dev teams and testers 8. Introduced Review Closure form 9. Introduced code reviews 10. Introduced test script and other documents
  9. 9. Slide title: QPI/020/11-10 Slide Number: 9 SM003Rev1 QPI Unit http://www.zenkara.com Typical Review process Ready for review Asks moderator Call for Review Reviewers assess and record on log Review held Review Closure (metrics) Author corrects, signs off and gets Closure signed Deliverable signed off
  10. 10. Slide title: QPI/020/11-10 Slide Number: 10 SM003Rev1 QPI Unit http://www.zenkara.com Review process - Types of review 1. Walkthrough - presentation of a new object that few staff are familiar with. Seen as an exchange of information. 2. Review - detecting defects and discussing solutions. 3. Inspection - detecting defects only. Formality Effectiveness
  11. 11. Slide title: QPI/020/11-10 Slide Number: 11 SM003Rev1 QPI Unit http://www.zenkara.com Review Process - Roles Moderator - also performs scribe, organiser, etc. Author Reviewer Quality Assurance
  12. 12. Slide title: QPI/020/11-10 Slide Number: 12 SM003Rev1 QPI Unit http://www.zenkara.com Forms/Workflow Call For Review •basic metrics (amount of time, size of deliverable to be reviewed) and recorded (form includes Completeness, Correctness, Traceability, Testability, Consistency, conformance, etc Review Log Form •This form enabled all defects to be recorded BEFORE reviews take place. Closure Form Used to ensure loop is closed
  13. 13. Slide title: QPI/020/11-10 Slide Number: 13 SM003Rev1 QPI Unit http://www.zenkara.com Issues with Forms/Workflow Forms were often combined into one Forms initially seen by moderators as expensive in terms of time to complete. Disagreements over simple classification of: •Defect types •Origin/Injection Points
  14. 14. Slide title: QPI/020/11-10 Slide Number: 14 SM003Rev1 QPI Unit http://www.zenkara.com Use of Checklists Seen as useful by staff initially (and who were glad they didn’t have to actually tick the checklist items). In one instance, use of checklists tapered off after 3 months. This was rejuvenated through SEPG intervention, newsletters and refresher training for moderators. In another organisation, reviews degenerated into form filling activities – who can find the most spelling errors…
  15. 15. Slide title: QPI/020/11-10 Slide Number: 15 SM003Rev1 QPI Unit http://www.zenkara.com Use of Checklists - Improvement Specific examples were added to assist in understanding Based on suggestions from staff, existing checklists were improved (by adding, changing items) . New checklists were created e.g. There was a specification checklist that was applied to both user requirements and software requirements reviews. A separate checklist was created for the user requirements, and the existing spec checklist modified accordingly.
  16. 16. Slide title: QPI/020/11-10 Slide Number: 16 SM003Rev1 QPI Unit http://www.zenkara.com Role of the S/EPG or SQAG A distinguishing feature of the implementation was a lead role taken by the S/EPG (Software Engineering Process Group/ Enterprise Process Group) and/or Software Quality Assurance Group members. SEPG gradually withdrew from reviews of specific types of documents. SEPG members performed an SQA role from then on. Ownership of the process was gradually transferred to the software development manager. Roles: SEPG - Driving the process implementation SQA - Conformance to standards and implemented processes
  17. 17. Slide title: QPI/020/11-10 Slide Number: 17 SM003Rev1 QPI Unit http://www.zenkara.com Management Commitment Managers provided verbal support – limited success Managers attended training sessions – limited success Managers attended reviews – some success – but limited to egalitarian/flat structure organisations Managers had their deliverables reviewed – significant motivator for staff - lead by example
  18. 18. Slide title: QPI/020/11-10 Slide Number: 18 SM003Rev1 QPI Unit http://www.zenkara.com Management Commitment Another form of commitment: Re-estimated programs/projects to enable sufficient budget (time and money) for reviews.
  19. 19. Slide title: QPI/020/11-10 Slide Number: 19 SM003Rev1 QPI Unit http://www.zenkara.com Use of metrics Metrics were used to guide both the conduct of the reviews and the effectiveness of the review process itself. Conduct Effort spent in reviewing document, and reviews themselves Common types of defects % Deliverables slipping through the process Effectiveness Customer Satisfaction – surveys and anecdotal % of defects detected later than injection phase
  20. 20. Slide title: QPI/020/11-10 Slide Number: 20 SM003Rev1 QPI Unit http://www.zenkara.com Graphs A large number of graphs were also produced as part of the analysis of metrics collected (number and class of defects, size, effort expended). Initial use was to identify current state of the development, and to assess what needed immediate attention. # Reviews per month # Defects identified in each phase # Defects per document at T0 # Hrs of rework for each phase at T0
  21. 21. Slide title: QPI/020/11-10 Slide Number: 21 SM003Rev1 QPI Unit http://www.zenkara.com Graphs - Initial Num of reviews per month 0 50 100 150 200 N D J F M A M J J A S O N D J F M A M J J A S O N Date Numofreviews Code Doc Ramp-up between 10-18 months
  22. 22. Slide title: QPI/020/11-10 Slide Number: 22 SM003Rev1 QPI Unit http://www.zenkara.com Graphs - Initial Defects identifed in each phase 0% 5% 10% 15% 20% 25% 30% 35% User S/wreq Arch DD C/UT Integ QA Accept Phase Numofdefects T0 T0 + 1Y T0 + 2Y T0 + 3Y
  23. 23. Slide title: QPI/020/11-10 Slide Number: 23 SM003Rev1 QPI Unit http://www.zenkara.com Analysis • Initial over-focus in certain phases • Overall cost reduction in moving defect correction earlier in life cycle • Overall customer satisfaction improvement through better deliverables making it to them. • Though initial negative impact – as latent defects were being identified – but not identified as positive result of reviews.
  24. 24. Slide title: QPI/020/11-10 Slide Number: 24 SM003Rev1 QPI Unit http://www.zenkara.com Graphs - Subsequent Subsequent use was to provide guidance on what was working and what needed improvement, together with justification both internally and to the customer of the expense of reviews. % of Reviews by Doc type # Defects per document at T0 + 1 Year # Defects injected in each phase # Hrs of rework for each phase at T0+ 1 Year # Defects injected in each phase vrs identified at each phase
  25. 25. Slide title: QPI/020/11-10 Slide Number: 25 SM003Rev1 QPI Unit http://www.zenkara.com Graphs - Impressions Management Initially interested in the number of reviews and workload required for reviews. Found the overall reduction in average defects per delivered module very useful - in terms of justifying cost of reviews to directors and to demonstrate commitment to improvement for customers.
  26. 26. Slide title: QPI/020/11-10 Slide Number: 26 SM003Rev1 QPI Unit http://www.zenkara.com Graphs – Impressions Staff General concern about anonymity in most organisations. Demonstrated that no individuals could be identified - as data was only reported at a team level and organisation level. The Defects identified per phase and Defects per type of document graphs were used to identify improvements and to avoid common types of errors. Defects per phase useful (as a percentage) to start to estimate expected defect levels in future projects.
  27. 27. Slide title: QPI/020/11-10 Slide Number: 27 SM003Rev1 QPI Unit http://www.zenkara.com Initial analysis - Findings A lot of effort and duration spent on rework Staff were not: – using the checklists systematically – using the perspectives Order of documents/deliverables needs to be well established. More training was needed Many documents sent to review too early - needed criteria
  28. 28. Slide title: QPI/020/11-10 Slide Number: 28 SM003Rev1 QPI Unit http://www.zenkara.com Subsequent analysis of review process Staff were relying on reviews to find defects, and using them as a ‘catch-all’. Improvements to the requirements management and design processes were identified.
  29. 29. Slide title: QPI/020/11-10 Slide Number: 29 SM003Rev1 QPI Unit http://www.zenkara.com Training Training was generally conducted on two levels: 1. A general overview of the review process 2. More in-depth training session and exercises on moderating/leading reviews. This training was essential during the early stages. Some organisations used buddy training for most engineering practices.
  30. 30. Slide title: QPI/020/11-10 Slide Number: 30 SM003Rev1 QPI Unit http://www.zenkara.com Followup and refresher courses Just as important as the initial training was regular followup and refresher courses. These were to help ensure that omissions and mistakes were corrected with appropriate information. • common mistakes • collecting information This followup also took the form of regular memos and newsletters. • why and strategies • graphs • war stories and benefits • improvements
  31. 31. Slide title: QPI/020/11-10 Slide Number: 31 SM003Rev1 QPI Unit http://www.zenkara.com Benefits of the reviews ROI of between 5:1 and 2:1 in the first year Identification of defects earlier Improved customer satisfaction through the communications/perceptions of a rigorous process Reduced common types of errors
  32. 32. Slide title: QPI/020/11-10 Slide Number: 32 SM003Rev1 QPI Unit http://www.zenkara.com Benefits of the reviews Improved quality of product for each customer in the delivery chain Reduced rework after delivery Information exchange - product AND process
  33. 33. Slide title: QPI/020/11-10 Slide Number: 33 SM003Rev1 QPI Unit http://www.zenkara.com Side effects of the reviews Participants identified the need for other processes and standards (to help improve the documents). Involved the customers much earlier than expected (customer was impressed with sophisticated level of reviews, and one customer actually wanted to implement the process at their own company). Introduced review metrics database which quickly evolved to include defect tracking, estimation and company’s Balanced ScoreCard.
  34. 34. Slide title: QPI/020/11-10 Slide Number: 34 SM003Rev1 QPI Unit http://www.zenkara.com Keeping the ball rolling Set coverage targets of 20%, 50%, 80%. Initially include participation in reviews as appraisal criteria. Then after 6 months and the process in bedded down, include team coverage targets in appraisal criteria. Link an award to the highest coverage rate.
  35. 35. Slide title: QPI/020/11-10 Slide Number: 35 SM003Rev1 QPI Unit http://www.zenkara.com Keeping the ball rolling Ensure someone monitors the rate and performs refresher when needed. Often SEPG/SQAG but more effective when performed by team leaders. Make it FUN! Get the participants to review the review process itself Get the developers to review management material
  36. 36. Slide title: QPI/020/11-10 Slide Number: 36 SM003Rev1 QPI Unit http://www.zenkara.com Strategy and method Begin with the end in mind - what goal do you want to achieve using reviews? Ensure that reviews are not seen as an excuse to shift responsibility. Don’t have too many meetings Hold light peer reviews (without the call-for-review) Try different approaches to the review process (tailor for the organisation) and to the implementation of the process.
  37. 37. Slide title: QPI/020/11-10 Slide Number: 37 SM003Rev1 QPI Unit http://www.zenkara.com Outsourcing and Cultural Issues Need careful adjustment of review processes due to multi-national teams being formed. Need to express reviews in different perspectives. Review process provides visibility into the processes at outsourcing partners
  38. 38. Slide title: QPI/020/11-10 Slide Number: 38 SM003Rev1 QPI Unit http://www.zenkara.com Metrics Start collecting metrics as soon as possible. Provide guidance figures based on metrics as soon as possible. Average of pages/hr - individual review Average of pages/hr - group review ‘Normal’ data - expected defects per page/KLOC
  39. 39. Slide title: QPI/020/11-10 Slide Number: 39 SM003Rev1 QPI Unit http://www.zenkara.com About – James Kelly James Kelly is CEO of Zenkara. He has extensive experience in process improvement and software engineering. He has consulted for over twenty years across software intensive industries including aerospace, defence, engineering, utilities and finance. James has presented at numerous conferences and workshops in areas such as business improvement, software metrics, risk management and quality. He has served on industry boards such as ASMA, SQA, and was involved in editing the ISO9126 Software Product Quality standard. James holds a Bachelors degree in Informatics, a GradDip in Software Quality and an MPhil in Software Quality. He can be contacted at james@zenkara.com
  40. 40. Slide title: QPI/020/11-10 Slide Number: 40 SM003Rev1 QPI Unit http://www.zenkara.com Who are we? Zenkara provides advanced technology for mission- critical software engineering companies. Head Office: 2/27 Waverley Street Annerley Qld 4064 Australia PO Box 1622 Milton QLD 4064 Mobile:+61 (0)410 616125 | direct: +61 (0)7 3137 1666 email: james@zenkara.com web: http://www.zenkara.com ABN: 85 050 409 660

×