• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Project Reviews

Project Reviews






Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • (updated by Wells 10/98 to OREVIW37.ppt) Definition: Reviews and meetings are gatherings of multiple people for a business purpose. This may seem like a trivial subject, but we spend a large part of our working life in meetings - and much time can be wasted and lost. So in the spirit of process improvement, we will look at ways to improve it. Some people consider meetings to be where work gets done . Others consider that meetings prevent work from getting done. Query: any students used REVIC before? Done FI before? (few or none) Anyone been in a meeting? (everyone) Let’s avoid “California meeting:” congenial, well-intentioned, inconclusive.
  • The purpose of this section is to (1) discuss the different types of reviews and meetings held on a software project (2) look at their purposes and pitfalls (3) examine steps for making them successful.
  • Three types of gatherings we will address here.
  • Project Plans (e.g. SDP para 5.18) requires approach to be followed for TRs and MRs. “The planning in each paragraph shall cover all contractual clauses regarding the identified topic. For both TRs and MRs, plan must include “a proposed set of reviews.” “ Reviews shall focus on in-process and final software products, rather than material generated especially for the review.” A technical review is for technical people only to discuss technical issues, not management issues. During requirements phase these technical experts might be the user and the customer, but during a code inspection, the user and the customer are probably not experts. Having the right people attend the right meeting makes everyone’s job more productive and the overall team more productive.
  • these 6 bullets from 12207.0 para; and calls for providing evidence that... 498 Para 5.18.1 says JTRs objectives are to review evolving software products, using as criteria the software product evaluation criteria in App. D... App D of 498 is mandatory, lists 29 software products and associated evaluation criteria (below) Criteria defined. Criteria can be tailored by acquirer. Instructions describe as-built product Adequate test cases, procedures, data Consistency Matches specified DID Covers all items (traceability) Feasible Follows software development plan Internally consistent Meets CDRL and SOW, if applicable Presents a sound approach Meets requirements under test Testable Understandable
  • TRs are conducted on single software products MRs on development phases (e.g., software design) SDR = system design review, per DOD-STD-2167A, at end of system requirements analysis/design phase in SDR, documents reviewed include System Spec System/Segment Design Document Prelim Software Reqts Spec (SRS) Prelim Interface Reqts Spec (IRS) Software Development Plan (SDP) these form the Functional Baseline for approval to proceed to Software Requirements Analysis phase. Final SRS and IRS lead to SSR - subject of next exercise (can show reviews in 2167A diagram on other viewgraph machine)
  • This meeting focuses on management issues. The technical issues worked on at the Technical Review are summarized for the managers. The management meeting focuses on schedule and risks, the two most important management issues on a project.
  • This from 12207.0 para
  • 2167A para 4.1.2 calls for formal reviews and audits as required. Cites MIL-STD-1521 for guidance. 1521 parts for FCA, PCA, FQR superseded by MIL-STD-972 (CM) 498 covers reviews in 5.18 and Appx E. “ Joint ” means acquirer and developer. 498 Appendix E describes CANDIDATE reviews - not a mandatory part of the standard - guidance only. These not required or preclude alternatives or combinations. 12207.0 Para 6.6 covers Joint review process . Two parties are reviewing party and reviewed party.12207.2 Annex G has Candidate JMR’s. cites IEEE std 1028 for more info. 12207 Para 6.7 covers Audit process for auditing party and audited party. Tech Reviews: not addressed in 2167A called “Joint Technical Reviews” in 498 called “Technical reviews” in 12207 498 Para 5.18.1 says JTRs objectives are to review evolving software products, using as criteria the software product evaluation criteria in App. D...
  • A Management Review implemented locally is the Management Project/Design Review. This instruction recently signed. D10 designated as DRC associate.
  • Third type of review/meeting is the informal meeting.
  • So what can go wrong in reviews and meetings? Instructions: poll class for ideas and write on board. Examples: - No agenda or focus - Real issues or problems not identified, solutions not found - status not revealed, information not disseminated - time of attendees wasted - morale lowered - action items not assigned - no due dates - too much time spent presenting wrong information - wrong people there, needed people missing (decision makers or needed technical experts) - Dog and pony shows - too much time spent presenting wrong information - No insight into actual developer activities - Developer spends excessive time preparing - Sponsor/manager leaves feeling project doing well, when it might have some real problems - Decisions and actions not recorded - Meeting was really not necessary - Meeting held only because it is regularly scheduled - Meeting attendees unprepared - Meeting leader unprepared Adds time, cost, and problems to the project!
  • Instructions: Each table HOLD A MEETING (3-5 minutes) to identify 3 suggestions for better meetings. (If short of time, ask for a few answers and proceed.) Instructor write suggestions on white board, students write them into book. Ask backup instructor to take notes for later review and inclusion in instructor’s answer.
  • Now we will review some ideas to make meetings more productive. (and compare to exercise just completed) Purpose: turn common sense into common practice The SPM (or person responsible for a review/meeting) can use these rules to conduct effective meetings Other attendees can use steps to gently urge improvements to poor meetings
  • Establish GOALS and OBJECTIVES. To an SPManager, goals and objectives are always important - what are you trying to accomplish? Clarify this to the team members. Determine this prior to the meeting, so everyone knows why the meeting is occurring. For those of you already following these 10 steps this is a review and an affirmation. For those who aren’t try to leave here by choosing one of these to implement. TR MR Staff meet
  • Establish ENTRANCE CRITERIA and EXIT CRITERIA. Entrance Criteria of a Process are the elements or conditions needed to begin. Exit Criteria of a Process are the elements or conditions needed for completion. e.g. - Entrance Criteria of Formal Inspection Inspection Meeting: Have all inspectors completed FI log All agree ready for meeting Exit Criteria - all action items assigned open issues identified
  • Aid: sample agenda FI Roles: moderator, presenter, recorder, inspector 3 Formal Inspection criteria for proper participants: skills, knowledge, interest. We saw in the Formal Inspection process that preparation was the key to success there.
  • Steps 4 and 5 are kickoff meetings. Both can occur via teleconference or video teleconference - they don’t have to be face-to-face in the same room. The main point is to check in and make sure everyone is ready and on the same page. In Formal Inspections: Phase 1 is planning meeting Phase 2 is Overview meeting (optional) Phase 3 is Preparation Phase 4 is Inspection...
  • In programs where there are multiple government organizations it is bad when a meeting is taken up by the government folks arguing. Most of this should be settled prior to the meeting.
  • Logistics, Facilities, Amenities, Name Tags... As in this course, you should have food, drinks and plenty of breaks. Most people can only stand about 60 minutes of information at a time before they need a break. Also it helps to break it up - some lecture, some exercises. This (SPM course) room is arranged so that you can talk to the other members of your team. If we just wanted you to listen and us to talk we would have set you up in a row. So all meetings should be an oval or a U, since people are there to communicate.
  • Aid: QMB Ground Rules Aid: Project Meeting Process and Rules of Order Conflicts are healthy - you want people to interact - if people are silent and every agrees all the time, this is unnatural. Issues should be discussed openly and resolved. Anything you can do to stimulate interaction is important, since again this is why you have a review meeting - to get other opinions on your work.
  • Aid: Meeting Minutes Take MINUTES of Meeting Results and assign ACTION ITEMS Priority is also helpful to the actionee so they will be able to decide which items you believe are most important just in case they don’t have time to do them all (oh, that never happens). (Not a job to be delegated to a secretary.)
  • Aid: QMB Meeting Evaluation Form Request FEEDBACK As modeled in this course, you should get feedback from the participants. They will have good ideas and since they are the ones attending the meetings, you should make it better for everyone.
  • Aid: Action Item Report Track, follow up on ACTION ITEMS Who does this? recorder, CM person, Secretary, SPM?? The acquirer (governmen) should track the action items - not the developer - it just makes for better tracking. By knowing the percent of outstanding action items, you will have a gauge if you are ready to proceed.
  • Here is a summary of the ten steps.
  • Here is a sample agenda with some of the important points.
  • Question (if time): What can you do if you are a lowly participant in a review or meeting that is going badly? Main points: reviews are important and we might as well make them the best we can - more productive. One way is to have the technical people concentrate on the technical stuff and the management people focus on the management stuff. Most projects skip requirements review and jump right to design reviews. In the next section we will see why this is a problem. The next section focuses on a specific Management Review - the Software Specification Review.

Project Reviews Project Reviews Presentation Transcript

  • Project Reviews and Meetings
  • Objectives
    • What Software Project Managers need to know to:
    • Identify the types of reviews and meetings
    • Understand when and why to hold reviews and meetings
    • Use the 10 Steps to productive reviews and meetings
  • Types of Reviews and Meetings
    • Technical Reviews Address technical issues: requirements, design, code
      • Example: Formal Inspections
    • Management Reviews Address project issues: status, budget, schedule Example: Design Review
    • Meetings Gathering of people for a business purpose Examples: staff meetings, committee meetings, training sessions
  • Technical Reviews
    • Address technical issues: evolving software products, services, solutions
    • Are attended only by persons with technical knowledge of the subject matter, not management
    • - Includes both acquirer and developer technical personnel - In requirements phase includes customers, users - Includes SQA, SCM, V&V, test as needed
    • Report the actual technical status of the project to management
    • Identify risks and issues to be raised at Management Reviews
    • Examples: Formal Inspection Code walkthrough Design tradeoff meeting Process review
  • Technical Review Criteria for a Software Product or Service
    • Is it complete?
    • Does it comply with standards and specifications?
    • Are changes properly implemented?
    • Does it adhere to the applicable schedule?
    • Is it ready for the next planned activity?
    • Is development being conducted according to the plans, standards, and guidelines of the project?
  • Technical Reviews provide inputs to Management Reviews Technical Review resolve defects Technical Review resolve defects Prelim I’face Spec. Management Review (software design review) Technical Review resolve defects Plan Prelim Reqts. Spec. status risks issues concerns questions
  • Management Reviews
    • • Address project issues: status versus plans, schedules, standards
    • • Keep management informed about status, direction, agreements
    • • Are attended by technical leaders, project managers, and managers
    • (with decision authority over cost and schedule)
    • • Identify and resolve risks
      • - Are we ready to continue? Should we continue?
    • • Receive input, resolve issues from several Technical Reviews
    • • Examples: Requirements review
    • Design review
    • Test readiness review
  • Management Review Criteria
    • Is progress according to plan?
    • Are schedules, standards, and guidelines being followed?
    • Are resources adequately allocated?
    • Are risks jeapordizing success?
    • Are we making good decisions based on metrics?
    • Do we need to change direction or revise plans?
  • Management Review Terminology DOD-STD-2167A MIL-STD-498 IEEE/EIA 12207 Formal Reviews (10) Joint Mgmt. Reviews (11) Project mgmt. reviews (11) Software plan review Software plan review Operational concept review Operational concept review System Reqts. Rev.(SRR) System/subsys. reqts rev. System/subsys. reqts rev. System Design Rev.(SDR) System/subsys. design rev. System/subsys design rev. Software Spec. Rev. (SSR) Software reqts review Software reqts. review Prelim Design Rev. (PDR) Critical Design Rev. (CDR) Software design review Software design review Test Readiness Rev. (TRR) Test readiness review Test readiness review Test results review Test results review Production Readiness Rev(PRR) -- -- Software usability review Software maintenance rev. Software supportability rev. Software supportability rev. Critical reqts. review Critical reqts. review Functional Config Audit (FCA) (FCA in MIL-STD-973) (FCA in IEEE Std 1042) Physical Config Audit (PCA) (PCA in MIL-STD-973) (PCA in IEEE Std 1042) Formal Qual. Review (FQR) (dropped by MIL-STD-073) -- (see MIL-STD-1521B) (see 498 Appendix E) (see 12207.2 Annex G) (see also IEEE Std 1028)
  • SSC SD Management Project/Design Reviews SPAWARSYSCEN SAN DIEGO INST 3912.1A of 18 Dec 1997
    • Development projects will be subject to periodic review
    • Purpose: to help project managers meet cost, schedule, and technical requirements
    • SC SD Department Heads to identify applicable projects
    • Program Managers to adhere to policies and procedures
    • Design Review Committee to coordinate reviews
    • Review topics:
    • Management practices Technical processes
    • Requirements and approaches Test and evaluation
    • Schedule and budget Documentation plans/status
    • Procurement status Product assurance plans/status
    • Instruction available at:
    • http://iweb.spawar.navy.mil/services/sti/publications/inst/subjects.html
    • Purposes:
    • Convey information to a group
    • Solicit information
    • Answer questions
    • Brainstorm
    • Make a decision as a group
    • Convince or persuade team of idea
    • Maintain team spirit, involvement
    • Examples: Weekly Status Meeting
    • All-Hands Meeting
    • Committee Meeting
    Meetings “ Are you lonely? Working on your own? Hate making decisions? HOLD A MEETING!”
  • Question: What are the Consequences of Poorly-Run Reviews and Meetings?
  • Exercise: How Can Reviews and Meetings Be More Productive?
  • The Steps to Successful Reviews and Meetings
  • Step 1: Establish Type of Review/Meeting and the G______ and O____________
    • Determine type of review/meeting: Technical Review, Management Review, program review, status meeting, staff meeting, etc.
    • What outcome or decision do you expect to reach?
    • Should be goal-oriented, value-added, and primarily non-adversarial
      • Examples:
      • “ Reach agreement on interface requirements.”
      • “ Review project status and risks to determine if requirements need to be reduced.”
      • “ Announce the new project organization and decide on new office spaces.”
  • Step 2: Establish E_______ C_________ and E______ C_________
    • • Entrance criteria : What must occur prior to the review or meeting in order to make it successful
        • Derived from goals/objectives
        • Examples: Completion of the work product to be approved
        • All attendees read IRS, review risks
    • • Exit criteria : What must be accomplished for the review or meeting to be closed
        • Example: Identify and document all discrepancies
    • • Both must be established prior to review/meeting
  • Step 3: Be Organized; Be Prepared
    • • Select the right participants - get a good mix
      • - Invite only those who have a stake in the outcome
      • - Continuity of participants is important!!
    • • Assign roles: leader, facilitator, timekeeper, recorder
    • • Have an agenda - keep to it
      • - Hand out agenda ahead of time
    • • Insist that participants be prepared
  • Step 4: * Hold a kick-off meeting for Reviews
    • • Review goals/objectives of the review with the developer (participants)
      • - Schedule at least two weeks prior to the meeting
      • - Doesn’t have to be face-to-face in the same room,
      • could be video teleconference or phone call
    • Example: Formal Inspection Overview Meeting
    • * - applies to reviews only
  • Step 5: *Hold a Government-only pre-review meeting (if applicable) • Evaluate goals/objectives of the review, controversial areas, known deficiencies • Purpose is to achieve Government consensus • Most important if multiple Government agencies are involved * - applies to Management Review only
  • Step 6: Get Off to a Good Start
    • Make the participants feel comfortable
      • - Ensure adequate facilities (space, lights, air conditioning, ...)
      • - Set up room to accommodate the objective
    • (for best communications, use U-shaped or oval)
    • • Arrange for food, drinks, breaks
    • • Provide welcome and introductions
    • • Summarize roles, goals, objectives, agenda
    • • Verify that Entrance Criteria have been met
  • Step 7: Establish Ground Rules
    • • Getting everyone’s input
      • - Use round robin or query those not contributing
      • - Show appreciation for constructive participation
      • - Encourage open communication
      • - Use everyone’s talents--that is why they are there
    • • Limiting the number and length of presentations
      • - Agree on time limits, assign timekeeper
    • • Controlling the group size
      • - If the group is over 10, divide the group into smaller
      • teams to divide up the issue to be discussed
    • • Using prototypes to assist participants in
    • understanding and communication
    • • Handling disagreements or conflicts
  • Step 8: Take M__________ of Proceedings and Assign A________ I________
    • • Sample contents:
    • Review name and objectives
    • Attendees
    • Results and Decisions
    • Action Items
    • Assign action items for open issues
      • - Specify due date, priority, and responsible person
    • • Review action items and decisions prior to close of review/meeting
      • - Action Items that can be answered during the review/meeting should be answered then and allow time for more detailed analysis of more profound Action Items
    • • Confirm that Exit Criteria are met
    • Send out minutes in a timely manner for review and comment
  • Step 9: Request F__________ on how to improve the review/meeting process
    • • Reviews and meetings span the life of all projects
    • • All attendees want reviews and meetings to be productive
    • • Example feedback questions
      • - Was the agenda available beforehand?
      • - How can we foster better communication?
      • - Do we have the right attendees?
      • - Were the physical facilities adequate?
      • - How can our reviews and meetings be improved?
  • Step 10: Track, Follow-up on A_______ I______
    • • Establish an Action Item tracking system
    • Sample Contents: A.I. number
    • Description
    • Priority
    • Date Assigned
    • Responsible person(s)
    • Estimated Completion Date
    • Status
    • Date Closed
    • • Collect the metric: outstanding action items
      • - Measures the health of a software project
    • • Schedule an in-progress (status) review or meeting if needed
    • • Prepare for next review/meeting
  • Summary: The Steps to Successful Reviews and Meetings
    • 1. Establish type of review/meeting and the goals and objectives
    • 2. Establish entrance criteria and exit criteria
    • 3. Be organized, be prepared
    • 4. Hold a kick-off meeting (for Reviews only)
    • 5. Hold a Government-only pre-review meeting (for reviews only)
    • 6. get off to a good start
    • 7. Establish ground rules
    • 8. Take minutes of proceedings and assign action items
    • 9. Request feedback on how to improve the review/meeting process
    • 10. Track, follow up on action items
  • Sample Agenda
    • XYZ Project Status Meeting, Jan 2, 2000
    • 10:00 Welcome and introductions Rex
    • 10:05 Meeting Objective : Assess impact of new Rex project requirements on schedule
    • 10:07 Entrance Criteria: Attendees review Rex change proposal ECP123 and our SDP
    • Exit Criteria: Agree on impact to schedules,
    • organization, and costs
    • 10:10 Proposed Ground Rules ... Colin
    • 10:15 Discussion Items:
    • 1. Review change proposal Mary
    • 2. Impacts to SDP, schedule, costs George
    • etc.
    • 11:15 Review of Action Items Jay
    • 11:20 Feedback on meeting process Rex
    • 11:29 Set date for next meeting Rex
    • 11:30 Adjourn
  • The Software Project Manager shall:
    • • Conduct reviews and meetings when appropriate
    • • Separate technical reviews from management reviews
    • • Apply the 10 key steps to make them successful
    • Implement Step 9: Please fill out your evaluation form for this section now.
  • References
    • • IEEE Std 1028, IEEE Standard for Software Reviews and Audits
    • • SPAWARSYSCEN SD INST 3912.1A Management Project/Design Reviews
    • • MIL-STD-1521B, Technical Reviews and Audits for Systems,
    • Equipments, and Computer Software . Describes 10 reviews and
    • audits for DOD-STD 2167A. Cancelled 10 April 1995.
    • • MIL-STD-973, Configuration Management. Supersedes 1521B for FCA, PCA, FQR.
    • • SEPO, Peer Review Process
    • • SEPO, SSC SD Software Management for Executives Guidebook
    • • Weinberg, Gerald M., Daniel P. Freedman, Handbook of Walkthroughs,
      • Inspections and Technical Reviews Evaluating Programs, Projects,
      • and Products