Riskapedia
            An ESMD Risk Management Service
               for NASA’s Project Managers,
                Project Engineers, and Risk
                       Practitioners


                       Presented by Isaac Addison Heard
                              ARES Corporation

             on behalf of ESMD Integrated Risk & Knowledge Management Systems
              at the NASA 2010 Program Management Challenge February 9th, 2010




                                                                                 1
Used with permission
2
Every year or so, web-based
technology is introduced which
revolutionizes the way we work…




                                  3
wiki




       4
Wikipedia is written collaboratively by volunteers from all around the world; anyone can edit it.
        Since its creation in 2001, Wikipedia has grown rapidly into one of the largest reference web
        sites, attracting at least 684 million visitors yearly by 2008. There are more than 75,000 active
        contributors working on more than 10,000,000 articles in more than 260 languages. As of today,
        there are 2,859,752 articles in English. Every day, hundreds of thousands of visitors from around
        the world collectively make tens of thousands of edits and create thousands of new articles to
        augment the knowledge held by the Wikipedia encyclopedia.




wi·ki        (wĭk'ē)
n.   pl. wi·kis
A collaborative website whose content can
be edited by anyone who has access to it.




                                          “The new force that is bringing
                                          people together on the net to
                                          create a giant brain.”
                                                                                                            5
ESMD has over 350
                                              wikis spaces in ICE

• Meeting Calendar Archive
• Contacts, ICD and IRD Development
• Upcoming meeting info and useful links




                    wikis in ESMD


                                           • Repository of
                                             Ares PDR data
                                           • Offers quick links
                                             to relevant pages
                                           • Schedules,
                                             contacts, and
                                             other useful info
 • News R&KM updates
 • R&KM related Process, People,
   Meetings, and Tools
 • Most recent directorate top risk list                            6
Riskapedia




             7
Riskapedia (pronounced risk·ĕ·pē·diə)

A collaborative website to help NASA
engineers identify, analyze, and mitigate
risks.

Risk Managers   Project Managers   Project Engineers




                                                       8
9
Astronauts use checklists and you
should too…



              Smart people have learned a
              lot of lessons learned into
              checklists that can help you
              identify risks on your project.
                                            10
TRIMS


                                         MITRE Risk
                                         Management
                                           Tool Kit




                                                Constellation
                                                  Program
                                                   Hazard
                                                  Analysis
                                                Methodology

                                  Software
  The Identify Risk              Development
                                  Taxonomy
section is a checklist            Carnegie
                                    Mellon
 based approach to
   defining risks                                               11
Risk identification wiki in practice
…3 easy steps for any engineer

1.   Select the Risk Identification icon and select the relevant risk identification
     checklists / taxonomies.

2.   Use one or more checklists / taxonomies to identify a candidate risk list.

3.   Augment Riskapedia identification checklists / taxonomy based on
     experience (if required)



                                                                                       12
Probability Density




                      Relative Area                                                                              Requirement                                                                    5
                      = Probability
                       of Meeting
                      Requirement                                                                                                                                                               4
                                                                                                                                                                                            Prob.
                                                                                                                                                                                                                                                             1.0
                                                                                                                            Failure of
                                                                                                                             System                                                             3                                                                                    12/31/98

                                                                                                                                                                                                                                                                                                                          Revised Evaluation
                                                                                                                                 Or                                                                                                                          0.8
                                                                                                                                                                                                                                                                                                                          Initial Evaluation



                                                                                                                                                                                                2                                                            0.6
                                                              Weight of A
                                                                   Failure                                  Failure of                       Failure of                    Failure of




                                                                                                                                                                                                                                                Likelihood
                                                                 Sub-System                               Sub-System B                     Sub-System C                  Sub-System D


                                                                                                                                                                                                1                                                            0.4



                                                                                              Failure of                    Failure of                    Failure of                                                                                         0.2
                                                                                            Sub-System E                  Sub-System F                  Sub-System G

                                                                                                                                                                                                    1   2     3      4   5                                   0.0
                                                                                                                                                                                                                                                             11/22/98   12/6/98   12/20/98   1/3/99   1/17/99   1/31/99     2/14/99     2/28/99   3/14/99
                                                                                                                                                                                                            Impact                                                                      Date CSOC Ready to Proceed



                                           PRA Techniques                                                                                                                                           Risk Matrices                        Schedule Analysis


                           Tools enable a team to assess risk
                                                                                                                    3.    LTA                          4. LTA Meeting



                                                                                                                                                                                            Decision Trees
                                                                                       2. LTA
                                                     1. LTA Processes                                                     Participation                Ground Rules
                                                                                           Resources
                                                                                                          1. Wrong People (too many,       1. Unclear Purpose
                                                   1. Agenda                                              too few, no portfolio)
                                                                                   1. LTA Time                                                    2. LTA Agenda
                                       2. Scheduling - Notification
                                                                                                                 2. LTA Preparation
                                      3. Submission/Input of Crit Read-Ahead            2. LTA Facility                                       3. Expectations (data v. discussion)

                                      4. Dist. of Read-Ahead Mat.                                                   3. Confrontational            4. Expectations (Content)
                                       5. Actions Assignment / Outcome        3. LTA Presentation Template
                                                                                                                         4. “Hidden Agendas                  5. Attendees
                                       6. LTA Minutes / Doc of Decisions
                                                                                       4. AV-Laptops/Projection          5. Too Quiet / Dominant                  6. PDA/cell Use
                                                     7. LTA Action Tracking




                                                                                                                                                                                                    AoA
                                                                                  5. Support                                                                            6. Convention
                                                                                                          4. Support                       5. Interactions              5. Graphics
                                                                                4. Security
                                                                                                   3. Engagement                   4. Consequences                    4. Density
                                                                           3. Access /
                                                                           Registration
                                                                                               2. LTA Time Mgmt                       3. Risks         3. Clarity (font, acronym)
                                                                        2. Usability                                                                    2. Format (ppt v.
                                                                        /Speed              1. Articulation of                    2. Issues             Narrative)
                                                                                            Values / Goals /
                                                          1. ThinkTank / CoPs /             Objectives /                                         1. Too Much/Little
                                                          Wikis / Portals                   Importance /           1. Activities / Sched
                                                                                            Alignment
                                         5. Other                                                                             8. LTA            9. LTA Content


                                                                                                                                                                                                                               Root Cause
                                                                                  7. LTA Leadership
                                                            6. LTA Web-Based Tools                                            Content Relevancy Format




                                                                      Fish Bones
                                                                                                                                                                                        Affinity Diagram
                                                                                                                                                                                                                             Balanced                                         Etc…
                           Cost/Benefit                                                                                                                                                                                      Scorecard                                                                                                     13
                            Analysis
Probablistic
   Risk
Assessment
                          5x5 matrix

                         1.   Summary
                         2.   How to
                         3.   Examples
                         4.   References




                                        Visual
                                       Literacy




               Critical Path
                 Analysis




                                                  14
Risk Assessment wiki in practice
                     1.                     Start with your risk records

                     2.                     Some risks have gone stale or need stronger characterization

                     3.                     Browse tools targeted at assessing risk and select one; perhaps
                                            the cause-and-effect diagrams

                     4.                     Hold a 2-hour facilitated team activity to re-energize the risk
                                            thinking amongst your team


                                                            Unclear Work                Handling Ring
 Brainstorm what went wrong in critical lifting process
                                                            Instructions                Design                        Time Pressure
                                                                                                                                                                     Participant        1    2    3    4      5     6
                                                                      - Vague                        - New
% Potential                                                                                                                                                          Add Photos          20   10   10
 Ideas on                                                              - No diagrams                   - Holes did not line up
                                               Structured                                                                             - Mixed messages from
   Topic
                                               Brainstorm                                                - Poor documentation         managers                       Explain rationale   10   25   15
                                                                            - No explanation
                            Unstructured        Round 2                                                                                                              for each step
                             Brainstorm
                              Round 1                                           - Training was not                                                                   Provide Tech Edit        45   5    ……..   etc
                                                                                adequate for this            - Background noise
                                                                                new operation                                            - SMA and lift team never   Provide video       50   10   60
              Structured                                                                                 - Headset static                pre-briefed
              Brainstorm
                                                                           - No time for training                                                                    Provide QA &        20   10   10
                                                                           entire crew
               Round 1                                                                                                                                               Lessons Learned
                                                                                                                                 Coordination
                                                            Lift Crew Training            Communications                         Between Lift Team
                                                                                                                                 & SMA




              Brainstorming                                                  Diagramming                                                                                          Prioritizing                           Action Plan   Teambuilding
                                                                                                                                                                                                                                                  15
Others have
solved similar
 risks and you
can learn from
what they did




             You are not alone…

 Risks similar to
yours have caused
 failures in other   … but you don’t have time to
programs and you               go search for this
  can learn from
                                      information
       these                                    16
Verification
                                Risks

                                                        Engineering
                                                        Design Risks




                                                                    Quality
                                                                   Assurance
                                                                     Risks



                                             Human
                                             Factors
                                               and
Nodes are focused on risk                   Software
                                            (recently
mitigations in engineering                   added)
        disciplines                                                            17
Informed By
                   Table of Contents
                   Synopsis
                   Fishbone
                   Risk Prevention Strategies
                   Design Engineering Risks Good Examples from ARM and IRMA
                   Experience-based Insights
Lets take a tour   Noted Space System Failures

through TOC…
                   Relevant Documents
                   References
                   Keywords
                   Related Topics

                                                                         18
Table of Contents
Synopsis
Fishbone
Risk Prevention Strategies
Design Engineering Risks Good Examples from ARM
and IRMA
Experience-based Insights
Noted Space System Failures
Relevant Documents
References
Keywords
Related Topics




                                                  19
Supporting Functions


Risk University

Get Help – Risk Squad

Discussion Forums
                                    20
Suggested study
                                      duration and timeline


                                    Syllabus designed to
Currently 8 self study              teach Level 3 and Level
courses                             2 risk managers the
• How to create well written risk   best practices of risk
                                    management
  statements
• Integration risks
• NASA risk policy
• Etc…
                                                              21
Lightweight      Process 2.0 –
                                                                                                                              Tank – Risk      Facilitation
 PANEL URGES SHIFT IN STATION’S ORBIT
                                                                                                                              Management       techniques to
                                                                                                                              Case Study       involve your
By WILLIAM J. BROAD




                                                                                                                                                                                                         Asynchronous
Published: June 9 , 1993

The expert panel advising the White House on redesigning the space station has called for the proposed astronaut outpost to be
launched into a “world orbit” where it could be reached not only by American space shuttles but also by Russian, Japanese, and




                                                                                                                                               group in
Chinese rockets.
                                                        …




                                                                                                                                                                                                         collaboration –
“It would change things in a fundamental way,” said Dr. Bruce Murray, a planetary scientist at the California Institute of Technology.
“It would say it’s not an American space station but an international one. It would say that the Cold War really is over and that we’re
enthusiastic about going on to the new phase instead of acting like we’re trying to prevent time from marching on.”
                                                        …




                                                                                                                                               solving your
Today, winged spaceships soaring out of Cape Canaveral usually fly into an orbit inclined 28.5 degrees to the Equator, a path beyond
the reach of the Russians. That orbit was also where the space station, proposed in 1984 amid the Cold War, was to be built




                                                                                                                                                                                                         brainstorm, bin, rank,
piecemeal as the American shuttle fleet carried its numerous parts into space.
                                                        …
Now, the 16-member White House advisory panel, headed by Dr. Charles M. Vest, president of the Massachusetts Institute of
                                         Now, the 16-member White House advisory panel, headed by
Technology, has endorsed a higher inclination for the American station in working papers and a draft report for President Clinton.
                                  Dr. Charles M. Vest, president of the Massachusetts Institute of Technology,


                                                                                                                                               problems
                                                        …
If the station is launched into an orbit the Russians can reach, the white paper said, the United States could “use their entire stable of
                                 has endorsed a higher inclination for the American station in working papers


                                                                                                                                                                                                         actions.
previously developed Soviet launch vehicles – as needed.”
                                                    and a draft report for President Clinton.
                                                        …
Cooperation with Russians could reduce costs, but the paper noted that the station’s current international partners, Japan, Canada,
and Europe, “generally disagree with us about the desirability of this orbital inclination.”

The drawback of the proposed path, it noted, is that shuttles flying to a higher inclination can lift less payload, up to 11,500 pounds
less than the craft’s top lifting power of 55,000 pounds…



                                                                                                                                              Brainstorm what went wrong in critical lifting process

                                                                                                                                             % Potential
                                                                                                                                              Ideas on                                      Structured
                                                                                                                                                Topic
                                                                                                                                                                                            Brainstorm
                                                                                                                                                                         Unstructured        Round 2
                                                                                                                                                                          Brainstorm
                                                                                                                                                                           Round 1


                                                                                                                                                           Structured
                                                                                                                                                           Brainstorm
                                                                                                                                                            Round 1
                                                                                                                                                                                                                                  22
• Ask a question
• Propose discussion on a topic of interest
• Answer someone else’s question




                                              23
Riskapedia

Designed to help you
  1. Identify risks
  2. Analyze risks
  3. Mitigate risks

        …And get work done
                         24
Riskapedia is currently based on
 input from some smart people…

 … but not as smart as everyone in
 this room.


We want your input, your insights, and
your examples to help NASA projects
benefit from your knowledge!
                                         25
What you can do…




                   … contribute!

                                   26
ITAR, SBU
                                        (not allowed in wikis)



Challenge: Contribute something
this week – just one thing.

  • Edit content in Risk ID, Risk Assessment, and Risk
    Mitigation notes (almost all content is editable).

  • Initiate or participate in discussions

  • Vote, rank, or comment on content usefuleness

  • In consultation with ESMD - become a subject
    matter practitioner with full editing rights to “own”
    a content area.
                                                            27
Your benefit for contributing
1. You will feel good about sharing your
   insights

2. You will have participated 1st hand in
   the wiki movement

3. Some NASA engineer, somewhere, will
   be grateful as they incorporate your
   advice into solving their problem.

                                            28
Questions?




             29

Heard

  • 1.
    Riskapedia An ESMD Risk Management Service for NASA’s Project Managers, Project Engineers, and Risk Practitioners Presented by Isaac Addison Heard ARES Corporation on behalf of ESMD Integrated Risk & Knowledge Management Systems at the NASA 2010 Program Management Challenge February 9th, 2010 1 Used with permission
  • 2.
  • 3.
    Every year orso, web-based technology is introduced which revolutionizes the way we work… 3
  • 4.
  • 5.
    Wikipedia is writtencollaboratively by volunteers from all around the world; anyone can edit it. Since its creation in 2001, Wikipedia has grown rapidly into one of the largest reference web sites, attracting at least 684 million visitors yearly by 2008. There are more than 75,000 active contributors working on more than 10,000,000 articles in more than 260 languages. As of today, there are 2,859,752 articles in English. Every day, hundreds of thousands of visitors from around the world collectively make tens of thousands of edits and create thousands of new articles to augment the knowledge held by the Wikipedia encyclopedia. wi·ki (wĭk'ē) n. pl. wi·kis A collaborative website whose content can be edited by anyone who has access to it. “The new force that is bringing people together on the net to create a giant brain.” 5
  • 6.
    ESMD has over350 wikis spaces in ICE • Meeting Calendar Archive • Contacts, ICD and IRD Development • Upcoming meeting info and useful links wikis in ESMD • Repository of Ares PDR data • Offers quick links to relevant pages • Schedules, contacts, and other useful info • News R&KM updates • R&KM related Process, People, Meetings, and Tools • Most recent directorate top risk list 6
  • 7.
  • 8.
    Riskapedia (pronounced risk·ĕ·pē·diə) Acollaborative website to help NASA engineers identify, analyze, and mitigate risks. Risk Managers Project Managers Project Engineers 8
  • 9.
  • 10.
    Astronauts use checklistsand you should too… Smart people have learned a lot of lessons learned into checklists that can help you identify risks on your project. 10
  • 11.
    TRIMS MITRE Risk Management Tool Kit Constellation Program Hazard Analysis Methodology Software The Identify Risk Development Taxonomy section is a checklist Carnegie Mellon based approach to defining risks 11
  • 12.
    Risk identification wikiin practice …3 easy steps for any engineer 1. Select the Risk Identification icon and select the relevant risk identification checklists / taxonomies. 2. Use one or more checklists / taxonomies to identify a candidate risk list. 3. Augment Riskapedia identification checklists / taxonomy based on experience (if required) 12
  • 13.
    Probability Density Relative Area Requirement 5 = Probability of Meeting Requirement 4 Prob. 1.0 Failure of System 3 12/31/98 Revised Evaluation Or 0.8 Initial Evaluation 2 0.6 Weight of A Failure Failure of Failure of Failure of Likelihood Sub-System Sub-System B Sub-System C Sub-System D 1 0.4 Failure of Failure of Failure of 0.2 Sub-System E Sub-System F Sub-System G 1 2 3 4 5 0.0 11/22/98 12/6/98 12/20/98 1/3/99 1/17/99 1/31/99 2/14/99 2/28/99 3/14/99 Impact Date CSOC Ready to Proceed PRA Techniques Risk Matrices Schedule Analysis Tools enable a team to assess risk 3. LTA 4. LTA Meeting Decision Trees 2. LTA 1. LTA Processes Participation Ground Rules Resources 1. Wrong People (too many, 1. Unclear Purpose 1. Agenda too few, no portfolio) 1. LTA Time 2. LTA Agenda 2. Scheduling - Notification 2. LTA Preparation 3. Submission/Input of Crit Read-Ahead 2. LTA Facility 3. Expectations (data v. discussion) 4. Dist. of Read-Ahead Mat. 3. Confrontational 4. Expectations (Content) 5. Actions Assignment / Outcome 3. LTA Presentation Template 4. “Hidden Agendas 5. Attendees 6. LTA Minutes / Doc of Decisions 4. AV-Laptops/Projection 5. Too Quiet / Dominant 6. PDA/cell Use 7. LTA Action Tracking AoA 5. Support 6. Convention 4. Support 5. Interactions 5. Graphics 4. Security 3. Engagement 4. Consequences 4. Density 3. Access / Registration 2. LTA Time Mgmt 3. Risks 3. Clarity (font, acronym) 2. Usability 2. Format (ppt v. /Speed 1. Articulation of 2. Issues Narrative) Values / Goals / 1. ThinkTank / CoPs / Objectives / 1. Too Much/Little Wikis / Portals Importance / 1. Activities / Sched Alignment 5. Other 8. LTA 9. LTA Content Root Cause 7. LTA Leadership 6. LTA Web-Based Tools Content Relevancy Format Fish Bones Affinity Diagram Balanced Etc… Cost/Benefit Scorecard 13 Analysis
  • 14.
    Probablistic Risk Assessment 5x5 matrix 1. Summary 2. How to 3. Examples 4. References Visual Literacy Critical Path Analysis 14
  • 15.
    Risk Assessment wikiin practice 1. Start with your risk records 2. Some risks have gone stale or need stronger characterization 3. Browse tools targeted at assessing risk and select one; perhaps the cause-and-effect diagrams 4. Hold a 2-hour facilitated team activity to re-energize the risk thinking amongst your team Unclear Work Handling Ring Brainstorm what went wrong in critical lifting process Instructions Design Time Pressure Participant  1 2 3 4 5 6 - Vague - New % Potential Add Photos 20 10 10 Ideas on - No diagrams - Holes did not line up Structured - Mixed messages from Topic Brainstorm - Poor documentation managers Explain rationale 10 25 15 - No explanation Unstructured Round 2 for each step Brainstorm Round 1 - Training was not Provide Tech Edit 45 5 …….. etc adequate for this - Background noise new operation - SMA and lift team never Provide video 50 10 60 Structured - Headset static pre-briefed Brainstorm - No time for training Provide QA & 20 10 10 entire crew Round 1 Lessons Learned Coordination Lift Crew Training Communications Between Lift Team & SMA Brainstorming Diagramming Prioritizing Action Plan Teambuilding 15
  • 16.
    Others have solved similar risks and you can learn from what they did You are not alone… Risks similar to yours have caused failures in other … but you don’t have time to programs and you go search for this can learn from information these 16
  • 17.
    Verification Risks Engineering Design Risks Quality Assurance Risks Human Factors and Nodes are focused on risk Software (recently mitigations in engineering added) disciplines 17
  • 18.
    Informed By Table of Contents Synopsis Fishbone Risk Prevention Strategies Design Engineering Risks Good Examples from ARM and IRMA Experience-based Insights Lets take a tour Noted Space System Failures through TOC… Relevant Documents References Keywords Related Topics 18
  • 19.
    Table of Contents Synopsis Fishbone RiskPrevention Strategies Design Engineering Risks Good Examples from ARM and IRMA Experience-based Insights Noted Space System Failures Relevant Documents References Keywords Related Topics 19
  • 20.
    Supporting Functions Risk University GetHelp – Risk Squad Discussion Forums 20
  • 21.
    Suggested study duration and timeline Syllabus designed to Currently 8 self study teach Level 3 and Level courses 2 risk managers the • How to create well written risk best practices of risk management statements • Integration risks • NASA risk policy • Etc… 21
  • 22.
    Lightweight Process 2.0 – Tank – Risk Facilitation PANEL URGES SHIFT IN STATION’S ORBIT Management techniques to Case Study involve your By WILLIAM J. BROAD Asynchronous Published: June 9 , 1993 The expert panel advising the White House on redesigning the space station has called for the proposed astronaut outpost to be launched into a “world orbit” where it could be reached not only by American space shuttles but also by Russian, Japanese, and group in Chinese rockets. … collaboration – “It would change things in a fundamental way,” said Dr. Bruce Murray, a planetary scientist at the California Institute of Technology. “It would say it’s not an American space station but an international one. It would say that the Cold War really is over and that we’re enthusiastic about going on to the new phase instead of acting like we’re trying to prevent time from marching on.” … solving your Today, winged spaceships soaring out of Cape Canaveral usually fly into an orbit inclined 28.5 degrees to the Equator, a path beyond the reach of the Russians. That orbit was also where the space station, proposed in 1984 amid the Cold War, was to be built brainstorm, bin, rank, piecemeal as the American shuttle fleet carried its numerous parts into space. … Now, the 16-member White House advisory panel, headed by Dr. Charles M. Vest, president of the Massachusetts Institute of Now, the 16-member White House advisory panel, headed by Technology, has endorsed a higher inclination for the American station in working papers and a draft report for President Clinton. Dr. Charles M. Vest, president of the Massachusetts Institute of Technology, problems … If the station is launched into an orbit the Russians can reach, the white paper said, the United States could “use their entire stable of has endorsed a higher inclination for the American station in working papers actions. previously developed Soviet launch vehicles – as needed.” and a draft report for President Clinton. … Cooperation with Russians could reduce costs, but the paper noted that the station’s current international partners, Japan, Canada, and Europe, “generally disagree with us about the desirability of this orbital inclination.” The drawback of the proposed path, it noted, is that shuttles flying to a higher inclination can lift less payload, up to 11,500 pounds less than the craft’s top lifting power of 55,000 pounds… Brainstorm what went wrong in critical lifting process % Potential Ideas on Structured Topic Brainstorm Unstructured Round 2 Brainstorm Round 1 Structured Brainstorm Round 1 22
  • 23.
    • Ask aquestion • Propose discussion on a topic of interest • Answer someone else’s question 23
  • 24.
    Riskapedia Designed to helpyou 1. Identify risks 2. Analyze risks 3. Mitigate risks …And get work done 24
  • 25.
    Riskapedia is currentlybased on input from some smart people… … but not as smart as everyone in this room. We want your input, your insights, and your examples to help NASA projects benefit from your knowledge! 25
  • 26.
    What you cando… … contribute! 26
  • 27.
    ITAR, SBU (not allowed in wikis) Challenge: Contribute something this week – just one thing. • Edit content in Risk ID, Risk Assessment, and Risk Mitigation notes (almost all content is editable). • Initiate or participate in discussions • Vote, rank, or comment on content usefuleness • In consultation with ESMD - become a subject matter practitioner with full editing rights to “own” a content area. 27
  • 28.
    Your benefit forcontributing 1. You will feel good about sharing your insights 2. You will have participated 1st hand in the wiki movement 3. Some NASA engineer, somewhere, will be grateful as they incorporate your advice into solving their problem. 28
  • 29.