The integrated master plan and integrated master schedule


Published on

The Integrated Master Plan (IMP) and Integrated Master Schedule( (IMS) provide a strategy for the incremental delivery of program outcomes through increasing maturity assessments with Measures of Effectiveness, Measures of Performance, Technical Performance Measures, and Key Performance Parameters.
These assessment assure the needed capabilities of the project are met at each assessment point to confirm physical percent complete as planned in the Integrated Master Plan

Published in: Technology, Business

The integrated master plan and integrated master schedule

  1. 1. The  Integrated  Master  Plan  And   Integrated  Master  Schedule   The  Integrated  Master  Plan  and  Integrated  Master  Schedule  are  one  of  the   six  elements  of  a  Credible  Performance  Measurement  Baseline  (PMB).   The  PMB  is  the  source  of  data  for  the  Program  Manager.  Some  of  this  data  is   used  in  the  Integrated  Program  Measurement  Report  (IPMR).  Some  is  used   in  the  assessment  of  the  Program’s  risk.  Some  define  the  deliverables  and   their  Technical  Performance  Measures.     1   5 V8.5  
  2. 2. The  IMP  tells  us  where  is  the  program  going?   The  Plan  describes  where  we  are  going,  the  various  paths  we  can  take  to  reach  our   desLnaLon,  and  the  progress  or  performance  assessment  points  along  the  way  to   assure  we  are  on  the  right  path.   These  assessment  points  measures  the  “maturity”  of  the  product  or  service  against   the  planned  maturity.  This  is  the  only  real  measure  of  progress  –  not  the  passage   of  Lme  or  consumpLon  of  money.   The  Integrated  Master  Plan  (IMP)  Is  A  Strategy  For   The  Successful  Comple=on  Of  The  Project   2   5.0  Start  with  the  IMP  
  3. 3. 3   5.0  Start  with  the  IMP  
  4. 4. Quick  View  to  IMP/IMS   !  VerLcal  traceability  defines  the  increasing  maturity  of   key  deliverables   !  Horizontal  traceability  defines  the  work  acLviLes   needed  to  produce  this  increasing  maturity   !  Both  are  needed,  but  the  verLcal  traceability  is  the   starLng  point   !  Program  Events,  Significant  Accomplishments,  and   Accomplishment  Criteria  must  be  defined  before  the   horizontal  work  acLviLes  can  be  idenLfied   !  For  all  IMP  elements,  Key  Risks  must  be  idenLfied  and   assigned  from  Day  One,  even  without  miLgaLons   4   5.0  Start  with  the  IMP  
  5. 5. The  IMP/IMS  is  Needed  on     Both  Sides  of  the  Contract   !  Ver%cal  traceability  defines  the  increasing  maturity  of   the  program’s  deliverables  measures  in  EffecLveness   (MoE)  and  Performance  (MoP)  for  the  Government.   !  Horizontal  traceability  defines  the  progress  to  plan  for   MoE’s  and  MoP’s  with  tangible  evidence  for  both  the   Government  and  the  Contractor.   !  Ver%cal  traceability  provides  the  Government  with   insight  into  the  progress  of  the  MoE’s  and  MoP’s  and   Technical  Performance  Measures.   !  Horizontal  traceability  provides  insight  into  Cost  and   Schedule  performance  for  the  Contractor,  reportable   through  the  IPMR  to  the  Government.   5   5.0  Start  with  the  IMP  
  6. 6. MisconcepLons  of     the  IMP  /  IMS   Why  we  don’t  need/want  an  IMP/IMS     !  Only  required  for  ACAT  I  programs   !  Too  big  and  burdensome  for  our  small  dollar  value  program   !  Contractor  spends  B&P  and  program  budget  generaLng  and   maintaining  the  IMP,  without  measurable  benefit   !  Doesn’t  apply  on  a  services  contractors   !  Management  tool,  not  a  technical  tool   !  Doesn’t  apply  to  technology  programs   !  Doesn’t  apply  to  R&D  efforts   !  Doesn’t  apply  to  the  government   !  Help  me  get  a  waiver  so  I  don’t  have  to  use  an  IMP/IMS   6   5.0  Start  with  the  IMP  
  7. 7. Aeributes  of  IMP   !  Traceability   –  Expands  and  complies  with  the  SOO,  Performance   Requirements,  CWBS,  and  CSOW   –  Based  on  the  customers  WBS   –  Is  the  basis  of  the  IMS,  cost  reports,  and  award  fees   !  Implements  a  measurable  and  trackable  program   –  Accomplishes  integrated  product  development   –  Integrates  the  funcLonal  acLviLes  of  the  program   –  Incorporates  funcLonal,  lower  level  and  S/C  IMPs   !  Provides  for  evaluaLon  of  Program  Maturity   –  Provides  insight  into  the  overall  effort   –  Level  of  detail  is  consistent  with  risk  and  complexity  per  §L   –  Decomposes  events  into  a  logical  series  of  accomplishments   –  Measurable  criteria  demonstrate  compleLon  /  quality  of   accomplishments   7   5.0  Start  with  the  IMP  
  8. 8. Aeributes  of  the  IMS   !  Integrated,  networked,  mulL-­‐layered  schedule  of   efforts  required  to  achieve  each  IMP   accomplishment   –  Detailed  tasks  and  work  to  be  completed   –  Calendar  schedule  shows  work  compleLon  dates   –  Network  schedule  shows  interrelaLonships  and   criLcal  path   –  Expanded  granularity,  frequency,  and  depth  of  risk   areas   !  Resource  loading   !  Correlates  IMS  work  with  IMP  events   8   5.0  Start  with  the  IMP  
  9. 9. The  Importance  of  the  IMP   !  The  IMP/IMS  is  the  single  most  important  document  to   a  program’s  success   –  It  clearly  demonstrates  the  providers  understanding  of  the   program  requirements  and  the  soundness  of  the  approach   a  represented  by  the  plan   !  The  program  uses  the  IMP/IMS  to  provide:   –  Up  Front  Planning  and  commitment  from  all  parLcipants   –  A  balanced  design  discipline  with  risk  miLgaLon  acLviLes   –  Integrated  requirements  including  producLon  and  support   –  Management  with  an  incremental  verificaLon  for   informed  program  decisions   9   5.0  Start  with  the  IMP  
  10. 10. Just  A  Reminder,  before  moving  on   10   Page  47,  Defense  AcquisiLon  Guide,  January  10,  2012   Page  317,  Defense  AcquisiLon  Guide,  January  10,  2012   5.0  Start  with  the  IMP  
  11. 11. Our  Goal  is  simple  …     How  can  we  recognize  the  Reality  of  the  Program’s   current  status  and  its  future  performance?   The  top  spins  conLnuous  while  in  a  dream  –  stops   spinning  in  the  real  world  –  Cobb’s  totem,  Incep=on   11   5.0  Start  with  the  IMP  
  12. 12. 12  
  13. 13. Principles  of  Building  a  Credible   Integrated  Master  Plan   Building  the  IMP  is  a  Systems  Engineering  ac<vity.  The  Integrated  Master  Plan  is   the  Program  Architecture  in  the  same  way  the  hardware  and  soBware  are  the   Product  Architecture.   Poor,  a  weak,  or  unstructured  Programma<c  Architecture  reduces  visibility  to   the  Product  Architecture’s  performance  measures  of  cost  and  schedule   connected  with  Technical  Performance  Measures.     1   6 V8.5  
  14. 14. Quick  View  of  Building  the  IMP   !  Start  with  each  Program  Event  and  define  the   Significant  Accomplishments  their  entry  and   exit  criteria  to  assess  the  needed  maturity  of   the  key  deliverables   !  Arrange  the  Significant  Accomplishments  in   the  proper  dependency  order     !  Segregate  these  Significant  Accomplishments   into  swim  lanes  for  IPTs   !  Define  the  dependencies  between  each  SA   2   6.0  Build  IMP  
  15. 15. A  Cri<cal  Understanding  of  the  IMP   3   The  IMP  defines  the  connec<ons  between  the  Product  maturity  –  Ver<cal  –  and  the   implementa<on  of  this  Product  maturity  through  the  Func<onal  ac<vi<es  –  the  Horizontal     6.0  Build  IMP  
  16. 16. Benefits  of  this  Formality   4   Objec&ve   Implementa&on   Event  Driven  Plan  versus  Schedule   Driven  Plan  is  based  on  comple<on  of   tasks  not  passage  of  <me   Separate  the  plan  (IMP)  from  the   schedule  (IMS)  but  link  elements  with   numbering  system   Condensed,  easy  to  read  “Plan”  showing   the  “Events”  rather  than  the  work  effort   Indentured,  outline  format  –  not  text   Pre-­‐defined  entry  and  exit  criteria  for   major  Program  Events   Significant  Accomplishments  for  each   key  Program  Event   Objec<ve  measures  of  progress  and   comple<on  for  each  Accomplishment   Pre-­‐defined  Accomplishment  Criteria   (AC)  for  each  Significant   Accomplishment  (SA)   Stable,  contracture  plan,  flexible  enough   to  portray  program  status   IMP  part  of  the  contract,  IMS  is  a  data   item   Capture  essence  of  the  func<onal   progress  without  manda<ng  a  par<cular   process  for  performing  the  work   Split  IMP  into  Product  and  Process   6.0  Build  IMP  
  17. 17. Risk  Management   Building  the  IMP  Starts  at  the  RFP   5   SOW   SOO   ConOps   WBS   Techncial  and  Opera<onal   Requirements   CWBS  &   CWBS  Dic<onary   Integrated  Master  Plan   (IMP)   Integrated  Master  Schedule   (IMS)     Earned  Value  Management   System   Objec<ve  Status  and  Essen<al  Views  to  support  the  proac<ve  management   processes  needed  to  keep  the  program  GREEN   Performance  Measurement  Baseline   Measures  of   Effec<veness   Measures  of   Performance   Measures  of   Progress   JROC     Key  Performance  Parameters   Program    Specific   Key  Performance  Parameters   Technical  Performance   Measures   1.0  Introduc<on  
  18. 18. The  IMP  /  IMS  Structure   6   IMS IMP Describes how program capabilities will be delivered and how these capabilities will be recognized as ready for delivery Supplemental Schedules Work Packages and Tasks Criteria Accomplishment Events or Milestones 1   6.0  Build  IMP  
  19. 19. The  IMP/IMS  provides  Horizontal  and   Ver<cal  Traceability  of     progress  to  plan   !  Ver<cal  traceability  AC  "  SA  "  PE   !  Horizontal  traceability  WP  "  WP  "  AC   7   Program Events Define the maturity of a Capability at a point in time. Significant Accomplishments Represent requirements that enable Capabilities. Accomplishment Criteria Exit Criteria for the Work Packages that fulfill Requirements. Work   Package   Work   Package   Work   Package   Work   Package   Work   Package   Work   Package   Work   package   6.0  Build  IMP  
  20. 20. The  IMP’s  role  during  Execu<on   8   Program ExecutionPMB for IBRProposal SubmittalDRFP & RFP Performance Measurement Baseline Tasks (T) BOE % Complete Statement of Work Program Deliverables IMP Accomplishments (A) Criteria (C) EVMS Events (E) Budget Spreads by CA & WPCAIV Capabilities Based Requirements X BCWS = Probabilistic Risk Analysis = Time keeping and ODC = Technical Performance Measure BCWP ACWP Cost & Schedule Risk Model BCWS Decreasing technical and programmatic risk using Risk Management Methods IMS Physical % Complete Continuity and consistency from DRFP through Program Execution WBS 6.0  Build  IMP  
  21. 21. 1st,  Principle  –     IMP  Building  is  a  Full  Contact  Sport   9   6.0  Build  IMP  
  22. 22. Our  First  Approach  to  the     IMP  /  IMS  Paradigm   !  The  1st  approach  defines  Program  Events  (PE),   Significant  Accomplishments  (SA),  and   Accomplishment  Criteria  (AC),  derived  from  the   Work  Breakdown  Structure  or  the  SOW   !  This  1st  approach  can  be  done  in  6  easy  steps   10   6.0  Build  IMP   1.  Iden<fy  the  Program  Events  (PE)   (as  the  ACQ  Guide  tells  us)   2.  Iden<fy  the  Significant   Accomplishments  (SA)  from  the   WBS  deliverables     3.  Iden<fy  the  Accomplishment   Criteria  (AC)  needed  to  produce   these  deliverables   4.  Iden<fy  the  Work  Packages  for   each  Accomplishment  Criteria   5.  Sequence  the  Work  Packages   6.  Assemble  the  IMP/IMS  
  23. 23. This  approach  doesn’t  give  us   visibility  into  what  “done”  looks  like   !  We  must  measure  increasing  product  maturity  in   units  meaningful  to  the  decision  makers   !  We  must  see  the  risks  before  they  arrive  so  we   can  take  correc<ve  ac<on   11   6.0  Build  IMP  
  24. 24. First  Look  at  a  Significant   Accomplishment  (SA)   !  SAs  are  interim  and  final  steps  to  define,  design,  develop,   verify,  produce,  and  deploy  the  product  or  system.     !  SAs  must  occur  in  a  manner  that  ensures  a  logical  path  is   maintained  throughout  the  development  effort.     !  SAs  are  event  related  and  not  just  <me  coincidental.     !  SAs  should  have  one  or  more  of  the  following   characteris<cs:     –  Consists  of  a  discrete  step  in  the  process  of  planned   development  that  must  be  complete  prior  to  an  event   –  Produces  a  desired  result  at  a  specified  event  that  indicates  a   level  of  design  maturity  (or  progress)  directly  related  to  each   product  and  process   –  Defines  interrela<onships,  interdependencies,  or  “hand-­‐off”   points  of  different  func<onal  disciplines  applied  to  the  program     12   6.0  Build  IMP  
  25. 25. First  look  at  an  Accomplishment   Criteria  (AC)   !  ACs  are  defini<ve  measures  directly  suppor<ng  successful   comple<on  of  a  significant  accomplishment.     !  ACs  show  objec<ve  evidence  of  work  progress  (maturity  of   a  product);  i.e.,  be  seen,  read,  demonstrated,  or  quan<fied.   These  results  are  usually  incorporated  into  a  report  or   document  as  evidence  of  accomplishment.     !  ACs  are  prerequisites  for  comple<on  of  an  SA  (i.e.,  exit   criteria).     !  The  ques<ons  that  need  to  be  repeatedly  asked  when   developing  ACs  are:   –  How  do  I  know  when  an  accomplishment  has  been  completed?   –  Is  the  criteria  directly  related  to  the  accomplishment?     –  Is  it  proof?     –  What  is  the  work  product?”   13   6.0  Build  IMP  
  26. 26. The  IMP  speaks  to  Measures  of  Effec<veness   (MoE)  and  Measures  of  Performance  (MoP)   14   6.0  Build  IMP   !  This  is  where  TPMs     are  connected  with   the  MoE’s  and   MoP’s   !  For  each   deliverable  from   the  program,  all   the  “measures”   must  be  defined  in   units  meaningful   to  the  decision   makers.   !  Here’s  some  “real”   examples.   1.  Provide  Precision   Approach  for  a  200   FT/0.5  NM  DH   2.  Provide  bearing  and   range  to  AC  plaporm   3.  Provide  AC   surveillance  to  GRND   plaporm     Measures  of   Effec<veness  (MoE)   1.  Net  Ready   2.  Guidance  Quality   3.  Land  Interoperability   4.  Manpower   5.  Availability   JROC  Key  Performance   Parameters  (KPP)   1.  Net  Ready   ! IPv4/6  compliance   ! 1Gb  Ethernet   2.  Guidance  quality   ! Accuracy  threshold  p70   @  6M   !   Integrity  threshold  4M   @  10-­‐6  /approach   3.  Land  interoperability   ! Processing  capability   meets  LB  growth  matrix   4.  Manpower   ! MTBC  >1000  hrs   ! MCM  <  2  hrs   5.  Availability   ! Clear  threshold  >99%   ! Jam  threshold  >90%   Measures  of  Performance   (MoP)   1.  Net  Ready   ! Standard  message  packets   2.  Guidance  Quality   ! Mul<path  alloca<on  budget   ! Mul<path  bias  protec<on   3.  Land  Interoperability   ! MOSA  compliant   ! Civil  compliant   4.  Manpower   ! Opera<ng  elapsed  <me   meters   ! Standby  elapsed  <me   indicators   5.  Availability   ! Phase  center  varia<ons     Technical  Performance   Measures  (TPM)   Mission  Capabili<es  and  Opera<onal  Need   Technical  Insight  –  Risk  adjusted  performance  to  plan    
  27. 27. The  1st  Problem  with  the  Ini<al   IMP/IMS  Paradigm   !  There  is  no  single  source  of  guidance  for   construc<ng  a  credible  IMP  and  IMS   – A  DOD  Guidebook,  but  s<ll  in  Version  0.9   – A  few  DOD  service  pamphlets   – A  commercial  guidebook   – Many  contractor  guidelines   !  No  defini<ve  guidance  from  DoD  on  what   makes  a  good  IMP   !  No  defini<ve  DID  requiring  an  IMP  on  specific   programs   15   6.0  Build  IMP  
  28. 28. The  IMP  is  a  good  start,  but  we’re   really  aBer  the  IMP  Narra<ves   !  The  IMP  Narra<ves  start  with  the  Statement   of  Objec<ves  (SOO)  or  Concept  of  Opera<ons   (ConOps)   – These  iden<fy  the  top  level  program  objec<ves   – They  define  the  big  picture  and  provide  pre-­‐award   trade  space   – They  provide  the  framework  for  the  Contractor  to   develop  the  proposal  through  the  IMP   !  With  the  Government’s  Execu<ve  summary,   provides  the  contractor  an  understanding  of   what  is  needed  and  what  is  important   16   6.0  Build  IMP  
  29. 29. The  IMP  Process  Narra<ve   The  IMP  Process  Narra<ve  describes  how  the   technical  and  business  elements  of  the  program   will  be  conducted,  monitored,  and  controlled.     Narra<ves  provide  the  customer  visibility  into   the  contractor’s  key  func<onal  processes  and   procedures,  the  rela<onship  of  these  processes   and  procedures,  and  an  overview  of  the  effort   required  to  implement  them.   17   6.0  Build  IMP  
  30. 30. IMP  Narra<ve  for  PDR   Program   Event   Event  Descrip&on   PE  Maturity  Assessment   shown  in  SAs   PDR   PDR  establishes  the  “design-­‐  to”  allocated   baseline  to  the  subsystem  level,  assures  this   design  meets  the  func<onal  baseline,  assures   system  requirements  have  been  properly   allocated  to  the  proper  subsystem.   PDR  establishes  the  feasibility  of  the  design   approach  to  meet  the  technical  requirements  and   provide  acceptable  interface  rela<onships   between  the  hardware  and  other  interfacing   items.     Any  changes  to  the  requirements  that  have   occurred  since  the  System  Requirements  Review   (SRR2)  will  be  verified  at  the  PDR.     PDR  assures  the  design  is  verifiable,  does  not  pose   major  IMS  or  Cost  risk,  and  is  mature  enough  to   advance  to  the  detailed  design  phase  –  CDR   !  Subsystem  level   opera<onal  concepts   defined   !  System  level  interfaces   baselined   !  Supportability  plans   established   !  SoBware  requirements   finalized   !  Subsystem  requirements   finalized  &  allocated   !  System  verifica<on,   valida<on  &  cer<fica<on   plans  updated   !  PDR  subsystem  design   completed   18   6.0  Build  IMP  
  31. 31. What  the  IMP  Narra<ve  Tells  Us?   !  States  the  objec<ve  of  the  processes  used  to  build  the   products   !  Provides  the  governing  documents,  compliance,  and   reference  for  the  process  ac<vi<es   !  Explains  the  process  approach   –  Portrays  the  key  ac<vi<es  of  the  approach   –  Illustrates  the  processes  tailored  for  the  specific  program   19   Inputs   Ac<vity   3   Ac<vity   5   Ac<vity   4   Tools   Ac<vity   1   Ac<vity   2   Outputs   Metrics   6.0  Build  IMP  
  32. 32. Some  Guidance     20   6.0  Build  IMP  
  33. 33. The  Defense  Acquisi<on  Guide   (DAG)  Tells  Us  About  the  IMP   21   6.0  Build  IMP  
  34. 34. The  DAG  Also  Tells  Us   22   6.0  Build  IMP  
  35. 35. 5000.01  Part  2.B.3  Acquisi<on  Strategies,   Exit  Criteria,  and  Risk  Management   “Event  driven  acquisi<on  strategies  and  program  plans   must  be  based  on  rigorous,  objec<ve  assessments  of  a   program’s  status  and  the  plans  for  managing  risk  during   the  next  phase  and  the  remainder  of  the  program.     The  acquisi<on  strategy  and  associated  contrac<ng   ac<vi<es  must  explicitly  link  milestone  decision  reviews   to  events  and  demonstrated  accomplishments  in   development,  tes<ng,  and  ini<al  produc<on.     The  acquisi<on  strategy  must  reflect  the   interrela<onships  and  schedule  of  acquisi<on  phases  and   events  based  on  logical  sequence  of  demonstrated   accomplishments  not  on  fiscal  or  calendar  expediency.”   23   6.0  Build  IMP  
  36. 36. What  makes  a  good     Program  Event  (PE)?   !  Events  are  the  conclusion  of  an  interval  of  major   program  accomplishments  (SA)  with  their  criteria  (AC)     !  IMP  events  represent  key  decision  transi<on  points   between  major  ac<vi<es  distributed  over  the  contract   period   –  The  IMP  is  a  Mini  Authoriza;on’s  to  Proceed  (ATP)  to  the   next  Program  Event  (PE)   !  Some  guidance  for  establishing  program/product   events:   –  Customer  Given  Events.     –  Key  Decisions  Needed   –  Risk  Mi<ga<on  Event   –  DOD  Systems  Engineering  Technology  Review  (SETR)   Guidance   24   6.0  Build  IMP  
  37. 37. What  makes  a  good   Significant  Accomplishment  (SA)?   !  IPT  can  manage  it  at  a  working  level   !  Shows  comple<on  and  result  s  of  discrete  steps  in  the   development  process   !  Indicates  maturity  of  the  product  through  MoEs  and  MoPs   !  Its  “significance”  measures  program  event  status   !  Relevant  and  logically  linked  to  the  proper  PE   –  Just  because  the  work  occurs  during  the  <me-­‐frame  for  PE  A   doesn’t  mean  it’s  logically  linked  to  PE  A   !  Uses  consistent  language,  style,  format  through  verb   dic<onary,  for  example:   –  Segment  build  4  detailed  design  completed   –  Analysis  of  structural  integrity  completed   –  Structural  integrity  verified   25   6.0  Build  IMP  
  38. 38. IMP  Significant  Accomplishments   !  SAs  are  NOT  just  a  list  of  “things”  to  do  before   the  Program  Event  (PE)   !  They  are  sequenced  accomplishments,  each  of   which  leads  to  the  PE,  e.g.  Cri<cal  Design  Review   (CDR)   –  SA  #  1  =  CDR  mee<ng  conducted   –  SA  #  2  =  CDR  ac<on  item  work-­‐off  plan  established   –  SA  #  3  =  85%  drawings  completed   –  SA  #  4  =  CDR  CDRLs  delivered   –  SA  #  5  =  Development  environment  opera<onal   –  SA  #  6  =  Cri<cal  methods  analyses  completed   –  SA  #  7  =  RVTM  approved   26   6.0  Build  IMP  
  39. 39. What  makes  a  good   Accomplishment  Criteria  (AC)?   !  Provides  objec<ve,  measureable  and,  explicit   proof  of  comple<on  and  closure  of  the  work   ac<vi<es   !  Defines  condi<ons  for  closing  the  Significant   Accomplishment  (SA)   !  Answers  the  ques<on  how  do  we  know  when   a  Significant  Accomplishment  has  been   completed?   27   6.0  Build  IMP  
  40. 40. !  Not  Significant   –  Too  small  to  significantly  contribute  to  successful  event  comple<on   –  Would  lead  to  trivial  tasks  (e.g.,  1  day  dura<on)   !  Ambiguous   –  Read  can’t  tell  what  Done  looks  like   !  Wrong  verb   –  Uses  a  verb  that’s  not  on  the  list  (Dic<onary)   –  Uses  a  listed  verb  incorrectly   –  Doesn’t  have  a  verb  at  all   !  Not  measurable   –  Can’t  tell  when  we’re  done   !  Too  Many  SAs  or  ACs   –  Confuses  the  reader  and  confuses  the  execu<on  process  of  the   program   –  Dilutes  the  MoE  and  MoP   –  Reduces  visibility  into  increasing  maturity   28   What  makes  a  Not  So  Good     SA  or  AC?   6.0  Build  IMP  
  41. 41. !  Program  Event  (PE)   –  A  PE  assess  the  readiness  or  comple<on  as  a  measure   of  progress   –  First  Flight  Complete   !  Significant  Accomplishment  (SA)   –  The  desired  result(s)  prior  to  or  at  comple<on  of  an   event  demonstrate  the  level  of  the  program’s   progress   –  Flight  Test  Readiness  Review  Complete   !  Accomplishment  Criteria  (AC)   –  Defini<ve  evidence  (measures  or  indicators)  that   verify  a  specific  accomplishment  has  been  completed   –  SEEK  EAGLE  Flight  Clearance  Obtained   29   F-­‐22  Example   6.0  Build  IMP  
  42. 42. !  PE’s  are  easy,  they  are  in  the  SETR  and  Integrated   Logis<cs  Lifecycle  Management  System   !  The  SA’s  can  be  defined  from  the  program’s   deliverables  –  these  may  not  be  obvious  but  can   be  discovered  with  a  Product  Development   Kaizen  process  (more  later)   !  It’s  the  AC’s  that  are  hard  part  –  the  ACs  must   represent  the  exit  criteria  for  the  series  of  Work   Packages  that  do  the  work  to  produce  the   product   30   Now  the  Hard  Part   6.0  Build  IMP  
  43. 43. 6  Steps  to  IMP  Development   Let’s  start  to  build  the  IMP.     This  step-­‐by-­‐step  process  needs  to  be  followed  carefully.   The  IMP  is  constructed  one  Program  Event  at  a  Cme  –  LeE  to  Right  in  Cme.     To  do  otherwise  allows  confusion  and  disconnecCon  between  Program  Events  to   occur  and  dilutes  our  focus  on  defining  what  Done  looks  like  for  each  Program   Event.       1   7 V8.5  
  44. 44. 2   7.0  6  Steps  
  45. 45. Quick  View  of  Step-­‐By-­‐Step  IMP   IdenCfy  Program  Events   IdenCfy  Significant  Accomplishments   IdenCfy  Accomplishment  Criteria   IdenCfy  Work  Packages  needed  to  complete   the  Accomplishment  Criteria   Sequence  the  Work  Packages  (WP),  Planning   Packages  (PP),  Summary  Level  Planning   Packages  (SLPP)    in  a  logical  network.   Adjust  the  sequence  of  WPs,  PPs,  &  SLPPs  to   miCgate  major  risks.   3   7.0  6  Steps   1 2 3 4 5 6
  46. 46. IdenCfy  the  Program  Events   Actors   Processes   Outcomes   Systems  Engineer   Define  the  process  flow  for   product  producCon  from   contract  award  to  end  of   contract   !  Confirm  Program  Events  represent  the   logical  process  flow  for  program   maturity   Program  Manager   Confirm  customer  is  willing  to   accept  the  process  flows   developed  by  the  IMP   !  Engage  with  contracts  and  customer   for  PE  definiCon   Project  Engineer   IdenCfy  interdependencies   between  program  event  work   streams   !  IdenCfy  Value  Stream  components  at   the  PE  level  before  flowing  them  down   to  the  SA  level   IMP/IMS  Architect   Capture  Program  Event   contents  for  each  IPT  or  work   stream   !  Establish  foundaCon  for  a  structure  to   support  the  descripCon  of  the   increasing  mature  as  well  as  the  flow   to  needed  work.   Copyright  ©  2012,  Glen  B.  Alleman,  Niwot  Ridge,  LLC   4   1 7.0  6  Steps  
  47. 47. Outcomes  of  Step   !  Confirm  the  end  to  end  descripCon  of  the   increasing  maturity  of  the  program’s   deliverables   !  Establish  of  RFP  or  Contract  target  dates  for   each  Event.     !  Socialize  the  language  of  speaking  in  “Events”   rather  than  Cme  and  efforts   5   1 7.0  6  Steps  
  48. 48. Events  Define  the  Assessment     of  the  Program’s  Maturity   6   ! Program  Events  are  maturity   assessment  points  in  the  program   ! They  define  what  levels  of  maturity   for  the  products  and  services  are   needed  before  proceeding  to  the  next   maturity  assessment  point   ! The  entry  criteria  for  each  Event   defines  the  units  of  measure  for  the   successful  compleCon  of  the  Event   ! The  example  below  is  typical  of  the   purpose  of  a  Program  Event   The  Cri+cal  Design  Review  (CDR)  is  a  mul+-­‐disciplined  product  and  process  assessment  to  ensure   that  the  system  under  review  can  proceed  into  system  fabrica+on,  demonstra+on,  and  test,  and   can  meet  the  stated  performance  requirements  within  cost  (program  budget),  schedule  (program   schedule),  risk,  and  other  system  constraints.     1 7.0  6  Steps  
  49. 49. IdenCfy  the  Significant  Accomplishments   (SA)  for  Each  Program  Event  (PE)   7   Actors   Processes   Outcomes   System  Engineer   IdenCfy  Integrated  Product  Teams   (IPT)  responsible  for  the  SA’s     !  Define  the  boundaries  of  these   programmaCc  interfaces   !  Define  technical  and  process  risk   categories  and  their  bounds   Technical  Lead   Confirm  the  sequence  of  SA’s  has   the  proper  dependency   relaConships   !  Define  the  product  development   flow  process  improves  maturity   !  Define  technical  risk  drivers   Project  Engineer   Confirm  logic  of  SA’s  for  project   sequence  integrity   !  Define  the  program  flows   improves  maturity     Control  Account   Manager   Validate  SA  outcomes  in  support  of   PE  entry  condiCons   !  Confirm  budget  and  resources   adequate  for  defined  work  effort   IMP/IMS  Architect   Assure  the  assessment  points   provide  a  logical  flow  of  maturity  at   the  proper  intervals  for  the   program   !  Maintain  the  integrity  of  the  IMP,   WBS,  and  IMS   2 7.0  6  Steps  
  50. 50. Outcomes  of  Step   !  The  Significant  Accomplishments  are  the   “road  map”  to  the  increasing  maturity  of  the   program   !  The  “Value  Stream  Map”  resulCng  from  the   flow  of  SA’s  describes  how  the  products  or   services  move  through  the  maturaCon  process   while  reducing  risk   !  The  SA  map  is  the  path  to  “done”     8   2 7.0  6  Steps  
  51. 51. SAs  define  the  entry  criteria  for   each  Program  Event   9   Preliminary  Design  Review  Complete   3 7.0  6  Steps  
  52. 52. IdenCfy  Accomplishment  Criteria  (AC)  for  each   Significant  Accomplishment  (SA)   Actors   Processes   Outcomes   CAM   Define  and  sequence  the  contents   of  each  Work  Package  and  select   the  EV  criteria  for  each  Task   needed  to  roll  up  the  BCWP   measurement   !  Establish  ownership  for  the   content  of  each  Work  Package   and  the  Exit  Criteria  –  the   Accomplishment  Criteria  (AC)   Project  Engineer   IdenCfy  the  logical  process  flow  of   the  Work  Package  to  assure  the   least  effort,  maximum  value  and   lowest  risk  path  to  the  Program   Event   !  Establish  ownership  for  the   process  flow  of  the  product  or   service   Technical  Lead   Assure  all  technical  processes  are   covered  in  each  Work  Package   !  Establish  ownership  for  the   technical  outcome  of  each  Work   Package   IMP/IMS  Architect   Confirm  the  process  flow  of  the  ACs   can  follow  the  DID  81650   structuring  and  Risk  Assessment   processes   !  Guide  the  development  of   outcomes  for  each  Work  Package   to  assure  increasing  maturity  of   the  program   10   3 7.0  6  Steps  
  53. 53. Outcomes  of  Step   !  The  definiCon  of  “done”  emerges  in  the  form   of  deliverables  rather  than  measures  of  cost   and  passage  of  Cme.   !  At  each  Program  Event,  the  increasing   maturity  of  the  deliverables  is  defined  through   the  Measures  of  EffecCveness  (MoE)  and   Measures  of  Performance  (MoP)   11   3 7.0  6  Steps  
  54. 54. ACs  are  higher  fidelity   descripCons  of  “Done”  than  SAs   12  Cri9cal  Design  Review  Complete   4 7.0  6  Steps  
  55. 55. IdenCfy  the  Work  for  Each  Accomplishment   Criteria  in  Work  Packages   Actors   Processes   Outcomes   Control  Account   Manager   IdenCfy  or  confirm  the  work   acCviCes  in  the  Work  Package   represent  the  allocated  work   !  Define  bounded  work  effort   defined  “inside”  each  Work   Package   Technical  Lead   Confirm  this  work  covers  the  SOW   and  CDRLs   !  Define  all  work  effort  for  100%   compleCon  of  deliverable  visible  in   a  single  locaCon  –  the  Work   Package   !  Confirm  risk  drivers  and  duraCon   variances   IMP/IMS  Architect   Assist  in  the  sequencing  the  work   efforts  in  a  logical  manner   !  Develop  foundaCon  of  the   maturity  flow  starCng  to  emerge   from  the  contents  of  the  Work   Packages   Earned  Value   Analyst   Assign  iniCal  BCWS  from  BOE  to   Work  Package   !  ConfirmaCon  of  work  effort   against  BOEs   !  Define  EVT  for  measures  progress   to  plan   13   4 7.0  6  Steps  
  56. 56. Outcomes  of  Step   !  The  work  idenCfied  that  produces  a   measurable  outcome.   !  This  work  defined  in  each  Work  Package   !  The  Accomplishment  Criteria  (AC)  state   explicitly  what  “done”  looks  like  for  this  effort   !  With  “done”  stated,  measures  of  Performance   and  measures  of  EffecCveness  can  be  defined   14   4 7.0  6  Steps  
  57. 57. Work  is  done  in  “packages”  that   produce  measureable  outcomes   15   Launch  Readiness  Review  Complete   5 7.0  6  Steps  
  58. 58. Sequence  Work  Packages  (ACs)  for  each   Significant  Accomplishment  (SA)   16   Actors   Processes   Outcomes   Control  Account   Manager   Define  the  order  of  the  Work   Packages  needed  to  meet  the   Significant  Accomplishments  for   each  Program  Event   !  Define  the  process  flow  of  work   and  the  resulCng   accomplishments.   !  Assure  value  is  being  produced  at   each  SA  and  the  AC’s  that  drive   them   IMP/IMS  Architect   Assure  that  the  sequence  of  Work   Packages  adheres  to  the  guidance   provided  by  DCMA  and  the  EVMS   System  descripCon   !  Begin  the  structuring  of  the  IMS   for  compliance  and  loading  into   the  cost  system   Program  Controls   Staff   Baseline  the  sequence  of  Work   Packages  using  Earned  Value   Techniques  (EVT)  with  measures  of   Physical  Percent  Complete   !  Develop  insight  to  progress  to   plan  with  measures  of  physical   progress  for  each  Work  Packages   (EVT)   5 7.0  6  Steps  
  59. 59. Outcomes  of  Step   !  Work  Packages  parCCon  work  efforts  into   “bounded”  scope   !  Interdependencies  constrained  to  Work   Package  boundaries  prevents  “spaghek  code”   style  schedule  flow   !  Visibility  of  the  Increasing  Flow  of  Maturity   starCng  to  emerge  from  the  flow  of   Accomplishment  Criteria  (AC)   17   5 7.0  6  Steps  
  60. 60. Sequence  Work  Packages  (AC’s)  into   an  IMS  for  each  Program  Event   18   6 7.0  6  Steps  
  61. 61. Assemble  Final  IMP/IMS   Actors   Processes   Outcomes   IMP/IMS  Architect   StarCng  with  the  AC’s  under  each   SA’s  connect  Work  Packages  in  the   proper  order  for  each  Program   Event   !  Establish  the  Performance   Measurement  Baseline   framework.   !  IdenCfy  MoE  and  MoP  points  in   the  IMP   Program  Manager   Confirm  the  work  efforts  represent   the  commiled  acCviCes  for  the   contract   !  Review  and  approval  of  the  IMS  –   ready  for  baseline.   !  Review  and  approve  risk  drivers   and  duraCon  variance  models   Project  Engineer   Assess  the  product  development   flow  for  opCmizaCons   !  Review  and  approval  of  the  IMS  –   ready  for  baseline.   !  IdenCfy  risk  drivers  and  their   miCgaCons   Systems  Engineer   Confirm  the  work  process  flows   result  in  the  proper  products  being   built  in  the  right  order   !  Confirm  risk  drivers  and  duraCon   variances.   !  Review  and  approval  of  the  IMS  –   ready  for  baseline   19   6 7.0  6  Steps  
  62. 62. Outcomes  of  Step   !  Both  the  maturity  assessment  criteria  and  the   work  needed  to  reach  that  level  of  maturity  are   described  in  a  single  locaCon   !  Risks  are  integrated  with  the  IMP  and  IMS  at   their  appropriate  levels   –  Risks  to  EffecCveness  –  risk  to  JROC  KPPs   –  Risks  to  Performance  –  risk  to  program  KPPs  and   TPMs   !  Leading  and  Lagging  indicator  data  provide   through  each  measure  to  forecast  future   performance     20   6 7.0  6  Steps  
  63. 63. The  Previous  6  Steps  Result  In  A   Credible  IMP/IMS   21   !  The  IMP  is  the  “Outer  Mold   Line”,  the  Framework,  the   “Going  Forward”  Strategy  for   the  Program.   !  The  IMP  describes  the  path   to  increasing  maturity  and   the  Events  measuring  that   maturity.   !  The  IMP  tells  us  “How”  the   program  will  flow  with  the   least  risk,  the  maximum   value,  and  the  clearest   visibility  to  progress.   !  The  IMS  tells  us  what  work  is   needed  to  produce  the   product  or  service  at  the   Work  Package  level.   Our  Plan  Tells  Us  “How”  We   are  Going  to  Proceed   The  Schedule  Tells  Us   “What”  Work  is  Needed  to   Proceed   7.0  6  Steps  
  64. 64. Significant  Accomplishments  for  an  actual   Program  Event   Flight  Test  Ascent  Aerodynamics  Confirmed   22   7.0  6  Steps  
  65. 65. Horizontal  and  VerCcal  Traceability   of  the  IMP/IMS   Integrated  Master  Schedule   Work  sequenced  to   produce  outcomes   for  each  WP.   !  VerCcal  traceability  AC  "  SA  "  PE   !  Horizontal  traceability  WP  "  WP  "AC   Program Events Define the maturity of a Capability at a point in time. Significant Accomplishments Represent requirements that enable Capabilities. Accomplishment Criteria Exit Criteria for the Work Packages that fulfill Requirements. Work   Package   Work   Package   Work   Package   Work   Package   Work   Package   Work   package   Work   Package   Work   Package   23   7.0  6  Steps  
  66. 66. The  IMP’s  connecCon  to  the  WBS   !  Start  with  the  Significant  Accomplishments  and  sequence   them  to  the  maturity  flow  for  each  Program  Event   !  The  WBS  connecCons  then  become  orthogonal  to  this  flow   24   7.0  6  Steps   Program  Event   SRR   SDR   PDR   CDR   TRR   ATLO   Work  Breakdown   Structure   4.920-­‐SDAI   A01,  A02   B01   C01,  C02   D01   E01   F01   4.200-­‐Sys  Test   A05   B03,  B04   D02,  D03   E02   F02   4.300-­‐Radar   A03   B02   C03   E03   4.330-­‐O&C  Sys   A06,  A07   B05   C04   D04   E04   F03,  F04   4.400-­‐  I&T   A08   C05   E05,  E06   F05   4.500-­‐Support   A09   D05   E07   F06,  F07  
  67. 67. 25   7.0  6  Steps  
  68. 68. 26   7.0  6  Steps  
  69. 69. 27   7.0  6  Steps  
  70. 70. 28   7.0  6  Steps  
  71. 71. 29   7.0  6  Steps  
  72. 72. 30   7.0  6  Steps  
  73. 73. Nuances  Of  These  6  Steps   Building  the  Program  Event,  to  Significant  Accomplishment,  to  Accomplishment   decomposi?on  is  straight  forward.   For  each  Program  Event,  simply  iden?fy  what  are  the  needed  Significant   Accomplishment  for  the  entry  and  exit  criteria,  and  the  Accomplishment  Criteria   for  the  Work  Packages  that  produce  the  AC.   Yea  Right,  no  problem   1   8 V8.5  
  74. 74. 2   8.0  Nuances  
  75. 75. Quick  View  of  the  Nuances   !  Unfortunately  building  a  credible  IMP/IMS  is  a   nuanced  process,  subject  to  many  opportuni?es   for  diversions,  blind  alleys,  and  false  starts     !  It  is  slightly  counter  intui?ve  from  the  tradi?onal   scheduling  approach  to  start  with  the  ver?cal   integra?on  –  but  it  is  cri?cal  to  start  ver?cally   !  Success  requires  the  full  par?cipa?on  of  Systems   Engineering,  CAMs,  and  the  Program  Manager   !  Success  requires  everyone  to  understand  the   nuances  of  the  IMP  building  efforts   3   8.0  Nuances  
  76. 76. The  1st  Nuance   We  need  to  change  the  Planning  Paradigm   Beginner     Intermediate   Advanced   ! Take  the  program  events   and  use  the  WBS  to  define   the  SAs  and  ACs.   ! Iden?fy  the  tasks  to  produce   the  ACs  and  SAs   ! Collect  them  into  Program   Events   ! Organize  the  tasks  by  Work   Package   ! Sequence  the  work  packages   (AC’s)     ! Examine  the  exit  criteria  for   the  Program  Events  –  what   does  Done  look  like?   ! Ask  what  the  entry  criteria   are  for  each  Program  Event   ! Build  the  AC’s  to  support   these  entry  criteria   ! Pull  these  together  under   each  SA   ! Determine  the  Technical  and   Programma?c  maturity  for   each  Program  Event  from   the  Concept  of  Opera?ons   ! Assess  the  SA’s  for  each   Integrated  Product  Team  in   terms  of  their  streams   maturity  at  that  point  in  the   program   ! Sequence  the  SA’s  for  each   PE  and  assess  the  units  of   measure  of  “maturity”   ! Build  the  AC’s  to  support   each  SA’s  level  of  maturity   Copyright  ©  2012,  Glen  B.  Alleman,  Niwot  Ridge,  LLC   4   8.0  Nuances  
  77. 77. The  2nd  Nuance   We  need  to  speak  about  Increasing  Maturity   Beginner   Intermediate   Advanced   ! The  sequence  of  work   closely  matches  the   horizontal  schedules  of  the   past   ! Align  this  work  with  the  WBS   elements   ! Focus  on  the  WBS  Dic?onary   of  the  delivered  products  or   services   ! No  explicit  TPM,  MOE,  and   MOP  elements     ! The  sequence  of  work  is   related  to  the  Program   Events,  but  essen?ally   “hangs”  from  the  PE  to  the   SA’s  and  then  the  AC’s   ! All  deliverables  are  visible   but  their  TPMs  and  other   system  measures  are  not   stated  in  the  IMP  or  its   narra?ve.   ! There  is  a  narra?ve  in  the   form  of  SA’s  and  AC’s  that   describes  how  the  program   moves  from  lef  to  right   alone  its  maturity  path   ! Risk  buy  down  and   re?rement  are  visible   ! Intermediate  Technical   Performance  Measures,   Measures  of  Effec?veness,   and  Measures  of   Performance  are  visible  in   the  IMP   Copyright  ©  2012,  Glen  B.  Alleman,  Niwot  Ridge,  LLC   5   8.0  Nuances  
  78. 78. The  3rd  Nuance   Everything  Foot  and  Ties  to  the  IMP  &  IMS   Beginner   Intermediate   Advanced   ! The  IMS  contains  all  the   proper  fields  in  columns  and   is  horizontally  linked   ! The  WBS  elements  can  be   found  for  all  work  elements   ! CDRL’s  are  visible  and  their   mul?ple  delivery  dates   connected  to  each  Program   Event   ! WBS  is  structured  in  a   product  manner  or  possibly   a  func?onal  manner  with   some  deliverables  defined  in   the  terminal  nodes   ! The  WBS  is  properly  formed   inside  each  AC  with   incremental  deliverables   ! WBS  numbers  form  a  “well   structured”  tree,  but  s?ll  is   not  “pure”  in  the  sense  of   deliverables  only,  no   func?onal   ! Each  column  and  each  field   can  be  “pivoted”  to  form  a   proper  “tree”  of  value  flow.   ! The  WBS  is  a  “pure”  Product   Breakdown  Structure  (PBS)   and  the  services  needed  to   produce  those  products   ! The  WBS  defines  the   structure  of  the  delivered   product  or  service   ! The  Ver?cal  trace  of  the  IPM   describes  the  flow  of   increasing  maturity  of  these   products  or  services   ! The  Horizontal  trace  of  the   IMP  describes  to  work  to  be   done  to  produce  this   maturity   6   8.0  Nuances  
  79. 79. The  4th  Nuance   IMP/IMS  is  Programma?c  Architecture   Beginner   Intermediate   Advanced   ! The  IMP  is  built  from  the   WBS  for  each  Program   Event.   ! The  IMP  is  seen  as  a   compliance  document  that   lists  the  Program  Events  and   a  “bunch  of  stuff”   underneath.     ! The  IMP  is  structured   around  separate  Program   Events,  but  below  the  SA’s   looks  like  a  “shop  floor”   schedule  with  lijle  ver?cal   connec?vity.   ! The  IMP  is  built  as  a  “value   stream”  flow  for  the   program  but  the  Systems   Engineers   ! This  programma?c   architecture  is  built  in  the   same  way  the  technical   system  architecture  is  built   ! It  is  derived  from  the   ConOps  and  Tier  1  System   Requirements   ! The  IMP  shows  explicitly   how  these  are  supported  in   the  flow  of  the  SA’s   7   8.0  Nuances  
  80. 80. The  5th  Nuance   The  IMP/IMS  connects  all  the  dots   8   8.0  Nuances   Measure  of   Effec?veness   Measure  of   Performance   Technical   Performance   Measure   Risk   Aleatory   Uncertainty   Epistemic   Uncertainty   Reference   Classes   Past   Performance   SME   Past   Performance   System   Architecture   AHP  
  81. 81. Connec&ng  The  IMP  To  Program   Performance  Measures   Assembling  the  IMS  from  the  IMP  appears  to  be  a  straight  forward  process  –   details  the  tasks  that  support  the  Accomplishment  Criteria.  But  there  are  some   cri&cal  steps  that  must  be  done  in  the  right  order  to  end  up  with  a  risk  tolerant   IMS.   Let’s  do  this  for  TSAS   1   9 V8.5  
  82. 82. The  Primary  Role  for  the  IMP  is  to  describe   what  done  looks  like  in  MoE’s  and  MoP’s   2  19  October  1899  Robert  Goddard  decided  that  he  wanted  to  "fly  without  wings"  to  Moon.   9.0  Framework  
  83. 83. Quick  View  to  IMP/IMS  Framework   !  Measures  of  increasing  maturity  for  the  key   deliverables  is  the  founda&on  for  increasing   the  Probability  of  Program  Success   !  Measures  of  Effec&veness  (MoE)  and   Measures  of  Performance  (MoP)  are  defined   in  the  Integrated  Master  Plan  (IMP)  Narra&ve   !  Key  Performance  Parameters  (KPP)  –  both   JROC  and  program  specific  are  needed   !  Technical  Performance  Measures  (TPM)  are   needed  for  all  key  deliverables   3   9.0  Framework  
  84. 84. Star&ng  out  on  the  Right  Foot  …   !  Most  IMP/IMS  literature  states  how  to  build  an  IMP  from  the  RFP  and   contractual  elements,  in  simple  and  maybe  simple  minded  terms   –  Decompose  the  events  into  SAs,  ACs,  and  their  Tasks  –  sounds  easy   !  This  approach  fails  to  provide  advice  for  several  things:   –  How  to  minimize  the  topological  connec&ons  between  Events   –  How  to  increase  the  concurrency  between  IPTs   –  How  to  increase  the  tolerance  of  the  IMS  to  disrup&ve  events   •  Known  and  knowable  risk   •  Unknown  and  possibly  unknowable  risk   !  The  construc&on  of  the  IMS  needs  to  take  place  in  what  seems  to  be  a   reverse  order   –  Build  the  IMP  as  a  Value  Stream  Map  describing  the  increasing  maturity  –  and   therefore  the  increasing  VALUE  of  the  deliverables  to  the  customer.   –  It’s  the  delivery  of  Customer  Value  that  inverts  the  management  process,  and   focuses  on  Keeping  the  Program  GREEN  as  planned  that  maximizes  value  to   the  customer  (the  Government)     4   9.0  Framework  
  85. 85. SETR  Program  Events   5  hdps://   9.0  Framework  
  86. 86. Build  PEs  Leg  to  Right   Start  with  SRR  (or  something  on  the  leA)  and  completely   define  its  compleDon,  before  moving  to  the  next  PE   !  Define  the  SAs  for  an  Event  and  construct  a  work  flow   of  the  ac&vi&es  needed  to  sa&sfy  the  SA   –  These  ac&vi&es  are  yet  tasks,  so  don’t  commit  too  soon  to   defining  the  detailed  work   –  Isolate  the  SAs  by  event  first  –  only  work  on  one  event  at  a   &me   !  Iden&fy  the  par&cipants  in  the  work   –  What  IPTs  par&cipate  in  this  work?   –  What  swim  lanes  are  needed  to  isolate  the  IPTs?   !  Define  the  elements     –  Ac&vi&es  performed  to  sa&sfy  the  SA   –  Deliverables  that  result  from  these  ac&vi&es   !  There  are  s&ll  not  Accomplishment  Criteria  (AC),  but   that  comes  next   6   9.0  Framework  
  87. 87. The  Accomplishment  Criteria  (AC)   !  A  defini&ve  measure  or  indicator  that  verifies   comple&on  of  work  for  the  accomplishment   – Completed  work  effort     •  Manufacturing  Plan  Completed   – Confirma&on  of  performance  compliance     •  Flight  Test  Report  Approved   – Incremental  verifica&on     •  Maintenance  Demonstra&on  Completed   – Completed  cri&cal  process  ac&vi&es   •  Risk  Management  Plan  Approved   7   9.0  Framework  
  88. 88. The  Accomplishment  Criteria  (AC)   !  Defines  the  measure  by  which  an  Accomplishment  (SA)  is   considered  “done”   !  Terms  like  complete,  delivered,  closed  have  no  “units  of   measure”  in  the  context  of  a  Significant  Accomplishment   (SA)  and  are  open  to  interpreta&on   !  Terms  like  …   –  Measures  of  comple&on  –  80%  of  drawings  approved  for  release   –  Counts  of  available  items  –  75%  of  pin-­‐outs  assigned  voltage   –  Fidelity  of  a  design  –  outer  mold  line  defined  within  90%  of   target   –  Error  bounds  –  spacecraA  mass  known  to  ±20%   –  Performance  parameters  –  disconnect  force  within  allowed   limits   –  Maturity  parameters  –  flight  arDcle  successful  in  last  3  tests   …  are  used  to  define  the  “exit  criteria”   8   9.0  Framework  
  89. 89. 2  Types  of  Accomplishment     Criteria:  Entry  and  Exit   !  Entry  Criteria  –  Substan&ates  readiness  for  the   review   !  Exit  Criteria  –  Substan&ates  successful   comple&on  of  the  review   !  Cri&cal  Design  Review  (CDR)  example   – Are  we  ready  for  the  Flight  Test  Readiness   Review?   – How  do  we  know  the  FTRR  a  success?   – What  did  we  learn  from  the  FTRR  that  increases   the  maturity  of  the  program’s  deliverables.   9   9.0  Framework  
  90. 90. The  IMP  Focuses  us  on  Measures   of  Effec&veness  and  Performance   10   MoE   KPP   MoP   TPM   Mission   Need   Acquirer  Defines  the  Needs  and  Capabili&es   in  terms  of  Opera&onal  Scenarios   Supplier  Defines  Physical  Solu&ons  that   meet  the  needs  of  the  Stakeholders   Opera3onal   measures  of  success   related  to  the   achievement  of  the   mission  or   opera3onal   objec3ve  being   evaluated.   Measures  that   characterize   physical  or   func3onal  aAributes   rela3ng  to  the   system  opera3on.   Measures  used  to   assess  design   progress,   compliance  to   performance   requirements,  and   technical  risks.   9.0  Framework  
  91. 91. Measure  of  Effec&veness  (MoE)   !  Measures  of  Effec&veness  …   !  Are  stated  in  units  meaningful  to  the  buyer,   !  Focus  on  capabili&es  independent  of  any   technical  implementa&on,   !  Are  connected  to  the  mission  success.   The  opera&onal  measures  of  success  that  are  closely  related  to  the   achievements  of  the  mission  or  opera&onal  objec&ves  evaluated  in   the  opera&onal  environment,  under  a  specific  set  of  condi&ons.   “Technical  Measurement,”  INCOSE–TP–2003–020–01   MoE’s  Belong  to  the  End  User   11   9.0  Framework  
  92. 92. Measure  of  Performance  (MoP)   !  Measures  of  Performance  are  …   !  Adributes  that  assure  the  system  has  the   capability  to  perform,   !  Assessment  of  the  system  to  assure  it  meets   design  requirements  to  sa&sfy  the  MoE.   Measures  that  characterize  physical  or  func&onal  adributes   rela&ng  to  the  system  opera&on,  measured  or  es&mated   under  specific  condi&ons.   “Technical  Measurement,”  INCOSE–TP–2003–020–01   MoP’s  belong  to  the  Program  –  Developed  by  the  Systems   Engineer,  Measured  By  CAMs,  and  Analyzed  by  PP&C     12   9.0  Framework  
  93. 93. Key  Performance  Parameters  (KPP)   Both  JROC  and  Program  Specific   !  Key  Performance  Parameters  …   !  Have  a  threshold  or  objec&ve  value,   !  Characterize  the  major  drivers  of   performance,   !  Are  considered  Cri&cal  to  Customer  (CTC).   Represent  the  capabili&es  and  characteris&cs  so   significant  that  failure  to  meet  them  can  be  cause  for   reevalua&on,  reassessing,  or  termina&on  of  the  program   “Technical  Measurement,”  INCOSE–TP–2003–020–01   The  acquirer  defines  the  KPPs  during  the  opera&onal   concept  development  –  KPPs  say  what  DONE  looks  like   13   9.0  Framework  
  94. 94.   Technical  Performance  Measures   (TPM)  for  key  deliverables     !  Technical  Performance  Measures  …   !  Assess  design  progress,   !  Define  compliance  to  performance   requirements,   !  Iden&fy  technical  risk,   !  Are  limited  to  cri&cal  thresholds,   !  Include  projected  performance.  “Technical  Measurement,”  INCOSE–TP–2003–020–01   Adributes  that  determine  how  well  a  system  or  system   element  is  sa&sfying  or  expected  to  sa&sfy  a  technical   requirement  or  goal   14   9.0  Framework  
  95. 95. What  are  Technical  Performance   Measures  Really?   !  TPMs  are  measures  of  the  system   technical  performance  that  have   been  chosen  because  they  are   indicators  of  system  success.  They   are  based  on  the  driving   requirements  or  technical   parameters  of  high  risk  or   significance  -­‐  e.g.,  mass,  power  or   data  rate.   !  TPMs  are  analogous  to  the   programma&c  measures  of   expected  total  cost  or  es&mated   &me-­‐to-­‐comple&on.  There  is  a   required  performance,  a  current   best  es&mate,  and  a  trend  line.   !  Actual  versus  planned  progress  of   TPMs  are  tracked  so  the  systems   engineer  or  project  manager  can   assess  progress  and  the  risk   associated  with  each  TPM.     !  The  final,  delivered  system  value   can  be  es&mated  by  extending  the   TPM  trend  line  and  using  the   recommended  con&ngency  values   for  each  project  phase.   !  The  project  life  trend-­‐to-­‐date,   current  value,  and  forecast  of  all   TPMs  are  reviewed  periodically   (typically  monthly)  and  at  all  major   milestone  reviews.   15   9.0  Framework  
  96. 96. !  Tracking  TPMs  and  comparing  them  to  the  resource  growth   provides  an  early  warning  system  to  detect  deficiencies  or   excesses   !  Reserve  alloca&ons  narrow  as  design  proceeds   !  TPMs  that  violate  reserve  alloca&ons  or  have  trends  that   do  not  meet  the  final  performance  trigger  correc&ve   ac&ons   16   Tracking  the  Technical     Performance  Measures   9.0  Framework  
  97. 97. No&onal  TPM  Design  Margin   Guidelines   17   9.0  Framework  
  98. 98. Sample  IMP:   A  Flight  Avionics  System  (Con&nued)   Hardware  PDR  –  Purpose     !  Ensure  system  hardware  ini&al  design  has  been  updated,  and  meets   func&onal  and  allocated  performance  requirements  within  program   constraints.     !  Opera&onal  security  concept  assessed.     !  Ensure  training  requirements  have  been  analyzed  and  their  objec&ves   have  been  defined  for  training  missions.     !  Confirma&on  training  objec&ves  and  MTC  design  and  integra&on  conform   to  the  Air  Force  syllabus  and  Ready  Aircrew  Program  (RAP).     !  Training  plan  will  be  updated.   !  Hardware  PDR  –  Expecta&ons   !  Team  agrees  system  hardware  ini&al  design  has  been  updated  and  can   proceed  to  the  detailed  design  phase.     !  Team  agrees  training  plans  and  objec&ves  correlate  with  the  Air  Force   syllabus,  RAP  and  training  planning,  development  can  con&nue.     !  System  Specifica&on  and  TTL  requirements  are  traceable  to  the  allocated   hardware  design.     18   9.0  Framework  
  99. 99. Sample  IMP:   A  Flight  Avionics  System  (Con&nued)   !  Hardware  PDR  –  Entry  Criteria   !  Func&onal  Baseline  Authen&cated  (FBA)   !  Ini&ate  system  hardware  ini&al  design  and   allocate  func&ons  to  the  appropriate   Configura&on  Items   !  All  specifica&ons  updated  and  required   documenta&on  is  made  available  including   an&cipated  lower  level  design  documenta&on   !  All  SRR/SFR  ac&on  items  closed  or   disposi&oned     19   9.0  Framework  
  100. 100. Sample  IMP:   A  Flight  Avionics  System  (Con&nued)   Hardware  PDR  –  Accomplishments   !  System  hardware  ini&al  design  complete.     –  Func&ons  allocated  to  one  or  more  hardware  configura&on  items  and   are  traceable  to  the  MTC  SSS  and  TSSC  SSS.     –  Human,  safety,  R&M,  EMI,  opera&onal  security,  instructor  and   operator  interfaces,  etc,  design  factors  have  been  reviewed.     !  Drag  instructor  and  operator  manuals  reviewed     !  Program  risks  updated,  assessed,  and  reviewed.     –  Mi&ga&on  plans  in  place.   !  Program  schedule  and  constraints  updated  and  reviewed.     –  Cri&cal  schedule  path  drivers  reviewed.     !  Design  criteria  for  the  simula&on  and  database  development   reviewed  and  updated.   !  Program  processes  and  metrics  reviewed.   !  Test  Planning  ac&vi&es  and  relevant  documenta&on  reviewed  by   test  team.   20   9.0  Framework  
  101. 101. Sample  IMP:   A  Flight  Avionics  System  (Concluded)   Hardware  PDR  –  Exit  Criteria     !  Hardware  (ownership,  visual,  IOS,  brief/debrief)  design  reviewed,  allocated  to  a   hardware  configura&on  item  and  updated  to  include  instructor  and  operators   interfaces,  malfunc&on  and  control  requirements,  etc.     !  RTM  updated,  MTC  SSS,  TSSC  SSS  and  TTL  traceable  to  allocated  hardware  design   to  include  ESOH  requirements.     !  Human,  safety,  R&M,  EMI,  opera&onal  security,  instructor  and  operator  interfaces,   etc,  design  factors  reviewed.     !  MTC  and  TSSC  allocated  baselines  established  and  controlled  by  appropriate  level   documenta&on  for  PDR.   !  Drag  instructor  and  operator  manuals  reviewed  with  user  concurrence  and   incorporate  sa&sfactory  human  factor  design  factors  into  the  operator  interfaces.     !  Risk  management  and  mi&ga&on  plans  updated,  in  place,  addresses  ESOH  plans   and  risks,  and  within  program  constraints.     !  Risks  assesses,  understood,  documented,  accepted  and  understood  by  team.   !  Program  schedule  reviewed     –  Cri&cal  path  drivers  iden&fied     –  IMS  Updated  and  reflects  cri&cal  paths   21   9.0  Framework  
  102. 102. One  More  IMP  Sample   !  (SA)  System  &  Segment   Requirements  Updated  &  Allocated   –  (AC)  SRR  /  SDR  Update  Review   Conducted   –  (AC)  Preliminary  System   Specifica&on  Documents  (A011)   Baselined   –  (AC)  Preliminary  Spacecrag   Segment  Specifica&on  Baselined   –  (AC)  Preliminary  Ground  Segment   Specifica&on  Baselined   –  (AC)  Preliminary  Specifica&on  Tree   Baselined   !  (SA)  Preliminary  ICDs  Baselined  For   Customer  Review   –  (AC)  Preliminary  Space-­‐Ground   ICD  Baselined     !  (SA)  PDR  System  Design  Completed   –  (AC)  Top  Level  System   Architecture  Updated   –  (AC)  PDR  Level  System  Analyses   Completed   –  (AC)  PDR  Level  Reliability  /   Availability  Analysis  Completed   –  (AC)  Preliminary  System  Level  Risk   Assessment  Completed   –  (AC)  System  Level  Plans  Updated   For  PDR   –  (AC)  Flight  Long  Lead  Review   Conducted   22   Preliminary  Design  Review   9.0  Framework  
  103. 103. The  “But”  for  this  Guidance   !  With  these  samples  and  the  SETR  guidance   we’ve  just  started   !  The  program  needs  to  define  program  specific   events  to  assure  the  actual  maturity  measures   are  captured   – The  IMP  provides  sufficient  definiDon  to  track  the   step-­‐by-­‐step  compleDon  of  the  required   accomplishments  for  each  event  and  to   demonstrate  saDsfacDon  of  the  compleDon   criteria  for  each  accomplishment.  [AFMC   PAMPHLET  63-­‐5]   23   9.0  Framework  
  104. 104. IMP  Verbs  for  Significant   Accomplishments   24   Integrated  Master  Plan  Allowable  Verbs   Allocated:  Segment  requirement  is  flowed  down   from  the  System  Specifica&on   Released:  Approved  item  for  delivery  for   intended  customer  or  supplier;  all  internal   distribu&on  and  sign  offs  complete.  An   electronic  version  is  made  accessible  on  the  IDE   Completed:  The  subject  item,  data,  document,   or  process  is  prepared  or  concluded,  and   reviewed  and  accepted  by  the  responsible  IPT.   Suppor&ng  documenta&on  is  available  through   IDE   Reviewed:  The  subject  item,  data,  document,  or   process  is  prepared  or  concluded,  and   documented  for  comple&on.  Suppor&ng   documenta&on  is  available  through  IDE   Conducted:  The  subject  mee&ng  or  review  has   been  held  with  all  required  par&cipants.  The   charts  or  minutes  are  available  through  the  IDE   Updated:  The  subject  process,  data,  or   document  has  been  reevaluated  using  later   informa&on,  and  adjustments  incorporated.   Defined:  The  subject  configura&on  items,  data,   or  document  was  submided  to  the  customer   Validated:  Requirements  are  validated,  received   contractor  approvals,  were  distributed,  and  are   available  through  the  IDE   Established:  The  subject  items  is  created  and  set   in  place  in  a  manner  consistent  with  its  intended   use  ager  review  and  accepted  by  the  IPT   Verified:  Requirements  are  verified  or  processed   in  accordance  with  established  prac&ce.   9.0  Framework  
  105. 105. The  IMP  Process  Narra&ve   !  ObjecFve:  a  brief  statement  explaining  why  this   process  set  is  applied  for  this  program   !  Governing  DocumentaFon:  lists  of  the  guidance   or  compliance  documents;  e.g.,  specifica&ons,   manuals,  and  procedures  including  company,   government,  and  industry  references   !  Approach:  concise  descrip&on  as  to  who  owns   each  process;  what  are  the  roles  and   responsibili&es;  and  the  overall  process  including   a  process  flow  diagram.   25   9.0  Framework  
  106. 106. Generic  IMP  Evalua&on  Criteria   !  Do  the  Program  Events  and  Accomplishments   reflect  the  logical  evolu&on  and  progress  of  the   overall  Program?   –  Do  program  events  and  their  defini&ons  clearly   demonstrate  the  maturity  of  the  program  over   its  life?   –  Do  the  selected  Accomplishments  and   associated  Criteria  iden&fy  meaningful  and   measurable  progress  toward  the  key  goals  of   the  Program?   !  Do  the  Accomplishments  for  each  event   demonstrate  a  meaningful  understanding  of  the   program  requirements  or  are  they  tasks  that   anyone  could  do  for  any  contract?   –  Do  they  reflect  your  SOW  requirements?     !  Does  the  IMP  structure  readily  map  to  the  IPT   structure  such  that  each  IPT  can  easily  visualize   the  scope  of  their  responsibility?     !  When  awarded,  could  the  contractor  use  the   Accomplishments  as  discrete  ac&vity  cost  accounts   in  their  earned  value  system?  Or  are  they  level  of   effort  in  nature?     !  Is  sufficient  visibility  provided  to  iden&fy  and  track   the  Program  Risk  Plan  and  associated  risk   mi&ga&on  accomplishments  and/or   con&ngencies?   –  Does  criteria  suppor&ng  the  accomplishments   include  key  performance  requirements?   !  Are  IPT  cross-­‐dependencies  and  dependencies   external  to  the  Program  appropriately  reflected  if   they  reflect  poten&al  schedule  or  performance   risks  to  the  success  of  the  Program?   !  Is  the  submided  "Contract  IMP"  (the  Product  IMP   that  is  to  be  included  as  part  of  the  Program   Contract)  defined  to  the  appropriate  level:   –  Do  the  Accomplishments  and  associated   Criteria  go  down  to  a  level  sufficient  to  provide   visibility  into  key  subcontractor  ac&vi&es  upon   which  the  success  of  the  Program  may  be   dependent?   –  Are  the  Events  and  Accomplishments  included   in  the  IMP  at  such  a  level  as  to  make  the   maintenance  of  the  'Contractual  IMP'  prac&cal   or  does  it  include  an  unnecessary  level  of   detail?   26   9.0  Framework  
  107. 107. Program  Management  Levels   Program Levels" IMP/IMS Elements" CWBS" Tier 1! Program Manager! Technical Leads! IPT Manager! Technical performance goals! Major Program Events (PE)! ! ! ! Significant Accomplishments (SA)! Level 1 & 2! Links to CLINs! Level 3 & 4! Control Packages! Link to PBS! Integrated EVMS! Tier 2! Control Account Managers ! Product Work Plan! Responsible organization elements! ! Accomplishment Criteria (AC)! ! ! ! ! Tasks (NA)! Level 5! Cost Account Package! Cost Collection Level! Links to WBS by OBS! Resource summaries! Early warning EVMS analysis! Tier 3! Work Package Manager! Detailed plans! Work package! Earned value calculations! 27   9.0  Framework  
  108. 108. Connec&ng  the  Components     of  the  IMP/IMS   !  The  assembled  IMP/IMS  links  all  work   ac&vi&es  ver&cally  to  the  ACs,  SAs,  and  PEs   28   IMS   Customer   Requirements   IMP   ORG   IPTs   ORG   IPTs   Performance   Analysis/   Management   Review   Events (E)  Events (E)   Accomplishments   Process   Narratives   Criteria   Integrated Master   Schedule (IMS)   Integrated Master   Schedule (IMS)   Control Account  Control Account   Work Package  Work Package   Work Package Tasks  Work Package Tasks   WBS   Program   Performance   Management   System   Supplemental Schedules   Risk and   opportunity   Risk and   opportunity   9.0  Framework