How DOORS Helps JPL Get to    Mars and Beyond   Best Practices in Metrics,     V&V and Traceability              Margaret ...
Outline            Overview of DOORS Use at JPL             Topic 1 Metrics             Topic 2 Verification and Validatio...
JPL use of DOORS at JPL                      Statistics      – 800+ users      – 600+ users have taken our in-house DOORS ...
DOORS Provenance – According to JPL    Source code and User Guide were         QSS                      Telelogic   IBM Ra...
Sample Project Sizes                                                                    installed                  Project...
Why Have a Standard Process?     • DOORS is a general purpose tool and there are many possible ways       to use it.     •...
What Has and Hasn’t Been Standardized?Has•   Ownership, allocation and negotiation mechanism•   Attributes and their defin...
Topic 1                       Metrics•   Maturity Definition:      +     V&V Method/Approach defined      +     Rationale ...
Example            Unknown                           Allocations            capability?     Gold           undecided or un...
What Has Changed Today?             Daily email to Project’s requirements owner            Assessing Requirements activity...
Topic 2                                 Verification                            Planning and Tracking                     ...
Verification Process       Project     Verification      Planning  Get DOORS                 Verification                 ...
Verification ElementsDefinitionsVI – Verification Item, is anything that needs to be verified. All requirementsbecome Vis ...
Verification Attributes                                                                                                   ...
Project Module ConfigurationsVIM – Verification Item (Requirements) ModuleVAM – Verification Activity ModuleVEM – Verifica...
Verification Item (VI) Burndown                                 250                                                       ...
Verification Activity Burndown                                      16                                                    ...
Verification Item Statistics                                                                                     "Bad" if ...
Verification Activity Statistics                                                                       "Bad" if > 0       ...
Topic 3      DOORS Trace Tree Tool (T3)JPL has parent-to-child requirements relationships where a parent has 1 to tens ofc...
Trace Tree Tool View – Example Index Page    ProjX    ProjX    ProjX    ProjX    ProjX   •••5/24/2012      Caltech – Jet P...
Project Map               Number of links/flowed-down requirements appear on edges5/24/2012    Caltech – Jet Propulsion La...
Module Page            ProjX  •••            …….5/24/2012           Caltech – Jet Propulsion Laboratory   J. Grimes - 23
Requirement Page                            ………………….                         ……..            Attributes continue below5/24...
Object Text Graphic                                                                                  ………………………    …………..  ...
Verification Approach Graphic                                                                                    ……………    ...
Typical DOORS T3 User Scenarios • Quickly obtain the contents of a requirement with a known ID       – Main page      Modu...
DOORS Trace Tree Tool (T3)            High-Level Process/Execution Flow5/24/2012             Caltech – Jet Propulsion Labo...
Development Tools Used• DXL – DOORS eXtension Language, a DOORS proprietary C-like, object-    oriented language available...
JPL’s Direction with DOORS•    Systems Engineers on JPL Projects are starting to infuse Model Based     Systems Engineer /...
Upcoming SlideShare
Loading in...5
×

How DOORS Helps JPL Get to Mars & Beyond

843

Published on

This paper was presented on June 4, 2012 at the IBM Rational Innovate Conference in Orlando Florida by Margaret Smith, NASA Jet Propulsion Laboratory.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
843
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
14
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

How DOORS Helps JPL Get to Mars & Beyond

  1. 1. How DOORS Helps JPL Get to Mars and Beyond Best Practices in Metrics, V&V and Traceability Margaret H. Smith Jim M. Grimes Ross M. Jones Trisha Jansma Jet Propulsion Laboratory California Institute of Technology
  2. 2. Outline Overview of DOORS Use at JPL Topic 1 Metrics Topic 2 Verification and Validation (V&V) Topic 3 Traceability5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 2
  3. 3. JPL use of DOORS at JPL Statistics – 800+ users – 600+ users have taken our in-house DOORS training. – 50 DOORS licenses – 50 projects have used DOORS, 22 are active – 15 years using DOORS, preceded by 8 years using in- house TRACER tool. – 5 years of a standard process with in-house training – 2 System Analysts for daily project support – 1 Full-time Equivalent (FTE) Systems Engineer for defining the method, teaching and project consultation5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 3
  4. 4. DOORS Provenance – According to JPL Source code and User Guide were QSS Telelogic IBM Rationaldistributed from NASA COSMIC site from DOORS DOORS DOORS1990 to ~1995. Developed by the Cassini Project and used by JPL and ESA (1996) (2000) (2008) JPL JPL TRACER DOORS TRACER mothballed adopted at (1988) (1995) JPL(1996) 5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 4
  5. 5. Sample Project Sizes installed Project # objects # links # Modules TTT website size (GB) DSN 98995 5709 125 1.9 Large legacy infrastructure project SMAP 17278 13311 91 1.9 Medium complexity earth orbiting mission GRAIL 36031 30650 123 2.7 Smaller complexity lunar mission (JPL managed) SES_JUNO 84844 43526 227 3.6 Large complexity deep space mission (JPL managed) MSL 25641 5808 106 1.2 Flagship mission # Modules Measurements were generated by the JPL Trace Tree Tool (TTT) 0 50 100 150 200 250 MSL SES_JUNO # links GRAIL # objects # Modules SMAP DSN 0 20 40 60 80 100 120 # Links, Objects (Thousands) Note: Number of objects includes all objects in formal modules: table cells, headers, regular objects with or without text or OLE content. The number of requirements is about one-third the number of objects.5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 5
  6. 6. Why Have a Standard Process? • DOORS is a general purpose tool and there are many possible ways to use it. • For the first 15 years we used DOORS, each new project reinvented the wheel. – Big projects could afford to invent their own method and have internal support. • We learned a lot from what the big projects did wrong and did right. – We now have more small projects and fewer big projects. • Small projects cannot afford to reinvent. Need a ready-made solution. • The most common mistake: trying to run the old document-centric process in DOORS and not taking advantage of the tool’s capabilities. • Projects were making mistakes, blaming DOORS, and delaying import of requirements into tool as late as possible. – Managing requirements in spreadsheets and not getting the benefits of early sharing and collaboration.5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 6
  7. 7. What Has and Hasn’t Been Standardized?Has• Ownership, allocation and negotiation mechanism• Attributes and their definitions – for requirements and V&V planning/tracking• Views – requirements traceability, allocation status, requirements to V&V traceability• High-level requirements template, boilerplate material• Scripts – Requirement Maturity Metrics, and V&V Burndown and Statistics, and numerous housekeeping scripts used by DOORS System Analysts (SAs).• Tools – Trace Tree Tool for graphically displaying individual requirements traceability and project requirements flow-down.• Practices – the JPL Systems Engineering practices that culminate in requirements.• Licensing and Support – the JPL institution purchases DOORS licenses for use by all projects and funds two Systems Analysts (support).• Training/Deployment – Engineers who are involved in requirements work attend a JPL developed course that teaches basic DOORS skills and the JPL standard method for development requirements and tracking verification in DOORS.Hasn’t• Requirements levels and element decomposition• Detailed requirements templates• DOORS collaboration method with partners and subcontractors We owe the success of the JPL Institutional Requirements and V&V Tracking process and training program to Richard Stoller (retired), and Karen Boyle (who sadly passed away in February, 2011).5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 7
  8. 8. Topic 1 Metrics• Maturity Definition: + V&V Method/Approach defined + Rationale defined + Linked to parent(s) + Links exist to this requirement from Satisfiers (including V&V activities) + Satisfier (lower level owner) accepts allocated requirement – TBD, TBR, TBCs• Requirement Maturity Metric script: – Implemented in DOORS Extension Language (DXL) – Generates per module statistics across all modules in a Project, one module per row – Produces .csv file of statistics. – Project can customize output to suit their Project Management/reporting needs. – Provides a quick summary of requirement progress and facilitates comparison between modules.5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 8
  9. 9. Example Unknown Allocations capability? Gold undecided or under plating? negotiation 3 2195/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 9
  10. 10. What Has Changed Today? Daily email to Project’s requirements owner Assessing Requirements activity and churn Module name Requirement number Type of change Requirements are marked Modified if any attribute, with the setting Affect change dates turned on, has been changed since the last time this report was generated.5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 10
  11. 11. Topic 2 Verification Planning and Tracking Other benefits:• JPL Projects have • Lower tooling cost– use existing been recording institutionally Verification supported tool and information in support DOORS for the last 10 years. • Enables useful views of requirements, verification• Initial benefit -- requirements planning & status, and changes were immediately visible verification events. to engineers responsible for • Projects can use as-is or customize a verifying requirements because generic DXL script to generate verification activity artifacts and verification statistics and graphics. verification events were now linked directly to requirements • Saves projects money through reuse of tooling and method. 5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 11
  12. 12. Verification Process Project Verification Planning Get DOORS Verification Verification Verification Training / Activity Activity Status consulting Planning Execution Reporting Verification Execute VA Activity Groups procedure, analyze Generate V&V Requirements defined results and determine burndown charts Development status and statistics and present at Reqmts to VA Record status of management mapping the VA (procedure) meetings captured run and create Develop V&V method, venue problem/failure and approach reports (if needed) Schedule VA Take corrective activity action (if completion (need necessary) by) date Cognizant person approves VA execution result Plans and procedures developed outside DOORS and hyper- Cognizant person linked from the DOORS approves that the VA requirement has been verified5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 12
  13. 13. Verification ElementsDefinitionsVI – Verification Item, is anything that needs to be verified. All requirementsbecome Vis when they are being verified.VA – Verification Activity. The testing activities described in a single TestPlan, Procedure or Analysis. A group of associated VAs are a VA Group (VAG) Test Data Directory 1VE – Verification Event. A run or execution of a VA. 1 0..1 Test Test Plan Activity Report (AR) 1 1..M Procedure 0..1 1 In Other Tools 0..M In DOORS 1 1 1 1..M 1 Verification Item (VI) Verification Activity (VA) Verification Event (VE) 1..M 1..M VE data elements originate in the AR Verification Activity and are pushed to VE to complete the status picture. Group (VAG) 5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 13
  14. 14. Verification Attributes Test Data Directory 1 Test Plan 0..1 Test Procedure 1 • version 1..M • version Activity Report (AR) 0..1 0..M 1 1 VA 1 VI or ITL item • activity name or activity 1•object identifier (ID) group name•statement of requirement • VA priority VE•rationale • VA status • VA id (a reference)•…. 1..M • VA owner 0..M • …. data imported from AR•method 1..M • venue • url of Activity Report 1•verification approach • date scheduled (for • url of test data directory•audit status completion)•audit completed on (date) • date completed•audited by • plans and procedures • PFRs Requirements V&V Activity Planning V&V Activity Execution Development V&V Status Reporting 5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 14
  15. 15. Project Module ConfigurationsVIM – Verification Item (Requirements) ModuleVAM – Verification Activity ModuleVEM – Verification Event Module VIM VAM Multiple VAMs reflecting the VIM VIM VAM VAM VEM Project’s requirements structure or V&V organization. VIM VIM VIM VAM VAM VAM VIM A single VAM for the Project’s complete requirement tree VIM VIM VAM VEM • partition VAM to give owners granular write access VIM VIM VIM VIM Combined VAM and VEM for projects that only need VIM VIM VAM / VEM information on the latest run/execution.5/24/2012 VIM VIM VIM Caltech – Jet Propulsion Laboratory M. Smith - 15
  16. 16. Verification Item (VI) Burndown 250 Current date Not ConfirmedNumber of Items to be Verified 200 Actual (auto status) A pick list allows selection of 150 any group of VIMs for this chart. 100 Scheduled Not Scheduled 50 0 4/1/2008 1/8/2008 2/5/2008 3/4/2008 7/8/2008 8/5/2008 9/2/2008 12/9/2008 1/6/2009 1/22/2008 2/19/2008 3/18/2008 4/15/2008 4/29/2008 5/13/2008 5/27/2008 6/10/2008 6/24/2008 7/22/2008 8/19/2008 9/16/2008 9/30/2008 10/14/2008 10/28/2008 11/11/2008 11/25/2008 12/23/2008 Count of VIs (modules included selected from pick list): Scheduled Scheduled = total number of VIs remaining to be completed per schedule Not Completed = total number of VIs remaining to be completed actually Actual Interval End Date Not Confirmed = total number of VIs remaining to be Confirmed-Final Not Confirmed Not Scheduled = total number of VIs remaining to be completed but not scheduled 5/24/2012 Caltech – Jet Propulsion Laboratory R. Stoller, K. Boyle - 16
  17. 17. Verification Activity Burndown 16 Actual 14Number of Activities to be Verified 12 Scheduled 10 8 Not Scheduled 6 A pick list allows selection of 4 any group of VAs for this chart. 2 0 Scheduled Scheduled - Count of notof VAs remaining to be completed per schedule = total number completed linked VAs scheduled to be completed Actual Not Completed - Count ofof VAs remaining to be completed actually = total number not completed linked VAs remaining to be completed (Actual) Not Scheduled = total number of VAsnot scheduled completed and are not Not Scheduled - completed-NF or that are not scheduled 5/24/2012 Caltech – Jet Propulsion Laboratory R. Stoller, K. Boyle - 17
  18. 18. Verification Item Statistics "Bad" if > 0 "Neutral" "Good" from V&V Burndown Script that generates data to Excel "Normal"Count of VIs in VIMs (shall statements only) A: B: C: D: E: F: G: H: I: J: K: L M: N: VIs VIs VIs VIs Not VIs VIs Not VIs Not VIs Not VIs Not V&V Engr VIM Linked to Linked to Linked to VIs Auto VIs Conf Linked to Linked to Linked to Auto Conf Verif and Responsible NAME Total VIs VAM VAGs VAs Verif Verif VAM Reject a VA Verif Verif Late for this VIMVIM1 625 600 15 575 200 150 25 10 50 425 450 5 John JonesVIM2 500 500 10 485 100 75 0 5 15 400 etc 5 Susie JonesVIM3 1000 1000 0 1000 0 0 0 0 0 1000 etc 0 Mary Smith… … … etc etc etc etc etc etc etc etc etc etcVIM n 200 200 5 190 25 5 0 5 10 175 etc 0 Total 2325 2300 30 2250 325 230 25 20 75 1975 etc 10Relationships: B=C+H; I is not included in D; E=C-I-D; J=B-E=H+I+D; K=B-F; L=B-GM = VIs not auto verified AND scheduled completion date is missed or has not been scheduled or is notlinked. Any object containing the word “shall” enclosed in quotes (“shall”) is not included in any statistic.5/24/2012 Caltech – Jet Propulsion Laboratory R. Stoller, K. Boyle - 18
  19. 19. Verification Activity Statistics "Bad" if > 0 from V&V Burndown Script that generates data to Excel "Neutral" "Good" "Normal" # of VI Are VAs Fully ** Is there a VA Are there any Schedule Days VA Parent Linked to VIs VA Sched- Plan &/or a Suspicious VI Remaining (if not V&V Engineer / NAME* Links ? VA Status uled? Procedure? Links? completed) Owner of VAVA-x 25 Yes Blank 08/10/09 No Yes 10 Jack JonesVA-y 20 Yes Completed-NF 07/26/09 Yes No -5 Mary JonesVA-z 100 Yes Completed No Yes No Susie Que… 350 … Blank No No No Not Sched Tom SmithVA-n 55 Yes Blank 08/20/09 Yes Yes 20 Susie Smith * VAs are organized in VAM under VAGs and sub-header VAGs, but tabulated separately here. ** Identifies if there is an http entry in the Plan and Procedure attribute. 5/24/2012 Caltech – Jet Propulsion Laboratory R. Stoller, K. Boyle -19
  20. 20. Topic 3 DOORS Trace Tree Tool (T3)JPL has parent-to-child requirements relationships where a parent has 1 to tens ofchildren, and links between levels can be many-to-many. We have trouble seeing theforest for the trees and navigating requirements flow within DOORS.• Primary Motivations – Provide JPL users with an alternative graphical presentation of requirement flow (links) – Allows navigation of requirements based on hierarchy or flow – Enables seamless navigation between modules following flow- down/links – Provides easy access via Web-based, platform independent interface• Secondary Motivation - evolved with implementation – Provide an indication of timeliness of requirements flow-down – Provide an efficient way to checkout verification planning• History – Conceived and developed by a JPL Software Systems Engineer (James Grimes) in 2006 – Enthusiastically used by 8 JPL Flight Projects5/24/2012 Caltech – Jet Propulsion Laboratory J. Grimes - 20
  21. 21. Trace Tree Tool View – Example Index Page ProjX ProjX ProjX ProjX ProjX •••5/24/2012 Caltech – Jet Propulsion Laboratory J. Grimes - 21
  22. 22. Project Map Number of links/flowed-down requirements appear on edges5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 22
  23. 23. Module Page ProjX ••• …….5/24/2012 Caltech – Jet Propulsion Laboratory J. Grimes - 23
  24. 24. Requirement Page …………………. …….. Attributes continue below5/24/2012 Caltech – Jet Propulsion Laboratory J. Grimes - 24
  25. 25. Object Text Graphic ……………………… ………….. ……………………………………………… ………….. ……………………….……………….. ……………….. ………….. ……………………… Attributes continue below5/24/2012 Caltech – Jet Propulsion Laboratory J. Grimes - 25
  26. 26. Verification Approach Graphic …………… …………………………… …………………………… …………………………… …………………………… …………… …………………………… …………………………… …………………………… …………………………… …… Attributes continue below5/24/2012 Caltech – Jet Propulsion Laboratory J. Grimes - 26
  27. 27. Typical DOORS T3 User Scenarios • Quickly obtain the contents of a requirement with a known ID – Main page Module page Requirement number • Scan a particular Project module for a specific requirement – Main page Module page, then • Scroll up and down at web browser speeds • Use browser’s find capability (not as flexible as DOORS, but faster) • Trace requirements flow-down starting at any requirement – Click on boxes in graph going up- or downstream in requirements flow – Go from there to text, attributes • Obtain immediate visual indication of timeliness of flow-down – E.g., has upstream requirement X been updated more recently than my requirement? If yes, then it’s red. Actual date is shown on X’s page. – Is flow-down complete? Does the requirement have a parent? • Provide supportive evidence during Project Reviews – Quickly show existence of flow-down – Quickly show V&V attribute, implementation status (if in attribute)5/24/2012 Caltech – Jet Propulsion Laboratory J. Grimes - 27
  28. 28. DOORS Trace Tree Tool (T3) High-Level Process/Execution Flow5/24/2012 Caltech – Jet Propulsion Laboratory T. Jansma - 28
  29. 29. Development Tools Used• DXL – DOORS eXtension Language, a DOORS proprietary C-like, object- oriented language available within the DOORS client• Perl – a scripting language widely available on UNIX, Mac OS X, Linux, Windows• Graphviz – open source Graph Visualization software developed at AT&T that includes the Dot language and dot program for drawing directed graphs• XML and Perl/C-based XML parsing software – open source software• Java for processing images• Standard UNIX utilities such as date, rm, ls, rsync, etc.• SVN – Subversion open-source version control tool5/24/2012 Caltech – Jet Propulsion Laboratory J. Grimes - 29
  30. 30. JPL’s Direction with DOORS• Systems Engineers on JPL Projects are starting to infuse Model Based Systems Engineer / SysML. – Modeling includes architecture, behavior and resources. – Many pockets of discipline specific modeling tools exist in the Spacecraft subsystem organizations – Vision is to use a central, standards based repository to interconnect and exchange model data. – Requirements will still be important but there is now an opportunity to augment requirements with more extensive and formal designs.• Small and technology development Projects may need a lighter weight DOORS-based process. Aimed at• JPL is working on more partnership projects. Module exchange needs to be influencing, negotiated and piloted early. sharing Lessons• We are piloting a Requirements Architecture, table-top review that will Learned and occur before Project and System requirements are developed. emerging best practices.• We have begun teaching just-in-time DOORS and process classes directly to and with Flight Projects. 5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 30
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×