How DOORS Helps JPL Get to Mars & Beyond

How DOORS Helps JPL Get to
Mars and Beyond

Best Practices in Metrics,
V&V and Traceability
Margaret H. Smith
Jim M. Grimes
Ross M. Jones
Trisha Jansma

Jet Propulsion Laboratory
California Institute of Technology

Outline

Overview of DOORS Use at JPL
Topic 1 Metrics
Topic 2 Verification and Validation (V&V)
Topic 3 Traceability

5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 2

JPL use of DOORS at JPL
Statistics
– 800+ users
– 600+ users have taken our in-house DOORS training.
– 50 DOORS licenses
– 50 projects have used DOORS, 22 are active
– 15 years using DOORS, preceded by 8 years using in-
house TRACER tool.
– 5 years of a standard process with in-house training
– 2 System Analysts for daily project support
– 1 Full-time Equivalent (FTE) Systems Engineer for
defining the method, teaching and project
consultation


DOORS Provenance – According to JPL

Source code and User Guide were QSS Telelogic IBM Rational
distributed from NASA COSMIC site from DOORS DOORS DOORS
1990 to ~1995. Developed by the Cassini
Project and used by JPL and ESA (1996) (2000) (2008)

JPL JPL TRACER DOORS
TRACER mothballed adopted at
(1988) (1995) JPL(1996)

Sample Project Sizes
installed
Project # objects # links # Modules TTT website
size (GB)
DSN 98995 5709 125 1.9 Large legacy infrastructure project
SMAP 17278 13311 91 1.9 Medium complexity earth orbiting mission
GRAIL 36031 30650 123 2.7 Smaller complexity lunar mission (JPL managed)
SES_JUNO 84844 43526 227 3.6 Large complexity deep space mission (JPL managed)
MSL 25641 5808 106 1.2 Flagship mission

# Modules Measurements were generated by
the JPL Trace Tree Tool (TTT)
0 50 100 150 200 250

MSL

SES_JUNO
# links
GRAIL # objects
# Modules
SMAP

DSN

0 20 40 60 80 100 120
# Links, Objects (Thousands)
Note: Number of objects includes all objects in formal modules: table cells,
headers, regular objects with or without text or OLE content. The number of
requirements is about one-third the number of objects.

Why Have a Standard Process?
• DOORS is a general purpose tool and there are many possible ways
to use it.
• For the first 15 years we used DOORS, each new project reinvented
the wheel.
– Big projects could afford to invent their own method and have internal support.
• We learned a lot from what the big projects did wrong and did right.
– We now have more small projects and fewer big projects.
• Small projects cannot afford to reinvent. Need a ready-made solution.
• The most common mistake: trying to run the old document-centric
process in DOORS and not taking advantage of the tool’s
capabilities.
• Projects were making mistakes, blaming DOORS, and delaying
import of requirements into tool as late as possible.
– Managing requirements in spreadsheets and not getting the benefits of early
sharing and collaboration.


What Has and Hasn’t Been Standardized?
Has
• Ownership, allocation and negotiation mechanism
• Attributes and their definitions – for requirements and V&V planning/tracking
• Views – requirements traceability, allocation status, requirements to V&V traceability
• High-level requirements template, boilerplate material
• Scripts – Requirement Maturity Metrics, and V&V Burndown and Statistics, and numerous housekeeping
scripts used by DOORS System Analysts (SAs).
• Tools – Trace Tree Tool for graphically displaying individual requirements traceability and project
requirements flow-down.
• Practices – the JPL Systems Engineering practices that culminate in requirements.
• Licensing and Support – the JPL institution purchases DOORS licenses for use by all projects and funds two
Systems Analysts (support).
• Training/Deployment – Engineers who are involved in requirements work attend a JPL developed course
that teaches basic DOORS skills and the JPL standard method for development requirements and tracking
verification in DOORS.
Hasn’t
• Requirements levels and element decomposition
• Detailed requirements templates
• DOORS collaboration method with partners and subcontractors

We owe the success of the JPL Institutional Requirements and V&V Tracking process and training
program to Richard Stoller (retired), and Karen Boyle (who sadly passed away in February, 2011).


Topic 1 Metrics

• Maturity Definition:
+ V&V Method/Approach defined
+ Rationale defined
+ Linked to parent(s)
+ Links exist to this requirement from Satisfiers (including V&V activities)
+ Satisfier (lower level owner) accepts allocated requirement
– TBD, TBR, TBCs

• Requirement Maturity Metric script:
– Implemented in DOORS Extension Language (DXL)
– Generates per module statistics across all modules in a Project, one module per
row
– Produces .csv file of statistics.
– Project can customize output to suit their Project Management/reporting needs.
– Provides a quick summary of requirement progress and facilitates comparison
between modules.


Example
Unknown Allocations
capability? Gold undecided or under
plating? negotiation

3
219


What Has Changed Today?
Daily email to Project’s requirements owner
Assessing Requirements activity and churn

Module name

Requirement number

Type of change
Requirements are marked Modified if any
attribute, with the setting Affect change dates
turned on, has been changed since the last time
this report was generated.


Topic 2
Verification
Planning and Tracking

Other benefits:
• JPL Projects have • Lower tooling cost–
use existing
been recording
institutionally
Verification supported tool and
information in support
DOORS for the last
10 years. • Enables useful views of
requirements, verification
• Initial benefit -- requirements planning & status, and
changes were immediately visible verification events.
to engineers responsible for • Projects can use as-is or customize a
verifying requirements because generic DXL script to generate
verification activity artifacts and verification statistics and graphics.
verification events were now
linked directly to requirements • Saves projects money through reuse of
tooling and method.


Verification Process
Project
Verification
Planning

Get DOORS Verification Verification Verification
Training / Activity Activity Status
consulting Planning Execution Reporting

Verification Execute VA
Activity Groups procedure, analyze Generate V&V
Requirements defined results and determine burndown charts
Development status and statistics and
present at
Reqmts to VA Record status of management
mapping the VA (procedure) meetings
captured run and create
Develop V&V
method, venue problem/failure
and approach reports (if needed)
Schedule VA Take corrective
activity action (if
completion (need necessary)
by) date
Cognizant person
approves VA
execution result
Plans and procedures
developed outside
DOORS and hyper- Cognizant person
linked from the DOORS approves that the
VA requirement has
been verified

Verification Elements
Definitions
VI – Verification Item, is anything that needs to be verified. All requirements
become Vis when they are being verified.

VA – Verification Activity. The testing activities described in a single Test
Plan, Procedure or Analysis. A group of associated VAs are a VA Group (VAG) Test Data Directory
1
VE – Verification Event. A run or execution of a VA.

1
0..1 Test
Test Plan Activity Report (AR)
1
1..M Procedure
0..1 1
In Other Tools 0..M
In DOORS
1 1 1
1..M 1
Verification Item (VI) Verification Activity (VA) Verification Event (VE)
1..M 1..M
VE data elements originate in the AR
Verification Activity and are pushed to VE to complete
the status picture.
Group (VAG)

Verification Attributes
Test Data Directory
1

Test Plan 0..1 Test Procedure 1
• version 1..M • version Activity Report (AR)
0..1 0..M
1 1
VA 1
VI or ITL item • activity name or activity 1
•object identifier (ID) group name
•statement of requirement • VA priority VE
•rationale • VA status • VA id (a reference)
•…. 1..M • VA owner 0..M • …. data imported from AR
•method 1..M • venue • url of Activity Report
1
•verification approach • date scheduled (for • url of test data directory
•audit status completion)
•audit completed on (date) • date completed
•audited by • plans and procedures
• PFRs

Requirements
V&V Activity Planning V&V Activity Execution
Development

V&V Status Reporting


Project Module Configurations
VIM – Verification Item (Requirements) Module
VAM – Verification Activity Module
VEM – Verification Event Module

VIM VAM
Multiple VAMs reflecting the
VIM VIM VAM VAM VEM Project’s requirements
structure or V&V organization.
VIM VIM VIM VAM VAM VAM

VIM
A single VAM for the Project’s
complete requirement tree
VIM VIM VAM VEM • partition VAM to give owners
granular write access
VIM VIM VIM

VIM
Combined VAM and VEM for
projects that only need
VIM VIM VAM / VEM information on the latest
run/execution.
5/24/2012
VIM VIM VIM Caltech – Jet Propulsion Laboratory M. Smith - 15

Verification Item (VI) Burndown
250

Current date
Not Confirmed
Number of Items to be Verified

200
Actual
(auto status)

A pick list allows selection of
150
any group of VIMs for this
chart.

100 Scheduled

Not Scheduled
50

0
4/1/2008
1/8/2008

2/5/2008

3/4/2008

7/8/2008

8/5/2008

9/2/2008

12/9/2008

1/6/2009
1/22/2008

2/19/2008

3/18/2008

4/15/2008

4/29/2008

5/13/2008

5/27/2008

6/10/2008

6/24/2008

7/22/2008

8/19/2008

9/16/2008

9/30/2008

10/14/2008

10/28/2008

11/11/2008

11/25/2008

12/23/2008
Count of VIs (modules included selected from pick list):
Scheduled
Scheduled = total number of VIs remaining to be completed per schedule
Not Completed = total number of VIs remaining to be completed actually
Actual Interval End Date
Not Confirmed = total number of VIs remaining to be Confirmed-Final
Not Confirmed
Not Scheduled = total number of VIs remaining to be completed but not scheduled

5/24/2012 Caltech – Jet Propulsion Laboratory R. Stoller, K. Boyle - 16

Verification Activity Burndown
16
Actual
14
Number of Activities to be Verified

12
Scheduled
10

8 Not Scheduled
6
A pick list allows selection of
4 any group of VAs for this
chart.
2

0

Scheduled Scheduled - Count of notof VAs remaining to be completed per schedule
= total number completed linked VAs scheduled to be completed
Actual Not Completed - Count ofof VAs remaining to be completed actually
= total number not completed linked VAs remaining to be
completed (Actual)
Not Scheduled = total number of VAsnot scheduled completed and are not
Not Scheduled - completed-NF or that are not
scheduled


Verification Item Statistics "Bad" if > 0
"Neutral"
"Good"
from V&V Burndown Script that generates data to Excel "Normal"

Count of VIs in VIMs (shall statements only)
A: B: C: D: E: F: G: H: I: J: K: L M: N:

VIs VIs VIs VIs Not VIs VIs Not VIs Not VIs Not VIs Not V&V Engr
VIM Linked to Linked to Linked to VIs Auto VIs Conf Linked to Linked to Linked to Auto Conf Verif and Responsible
NAME Total VIs VAM VAGs VAs Verif Verif VAM Reject a VA Verif Verif Late for this VIM

VIM1 625 600 15 575 200 150 25 10 50 425 450 5 John
Jones

VIM2 500 500 10 485 100 75 0 5 15 400 etc 5 Susie
Jones

VIM3 1000 1000 0 1000 0 0 0 0 0 1000 etc 0 Mary
Smith
… … … etc etc etc etc etc etc etc etc etc etc

VIM n 200 200 5 190 25 5 0 5 10 175 etc 0

Total 2325 2300 30 2250 325 230 25 20 75 1975 etc 10

Relationships: B=C+H; I is not included in D; E=C-I-D; J=B-E=H+I+D; K=B-F; L=B-G
M = VIs not auto verified AND scheduled completion date is missed or has not been scheduled or is not
linked. Any object containing the word “shall” enclosed in quotes (“shall”) is not included in any statistic.

Verification Activity Statistics "Bad" if > 0
from V&V Burndown Script that generates data to Excel "Neutral"
"Good"
"Normal"

# of VI Are VAs Fully ** Is there a VA Are there any Schedule Days
VA Parent Linked to VIs VA Sched- Plan &/or a Suspicious VI Remaining (if not V&V Engineer /
NAME* Links ? VA Status uled? Procedure? Links? completed) Owner of VA

VA-x 25 Yes Blank 08/10/09 No Yes 10 Jack Jones

VA-y 20 Yes Completed-NF 07/26/09 Yes No -5 Mary Jones

VA-z 100 Yes Completed No Yes No Susie Que

… 350 … Blank No No No Not Sched Tom Smith

VA-n 55 Yes Blank 08/20/09 Yes Yes 20 Susie Smith

* VAs are organized in VAM under VAGs and sub-header VAGs, but tabulated separately here.
** Identifies if there is an http entry in the Plan and Procedure attribute.

5/24/2012 Caltech – Jet Propulsion Laboratory R. Stoller, K. Boyle -19

Topic 3 DOORS Trace Tree Tool (T3)
JPL has parent-to-child requirements relationships where a parent has 1 to tens of
children, and links between levels can be many-to-many. We have trouble seeing the
forest for the trees and navigating requirements flow within DOORS.

• Primary Motivations
– Provide JPL users with an alternative graphical presentation of
requirement flow (links)
– Allows navigation of requirements based on hierarchy or flow
– Enables seamless navigation between modules following flow-
down/links
– Provides easy access via Web-based, platform independent interface
• Secondary Motivation - evolved with implementation
– Provide an indication of timeliness of requirements flow-down
– Provide an efficient way to checkout verification planning
• History
– Conceived and developed by a JPL Software Systems Engineer (James
Grimes) in 2006
– Enthusiastically used by 8 JPL Flight Projects

5/24/2012 Caltech – Jet Propulsion Laboratory J. Grimes - 20

Trace Tree Tool View – Example Index Page

ProjX
ProjX
ProjX
ProjX
ProjX
•••


Project Map

Number of links/flowed-down requirements appear on edges


Module Page
ProjX

•••

…….

Requirement Page

…………………. ……..

Attributes continue below

Object Text Graphic

………………………
………….. ………………………
……………………… ………….. ……………………….
……………….. ………………..
………….. ………………………



Verification Approach Graphic

……………
……………………………
……………………………
……………………………
……………………………

……………
……………………………
……………………………
……………………………
……………………………
……



Typical DOORS T3 User Scenarios
• Quickly obtain the contents of a requirement with a known ID
– Main page Module page Requirement number
• Scan a particular Project module for a specific requirement
– Main page Module page, then
• Scroll up and down at web browser speeds
• Use browser’s find capability (not as flexible as DOORS, but faster)
• Trace requirements flow-down starting at any requirement
– Click on boxes in graph going up- or downstream in requirements flow
– Go from there to text, attributes
• Obtain immediate visual indication of timeliness of flow-down
– E.g., has upstream requirement X been updated more recently than my
requirement? If yes, then it’s red. Actual date is shown on X’s page.
– Is flow-down complete? Does the requirement have a parent?
• Provide supportive evidence during Project Reviews
– Quickly show existence of flow-down
– Quickly show V&V attribute, implementation status (if in attribute)

DOORS Trace Tree Tool (T3)
High-Level Process/Execution Flow

5/24/2012 Caltech – Jet Propulsion Laboratory T. Jansma - 28

Development Tools Used
• DXL – DOORS eXtension Language, a DOORS proprietary C-like, object-
oriented language available within the DOORS client
• Perl – a scripting language widely available on UNIX, Mac OS X, Linux,
Windows
• Graphviz – open source Graph Visualization software developed at AT&T
that includes the Dot language and dot program for drawing directed
graphs
• XML and Perl/C-based XML parsing software – open source software
• Java for processing images
• Standard UNIX utilities such as date, rm, ls, rsync, etc.
• SVN – Subversion open-source version control tool


JPL’s Direction with DOORS
• Systems Engineers on JPL Projects are starting to infuse Model Based
Systems Engineer / SysML.
– Modeling includes architecture, behavior and resources.
– Many pockets of discipline specific modeling tools exist in the Spacecraft
subsystem organizations
– Vision is to use a central, standards based repository to interconnect and
exchange model data.
– Requirements will still be important but there is now an opportunity to
augment requirements with more extensive and formal designs.
• Small and technology development Projects may need a lighter weight
DOORS-based process.
Aimed at
• JPL is working on more partnership projects. Module exchange needs to be influencing,
negotiated and piloted early. sharing
Lessons
• We are piloting a Requirements Architecture, table-top review that will Learned and
occur before Project and System requirements are developed. emerging best
practices.
• We have begun teaching just-in-time DOORS and process classes directly
to and with Flight Projects.


How DOORS Helps JPL Get to Mars & Beyond

Recommended

Recommended

More Related Content

Similar to How DOORS Helps JPL Get to Mars & Beyond

Similar to How DOORS Helps JPL Get to Mars & Beyond (20)

More from Bill Duncan

More from Bill Duncan (20)

Recently uploaded

Recently uploaded (20)

How DOORS Helps JPL Get to Mars & Beyond