How Rocket Scientists Do It

  • 433 views
Uploaded on

This article is a summarized version of the process that NASA uses to develop mission critical software.

This article is a summarized version of the process that NASA uses to develop mission critical software.

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
433
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
7
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Tony DaSilva : How Rocket Scientists Do It This page last changed on Aug 04, 2008 by tonyd. quot;Okay, so you're a rocket scientist. That don't impress me much. So you got the brain but have you got the touch.quot; - Shania Twain Introduction • Process Recommendation Highlights • Process Context And Overview • The People • The Requirements Definition Phase • The Requirements Analysis And Specifications Phase • The Preliminary Design Phase • The Detailed Design Phase • The Implementation Phase • The System Testing Phase • The Acceptance Testing Phase • The Maintenance and Operations Phase • Keys To Success • Dos And Don'ts • Introduction Unlike the IEEE and SEI organizations, NASA'a Goddard Space Center builds large, mission critical, computationally intensive, real-time software. Thus, I listen more closely, and with a less skeptical attitude, when they speak about SW development. Via the beauty of web push technology, I recently stumbled upon this freely available NASA document: quot;Recommended approach to SW developmentquot;, NASA Software Engineering Laboratory (SEL), Goddard Space Center 1992. I would have uploaded it and attached it to this Wiki page, but it exceeds the 10MB attachment size limit. However, in case you're interested in reading it yourself, here's the external link to it: SEL SW Dev Approach. The purpose of this article is to review and summarize the SEL recommended approach to SW development as described in their document. Full Disclosure The content of this article below the box that your reading is colored by my personal interpretation of the SEL's intent and meaning. Words in quot;quotesquot; were lifted directly from the document. I tried to bend their advice to fit our ATS application domain and I also tried to map their language into the language that we (sort-of) speak in ATS. If you'd like, read the document yourself to form your own opinions. We can then compare notes The date that the SEL document was published, 1992, shows that it isn't very current. Usage of the terms quot;VAXquot;, quot;FORTRANquot;, quot;Adaquot;, quot;JCLquot;, quot;DCLquot;, and my favorite, quot;System Delivery Tapequot; may trigger a light- hearted chuckle. However, aged material aside, it's seriously chock full of timeless insights and golden nuggets of wisdom. The process described in the document is founded on data collected from over 100 SW development projects. SW product sizes ranged from 30 KSLOC to 300 KSLOC. The SEL calls the application domain Document generated by Confluence on Aug 04, 2008 09:17 Page 1
  • 2. covered by their SW products flight dynamics mission SW. Like our SW products, failure of their SW can lead to, or directly cause, large financial loss and/or loss of human life. One of the great attributes of the document is that it is totally self-contained. The recommended artifacts are not spread out all over the place among different documents and difficult to link/associate together. Definitions and descriptions of practices and methods and tools and roles and phases and review templates and document templates are all cohesively located in this one 228 page document. Process Recommendation Highlights After reading the entire document, I went back over it and cherry-picked (<--- that's for you Phil ) the concepts and ideas that resonated with me. Here's the list: • The SEL's guiding philosophy is: quot;.. Intellectual control and reliance on the human mind to build a quality product the first time.quot; This nearly 20 year old philosophy is consistent with one of the agile manifesto's four main tenets: we value quot;Individuals and interactions over processes and toolsquot;. • Out of the 8 phases in the recommended process, there are TWO requirements phases: A Requirements Definition phase, and a Requirements Analysis & Specifications phase. Unlike every other process definition that I've ever seen (obviously, I haven't seen them all), this fact drives home the importance of getting the requirements right early in the process for real-time, mission critical, SW systems. This characteristic clearly flies in the face of modern agile approaches that recommend against BDUF (Big Design Up Front) and advise getting to working code ASAP. However, considering the class and application domain of SW that NASA develops, and the consequences of failure, I'm on board with the SEL's 2 requirements phase recommendation. • When Management team responsibilities are defined in each phase, the tasks are not just defined as quot;status takingquot; and quot;schedule hawkingquot;. There's technical content too: ° quot;Both the Management team and the Development team review the test results to ensure that all discrepancies are identified and corrected.quot; ° quot;All build test plans are reviewed for correctness and completeness by the Management team.quot; ° quot;Prepare a preliminary build plan.quot; ° quot;Reviews technical productsquot; • Training is recommended LOTS of times throughout the document. • The SEL maintains and uses a quot;project historiesquot; database to store and reuse lessons learned. The SEL database was one of the main drivers behind the process document being reviewed in this article. • The SEL classifies requirements into 5 types: mandatory, requires review, needs clarification, information only, and TBD. The numbers of these metrics (especially TBDs) are tracked rigorously and measured frequently throughout the project. metrics are used as prime indicators of project and product health. • The importance of pragmatic, down-to-earth, tailorability is expressed in a section right up front. The number and formality of reviews, the number of documents produced, and the level of detail provided in each document should vary depending on the size and complexity of the SW, and the extent of the modifications that need to be made to existing SW. • The SEL doesn't mandate either Object Oriented Analysis and Design methods nor Structured Analysis and Design methods. They leave it up to the team to decide which method to employ. They do, however, recommend not mixing the two. quot;The analysis methodology for the project should be appropriate to the type of problem the system addresses.quot; • The SEL approach uses quot;Requirements Question and Answerquot; forms as one of the important metrics to track the health of the product and project. The requirements definition team quot;responds to ALL developer questionsquot;. • Separate System Test and Acceptance Test teams are identified. The System Test team is comprised of SW developers. The Acceptance test team is comprised of analysts and Requirements Definition team members. Document generated by Confluence on Aug 04, 2008 09:17 Page 2
  • 3. • Metrics are a highly important feature of the SEL SW development process. They are tracked, evaluated, and updated throughout the project as objective indicators of how well the team and the product are progressing. A sample metrics list is given as: ° Number of Requirements changes/modifications ° Number of Requirements additions ° Number of TBDs ° Number of Requirements questions ° Number of Requirements answers ° Number of planned SW units ° Number of designed units ° Number of reviewed units ° Number of planned tests ° Number of tests executed ° Number of tests passed ° Number of defects reported ° Number of defects resolved Process Context And Overview The organizational context in which flight dynamics SW is developed is shown in the figure below. Computationally intensive mission support SW (orbit determination, orbit adjustment, altitude determination, manuever planning) is engineered in the Flight Dynamics Facility. The Systems Technology Laboratory develops simulators, embedded systems, flight dynamics utilities, and SW that supports advanced system studies. The 8 phase SEL SW development life cycle is illustrated below. Notice that, at any given point in time, multiple activities are being executed in parallel. Requirements definition activity continues throughout the process. In order to bound the project and control scope creep, there's early baselining of requirements. Cllear entry/exit criteria are defined for each phase, but there's no quot;pens down, we're done with requirementsquot; proclamations to convey a false sense of cosmetic security to stakeholders. Each phase is defined by • Entry & exit criteria • Output products/artifacts Document generated by Confluence on Aug 04, 2008 09:17 Page 3
  • 4. • Metrics collected • Methods & tools • Key activities, and who is responsible for doing them The Maintenance & Operation phase, like in most process descriptions, is (curiously) not addressed in the document. The SEL doesn't use the DoD CSCI/CSC/CSU terminology that we internally use in ATS to characterize the design of their SW. They use subsystem levels (1..N), packages, and units to document their designs. A Unit is flexibly defined as: quot;any set of program statements that are logically treated as a whole. A main program, a subprogram, or a subroutine may each be termed a unit.quot; A module is defined as a collection of logically related units (typically 5 to 10 units). A component is the most abstract concept that the SEL uses for describing the SW. It is defined as quot;any constituent partquot;. Each reviews for each phase is specified in terms of its format and its hard copy materials content. Presenters, participants, schedule, agenda, and materials distribution characterize the review format. Content templates describe the hard copy review material. The People In the SEL process, the following individual roles are defined: 1. SW engineer 2. Analyst (determines detailed requirements AND performs acceptance tests). 3. Project manager (allocates/tracks resources, performs active monitoring, acts as a technical consultant). 4. Task Leader (provides technical direction and daily technical oversight). 5. Application Specialist(developer who has engineered similar SW previously and also understands the complex mathematics and physics of flight dynamics). 6. QA representative 7. Project librarian In addition to individual roles assigned to specific people, the following group roles are defined: 1. The Requirements Definition team 2. The SW Development team 3. The Management team 4. The Maintenance and Operation group 5. The Configuration Control Board (CCB) 6. The System Test team 7. The Acceptance Test team A typical Requirements Definition team is composed of Flight Dynamics analysts and application specialists. A typical SEL SW Development team is composed of: a project manager, a task leader, N developers, K application specialists, a QA representative, a project librarian, and a CCB. A typical Acceptance Test team is comprised of analysts and Requirements Definition team members. A typical System Test team is comprised of members of the SW Development team and J analysts. The Requirements Definition Phase quot;The purpose of the Requirements Definition phase is to produce a clear, complete, consistent, and testable specification of the technical requirements for the product.quot; The major actors in this phase are: • The Requirements Definition team. The team collects and itemizes all high level requirements. It develops the operational concept. The Requirements Definition team develops system concepts, defines detailed requirements, defines all external interfaces, and derives specifications. The system functions and algorithms that meet each detailed requirement are defined and specified, and the overall architecture concept is defined. Document generated by Confluence on Aug 04, 2008 09:17 Page 4
  • 5. • The Management team: The team develops a plan for this phase, staffs and trains the Requirements Definition team, and interfaces with the customer. The major output artifacts of the Requirements Definition phase are: 1. The System and Operations Concept (SOC) Document 2. The Requirements and Specifications Document in 3 parts: a. Detailed requirements b. Functional or Object Oriented Specifications (data in, data out, in-to-out transformation steps). c. Necessary mathematical background information The reviews executed in this phase are: The System Concept review (SCR) and The System Requirements Review (SRR). The tools employed to develop the outputs are selected from this set: Structured Analysis, Object Oriented Analysis, Walk-throughs, inspections, and prototyping (to resolve requirements issues). The Requirements Analysis And Specifications Phase quot;The purpose of the Requirements Analysis phase is to ensure that the requirements and specifications are feasible, complete, and consistent, and that they are understood by the development team.quot; The major actors in this phase are: • The Requirements Definition team. The team continues to resolve ambiguities, discrepancies, and TBDs. It responds to all developer questions. • The SW Development team. The team analyzes and classifies requirements. It produces the Data Flow Diagrams or Object Oriented diagrams that characterize the required behavior of the product. Structured or Object Oriented analysis is used to clarify & amplify the behavioral requirements. The team scrutinizes the Requirements and Specifications document for omissions, contradictions, and TBDs. • The Management team: The team prepares the SW Development Plan, staffs and trains the SW Development team, facilitates requirements issue resolution, and reviews technical products. The major output artifacts of the Requirements Analysis And Specifications phase are: 1. The Requirements Analysis Report 2. The SW Development Plan 3. A Requirements and Specifications artifact update 4. Prototyping plans. The Review that takes place in this phase is the SW Requirements and Specifications Review. The tools employed to develop the outputs are selected from this set: Walk-throughs, Requirements Q & A forms, requirements classification, project library, and prototyping. The Preliminary Design Phase quot;The purpose of the Preliminary Design phase is to define the high level SW architecture that will best satisfy the requirements and specifications for the system.quot; The major actors in this phase are: • The Requirements Definition team. The team continues to resolve requirements issues, answer developer questions, and it participates in SW design walk-throughs. • The SW Development team. The team defines the high level functions or objects in the system, prepares the preliminary design report, conducts the preliminary design review, and develops scaffolding skeleton code for prototyping and performance analysis. Document generated by Confluence on Aug 04, 2008 09:17 Page 5
  • 6. • The Management team. The team reassess schedules/staffing/training needs, and controls requirements changes. The focus switches from planning to controlling. The team prepares a preliminary incremental build plan and establishes the independent Acceptance test team. The major output artifacts of the Preliminary Design phase are: 1. The Preliminary Design Report 2. The System Test Plan The Review that takes place in this phase is the Preliminary Design Review. The tools employed to develop the outputs are selected from this set: Walk-throughs, design inspections, prototyping, functional decomposition, object oriented design. The Detailed Design Phase quot;The purpose of the Detailed Design phase is to produce a completed design specification that will satisfy all requirements for the system and that can be directly implemented in code.quot; The major actors in this phase are: • The Requirements Definition team. The team continues to resolve requirements issues, answer developer questions, and it participates in design walk-throughs. • The SW Development team. The team prepares detailed design diagrams, conducts walk-throughs, prepares the detailed design document, and conducts the Critical Design Review. • The Management team. The team assess lessons learned from the previous phases, controls requirements changes, and prepares the build plan. • The Acceptance test team: The team begins work on the acceptance test plan and the build 1 test plan (the order in which units should be implemented & integrated, capabilities in the build). The major output artifacts of the Detailed Design phase are: 1. The Detailed Design Report 2. The Build Plan 3. The Build Test Plan The Review that takes place in this phase is the Critical Design Review. The tools employed to develop the outputs are selected from this set: Walk-throughs, design inspections, functional decomposition, object oriented design. The Implementation Phase quot;The purpose of the Implementation phase is to build a complete, high-quality SW system from the blueprint provided in the detailed design document.quot; The major actors in this phase are: • The Requirements Definition team. The team continues to resolve requirements issues, answer developer questions, and it participates in the Build Design reviews. • The SW Development team. The team codes new units and revises existing units. It tests and integrates units, conducts the build tests, prepares the System Test plan from the Requirements and Specifications artifact, and drafts the User's Guide. • The Management team. The team reassess schedules/staffing/training, and controls requirements changes. • The Acceptance test team: The team completes the draft acceptance test plan. The major output artifacts of the Implementation phase are: Document generated by Confluence on Aug 04, 2008 09:17 Page 6
  • 7. 1. The Source code 2. Build test results 3. The System Test plan 4. Draft user's guide The Review that takes place in this phase is the Build Design Review. The tools employed to develop the outputs are selected from this set: code reading, unit testing, integration testing, build testing, source code configuration management. The System Testing Phase quot;The purpose of the System Testing phase is to verify the end-to-end functionality of the system in satisfying all requirements and specifications.quot; Discrepancies found during testing are classified as either code, design, or requirements errors. In addition, errors are prioritized into 3 levels of severity:L1, L2, L3. In the SEL process, the documents are fixed first, and then the code is fixed. The major actors in this phase are: • The SW Development team. The team verifies end-to-end functionality against the Requirements and Specifications document according to tests recorded in the System Test plan. The team also identifies SW units and modules for reuse and develops the System Description document. • The Management team. The team reassess schedules/staffing/training, controls requirements changes, and conducts configuration audits. • The System Test team: The team executes the tests in the System Test plan, analyzes and reports the test results, and evaluates the User's Guide. • The Acceptance Test team: The team finalizes the acceptance test plan. The major output artifacts of the System Testing phase are: 1. The tested source code 2. The System Description document 3. The Acceptance Test plan 4. User's guide The Review that takes place in this phase is the Acceptance Test Readiness Review. The tools employed to develop the outputs are selected from this set: regression testing, source code configuration management, discrepancy reports, configuration audits. The Acceptance Testing Phase quot;The purpose of the Acceptance Testing phase is to demonstrate that the system meets its requirements in its operational environment.quot; Testing is performed on the operational equipment as opposed to a lab environment. No new requriements or enhancements can be accepted during the Acceptance Testing Phase. Five levels of test disposition are defined: Cannot be evaluated; Failed and no workaround exists; Failed but workaround exists; Cosmetic errors; Passed. Important Quality Information Measured By The SEL As of 1992, the SEL measured an average error density of 4.5 errors/KSLOC at the end of Acceptance Testing. Furthermore, they decomposed this number further into; 2.6 errors/KSLOC at the end of the Implementation phase; 1.3 errors/KSLOC at the end of the System Testing phase; and .6 errors/KSLOC at the end of the Acceptance Testing phase. The SEL also found that error detection rates decreased by 50% from phase to phase. The major actors in this phase are: • The SW Development team. The team trains the Acceptance Test team in how to use the system. It also corrects discrepancies found during testing. Document generated by Confluence on Aug 04, 2008 09:17 Page 7
  • 8. • The Management team. The team ensures the quality and progress of the testing and starts preparing the SW Development History. • The Acceptance test team: The team executes the tests in the Acceptance Test plan, analyzes and reports the test results, and formally accepts the system. The major output artifacts of the Acceptance Testing phase are: 1. The system source code and document set 2. The finalized System Description document 3. The finalized User's guide 4. The Acceptance test results 5. The phase by phase SW Development History (no later than 1 month after formal system acceptance). The tools employed to develop the outputs are selected from this set: regression testing, source code configuration management, discrepancy reports. The Maintenance and Operations Phase In their process document, the SEL, like most other process sponsors, doesn't define the details of the Maintenance and Operations Phase. They state that the overall recommended approach defined in the previous sections of the document can be tailored to apply to this phase. Keys To Success At the tail end of the SEL SW process document is what I think is the most valuable section. It is appropriately named quot;Keys To Successquot;. Based on their extensive experience, here are their insights: 1. Understand your environment. The environment can be expressed as; the nature of the problem to be solved, the limits and capabilities of your staff, and the supporting SW and HW infrastructure in your organization. 2. Match the process to the Environment. Tailor the process and ensure that the activities have a rationale and can be enforced. If you cannot or won't enforce a behavior/activity, then don't superficially include it in the plan. Make metrics collection easy, and an integral part of the process. 3. Experiment to improve the process. Stretch it a little at a time to improve it continuously. Employ new techniques and extensions, cut out the fat, and evaluate the impact of the change as objectively as possible. Don't change too many things at once. 4. Don't attempt to use excessively foreign technology. Do not select and attempt a significantly different technology just because it was successful in other situations. The technology must fit the local environment and culture. Dos And Don'ts In addition to the quot;Keys to Successquot; section, the SEL has arrived at a set of 9 Do and and 8 Don't recommendations to facilitate success. I've already listed these in a separate Wiki page here, but I think they are so valuable that I'm repeating them below: 1. Do adhere to a quot;livingquot; Software Development Plan. Update it frequently, at least once per phase. 2. Do empower project personnel. Clearly assign responsibilities and decision-making authority to specific roles, and assign specific people to those roles. 3. Do minimize bureaucracy. Establish the minimum documentation level and meeting frequency necessary to communicate status and decisions to all stakeholders. Resist the temptation to address difficulties by adding more meetings, documents, and tightening management control. quot;More meetings plus more documentation plus more management does not equal more success.quot; 4. Do establish and manage the software baseline. Prioritize and track the number of TBDs, number of Requirements & Specifications, and determine schedule/cost impacts throughout the project. 5. Do take periodic snapshots of project health and progress and don't be afraid to replan. Adjust the scope, staffing, schedule when necessary. 6. Do re-estimate size, staffing, and schedules regularly (we recommend monthly). More information, experience and knowledge is acquired as the project progresses and ambiguity decreases. 7. Do define and manage phase transitions. Publish clear and unambiguous entry/exit criteria. Document generated by Confluence on Aug 04, 2008 09:17 Page 8
  • 9. 8. Do foster a team spirit. Maximize commonality and minimize differences between the staff. Establish a common language specific to the project. Cross-train. Help and applaud each other. 9. Do start the project with a small senior staff. This group should establish the approach, priorities, and organize the work. In addition the group should establish reasonable schedules. 1. Don't allow team members to proceed in an undisciplined manner. Provide training on specific methods and techniques. 2. Don't set unreasonable goals. This behavior is worse on morale than not setting any. Don't hold personnel accountable for impossible commitments. 3. Don't implement changes without assessing their impact. Assess cost, schedule and technical risk. Little changes add up over time. 4. Don't gold plate. Implement only what is required. 5. Don't overstaff, especially early in the project. Bring on additional staff only when there is useful work for them to do. 6. Don't assume that an intermediate schedule slippage can be absorbed later. It may be absorbed if the right decisions are made, but don't assume it to be absolutely true. 7. Don't relax standards in an attempt to reduce costs/schedule. This behavior lowers the quality of intermediate products/artifacts and leads to more rework downstream. It also sends the message that schedules are more important than quality. 8. Don't assume that a large amount of documentation ensures success. Determine the level of formality and amount of detail required based on the size, life cycle duration, and lifetime of the system. Document generated by Confluence on Aug 04, 2008 09:17 Page 9