Technical documentation process for a Northrop Grumman platform

1. Service Design Case Study
September 2018
Jeff Onken Service Designer and UX Design Mgr
Mimi Chen UX Design Mgr
Ben Schwartz Human Factors Engineer
Doug Radecki UX Designer
Jennifer VanBerschot Behavioral Research Scientist
Platform Technical Documentation Process
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

2. Agenda
1. Background on UX and Design Efforts
2. Root Causes of the Escapes
3. Metrics
4. Five Innovations that enable the To-be Process
5. To-be Process
6. Roles, Responsibility, Accountability, and Authority
7. Transition Steps
September 20182
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

3. What We Did
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

4. UX Design Application Overview
1. Background
– Customer identified escapes in recent technical manuals
– Kicked off user experience initiative
2. Define and research tech orders (TOs)
– Documented “As-is” process and identified pain points*
– Researched tech order impacts on operational
effectiveness
– Briefed UX and initial findings to build support and
advocacy
3. Service Blueprinting
– Week 1: Cross-function process improvement
workshop**
– Week 2: TO exchange meeting and follow-up interviews
4. Develop implementation roadmap
– Effectively build on near-term efforts
– Metrics to measure progress
4
“As-is”
Process
Pain
points
switch to
transformed
process
“To-be”
Process
kickoff
Near-
term
Actions
Transition
Roadmap
*https://hbr.org/1990/07/reengineering-work-dont-automate-obliterate
**http://www.practicalservicedesign.com/resources/ and http://www.practicalservicedesign.com/download-the-guide/
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18
September 2018

5. Design Effort To-Date
• Week 1: Cross-Function Process Improvement Workshop
Day 1: Fishbone diagram to explore root causes to specific escapes***
Day 2: How Might We (HMW) tool to create problem statements to generate ideas^
Day 3: Ideation and blueprinting^
Day 4: Ethnography with technical writers
• Week 2: TIM and Follow-Up Interviews
– Conducted additional interviews to fill gaps and gain input from other specialties
– Listened into to TIM to better understand customer perspective
– Analyzed findings to ideate potential new process concepts
– Created draft implementation roadmap
September 20185
***https://www.isixsigma.com/tools-templates/cause-effect/determine-root-cause-5-whys/
^http://www.designkit.org/case-studies/3
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

6. What We Found
September 2018
6
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

7. Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

8. Workshop Results
September 20188
Escapes Causes Identified Insights Summarized
Poor quality and
incomplete initial
document submissions
1. Budget constraints
2. Maint. manuals and flight manuals are completely different
3. Customer culture
4. Error reports are not user friendly
5. No input on needs during proposal
6. Alignment of source data to support module development
7. Don’t have the right source data at the right time
1. Don’t know which docs affected
by SOW
2. Experience is seen as a shortcut
for validation
3. Customer doesn’t pay for rework
4. Driven by schedule pressure
over quality
5. We keep making the same mistakes
6. Customer thinks we are further along
than we really are
7. We value job completion over quality
8. Customer compare tech pubs
with others
9. Driven by multiple customer
expectations
10.Process scalability hasn’t worked
11.Customer expectations need to match
(acquisition and operations)
12.Customer reviewer personnel
bottleneck
Schedule overrun at the
end of the period of
performance (POP)
1. Lack of understanding of the SOW
2. Reliance on tribal knowledge
3. Not enough budget for sufficient reviewing
4. Content interdependencies
Customer complaints of
quality after final
validation and
verification (V&V)
1. Lack of understanding how to create high-quality docs
2. IT needs improvement
3. Redoing work due to tools; capability between software files
4. Inconsistent with provided engineering notes
5. Attitude ‘easy thing at the end’
6. Could not return ‘the bid’
7. ‘Rubber stamp’ process -> passed to junior/new
8. Logistics summit
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

9. High-level Summary
1. We found that the Tech Order escapes are symptoms.
2. Robustly solving the escapes required re-engineering
the whole, end-to-end process.*
3. A transformed process enables us to achieve and sustain
high-quality deliverables and high customer satisfaction.*
4. Lack of collaboration is not a root cause of any problem.
And collaboration in general isn’t a solution.^^
September 20189
*https://hbr.org/1990/07/reengineering-work-dont-automate-obliterate
^^https://hbr.org/2011/12/eight-dangers-of-collaboration
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

10. Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

11. explore for
root causes
diagnose primary
problems
brainstorming
blueprinting
process steps
~280 causes 12 insights ~480 ideas 38 steps
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

12. What We Recommended
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

13. Five Innovations: “To-be” process enablers
1. Prioritize quality over schedule^^^
2. Clearly defined deliverable acceptance criteria^^^
– the Test that the team must pass, shared list of acceptance criteria, and checklist
3. Case management approach†
– “case manager,” the Test and its acceptance criteria, checklist, committed staff
4. Engineer and writer pairing to write procedures^^^
5. Supplier account management
– requirements flow-down, SOW, technical deliverables, and notifications
September 201813
^^^http://agilemanifesto.org/principles.html and https://www.scaledagileframework.com/test-first/
†https://www.commonwealthfund.org/publications/newsletter/case-study-aetnas-embedded-case-managers-seek-strengthen-primary-care
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

14. Five Innovations
1. Prioritizing Quality over Schedule
• Quality has taken a back seat to schedule
and budget, and this has shown up in the
CARs.^^^
• We need to commit, together with the
customer, to deliver when the quality is
good enough.
• And we define that by #2 . . .
2. Clear Acceptance Criteria
• Inspired by the software dev best practice of
Test-driven Development (TDD), we begin
our work by defining the test that the team
must pass.^^^
• Then we begin engineering by defining
acceptance criteria for our work, the todo
list of tasks for the team, and we can deliver
only when the test passes.
• We bundle this work together in #3 . . .
September 201814 ^^^http://agilemanifesto.org/principles.html and https://www.scaledagileframework.com/test-first/
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

15. Five Innovations
• 3. Case Management
• Organizing our work in cases
centered on delivering quality and
customer satisfaction: †
• the test that the team must pass
• acceptance criteria
• checklist (todo list)
• assigned staff
• Every ECP must have a
responsible “case manager”
5. Supplier Account Mgmt.
• Procedure writing is one,
specific place where people
working together is required for
a quality product.^^^
• Collaboration is a means to an
end, and writing the procedures
is one of the outcomes.
• Each ECP assigns an engineer
and a tech writer to be paired
up for the duration of the ECP.
September 201815
4. Paired Eng. & Writer
• Deliberate management of
supplier relationships ensures
clearer flow of expectations and
information in both directions.
• requirements flow down
• technical deliverables
• SOW
• DMS management and other
notifications
^^^http://agilemanifesto.org/principles.html and https://www.scaledagileframework.com/test-first/
†https://www.commonwealthfund.org/publications/newsletter/case-study-aetnas-embedded-case-managers-seek-strengthen-primary-care
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

16. Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

17. 1 2
Prepare
proposal
for Eng,
Pubs, and
3 4
1.1 1.2 1.3 1.4 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14 3.15 3.16 3.17 3.18 3.19 4.1 4.2 4.3 4.4 4.5
Commit to the
deliverable
being defined
as the a
complete
package that's
Commit to
delivery of the
whole
deliverable
when quality
criteria are
Commit
performers until
ECP case
completion and
delivery
Commit to ECP
case quality
checklists being
complete
before delivery
Conduct the
ECP kickoff,
chaired by the
ECP case
manager and
includes PM,
Conduct
external
stakeholder
program
review, inc. end
user
Assess
engineering
and product
support scope
Assess supplier
scope via
supplier
account
management
Reconcile the
differences
between
customer
acceptance
criteria for
Write the test
the team must
pass before the
proposal is
submitted
Decide quality
metrics that
holds
responsible
entities,
organizations,
Declare quality
dependencies
for each other
amongst
Engineering,
Product
Identify and
coordinate
dependencies
defined in the
Integrated
Master
Generate ECP
case quality
checklist that
describes the
work to be
performed to
Coordinate and
assemble
proposal inputs
for proposal
submittal
Conduct Task
Order kickoff
when work is
awarded
Initiate
configuration
branch after
kickoff
Organize work
in appropriate
configuration
branch
Schedule and
coordinate
dependencies
defined in the
integrated
master
Develop
Systems
Engineering
Management
Plan to
incorporate ECP
Design system
to meet
customer and
internal
acceptance
criteria for
Flowdown
supplier
requirements
and coordinate
documentation
Freeze
engineering
configuration
before
integrated end-
user acceptance
Conduct
Stakeholder
Gate Review
Perform
hardware
production
Pair co-creation
of technical
publications
Author
technical
publications in
compliance
with required
standards and
Raise the alarm
if schedule or
quality slips are
being realized
(Continous)
Assess for
impact if
engineering
changes occur
after
configuration
Verifiy and sign
off that supplier
products satisfy
the
requirements
Perform
engineering
verification of
satisfaction of
system
requirements
Perform
integrated flight
test of whole
deliverable
Perform end-
user V&V of
whole
deliverable in
users'
operational
Perform final
customer V&V
of the whole
deliverable
Handoff of
completed
package
Conduct case
retrospective to
encourage
learning and
continuous
improvement
Update
templates and
guides based
on retrospective
results
Conduct case
completion
celebration
Conduct case
closeout
Customer - SPO
(CTOCU,
TOMA)
Customer - SPO
(CTOCU,
TOMA)
Customer - SPO
(CTOCU,
TOMA)
Customer - SPO
(CTOCU,
TOMA)
Customer - SPO
(CTOCU,
TOMA)
Customer - SPO
(CTOCU,
TOMA)
Case Manager Case Manager Case Manager Case Manager Case Manager Case Manager Case Manager Case Manager Case Manager Case Manager Case Manager Case Manager Case Manager Case Manager Case Manager Case Manager Case Manager
Customer - SPO
(CTOCU,
TOMA)
Case Manager Case Manager Case Manager Case Manager Case Manager Case Manager Case Manager Case Manager
End User
Community -
ACC,
Warfighters,
Maintenance,
Sensor
Customer - SPO
(CTOCU,
TOMA)
Customer - SPO
(CTOCU,
TOMA)
Customer - SPO
(CTOCU,
TOMA)
Case Manager
Customer - SPO
(CTOCU,
TOMA)
Customer - SPO
(CTOCU,
TOMA)
End User
Community -
ACC,
Warfighters,
Maintenance,
Sensor
End User
Community -
ACC,
Warfighters,
Maintenance,
Sensor
End User
Community -
ACC,
Warfighters,
Maintenance,
Sensor
End User
Community -
ACC,
Warfighters,
Maintenance,
Sensor
End User
Community -
ACC,
Warfighters,
Maintenance,
Sensor
End User
Community -
ACC,
Warfighters,
Maintenance,
Sensor
Mission
Engineer
Mission
Engineer
Mission
Engineer
SEIT Team
Mission
Engineer
Mission
Engineer
Program
Manager
SEIT Team
Proposal
Manager
Program
Manager
Configuration
Management
Team
Program
Manager
Program
Manager
SEIT Team SEIT Team SEIT Team SEIT Team
End User
Community -
ACC,
Warfighters,
Maintenance,
Sensor
SEIT Team SEIT Team SEIT Team
Program
Manager
Program
Manager
Supplier
Account
Manager
Engineering
Team
Mission
Engineer
Case Manager Case Manager Case Manager
End User
Community -
ACC,
Warfighters,
Maintenance,
Sensor
SEIT Team
End User
Community -
ACC,
Warfighters,
Maintenance,
Sensor
End User
Community -
ACC,
Warfighters,
Maintenance,
Sensor
Case Manager Case Manager Case Manager Case Manager Case Manager Case Manager
Program
Manager
Program
Manager
Program
Manager
Product
Support Team
Program
Manager
SEIT Team SEIT Team
Quality
Assurance
Team
Case
Management
Workflow tool
exports a
customized text
document that
SEIT Team
Case
management
workflow tool
SEIT Team SEIT Team
Case
management
workflow tool
Product
Support Team
Product
Support Team
Configuration
Management
Team
Case Manager
Production
Team
Product
Support Team
Product
Support Team
SEIT Team SEIT Team
Program
Manager
Quality
Assurance
Team
Program
Manager
Mission
Engineer
Configuration
Management
Team
Program
Manager
Case Manager
Product
Support Team
Case Manager Case Manager
Mission
Engineer
Mission
Engineer
Mission
Engineer
Mission
Engineer
Mission
Engineer
Mission
Engineer
SEIT Team SEIT Team SEIT Team
Engineering
Team
SEIT Team
Product
Support Team
Product
Support Team
Business
operations tool
to help organize
across silos the
necessary
resources
Shared,
documented
enforceable
commitment
(e.g. charter)
Product
Support Team
Design +
Development
tool brings all
stakeholders
into a
collaborative
Product
Support Team
Product
Support Team
Version control
software that
keeps the FINAL
version of each
program -
separated by
Engineering
Team
Engineering
Team
Writing
evaluation tool
helps the
writers maintain
the standards
and specs.
Mission
Engineer
Configuration
Management
Team
Engineering
Team
Engineering
Team
Product
Support Team
Product
Support Team
Configuration
Management
Team
Case
management
workflow tool
SEIT Team SEIT Team SEIT Team
Quality
Assurance
Team
Mission
Engineer
Engineering
Team
Mission
Engineer
Mission
Engineer
Proposal
Manager
Proposal
Manager
Proposal
Manager
Proposal
Manager
Proposal
Manager
Proposal
Manager
Product
Support Team
Supplier
Account Team
Product
Support Team
Supplier
Account Team
Product
Support Team
Engineering
Team
Engineering
Team
Shared,
documented
enforceable
commitment
(e.g. charter)
Roles and
Responsibilities
(e.g. RACI, RAA)
Engineering
Team
Version control
software that
keeps the FINAL
version of each
program -
separated by
Engineering
Team
Engineering
Team
Shared,
documented
enforceable
commitment
(e.g. charter)
Case
management
workflow tool
Supplier
Account Team
Case
management
workflow tool
Program
Manager
R/T tool
provides a
progress +
diagnostic
report that
allows for deep
Supplier side: A
portal to upload
necessary
documents for
evaluation
Configuration
Management
Team
Engineering
Team
Engineering
Team
Quality
Assurance
Team
Test tracable to
requirements e
Database
Product
Support Team
Product
Support Team
Case
management
workflow tool
Case
management
workflow Tool
Capture
Manager
Supplier
Account Team
Proposal
Manager
Proposal
Manager
Capture
Manager
Capture
Manager
Capture
Manager
Capture
Manager
Program
Manager
Program
Manager
Engineering
Team
Product
Support Team
Engineering
Team
Quality
Assurance
Team
Engineering
Team
Supplier
Account Team
Supplier
Account Team
Roles and
Responsibilities
(e.g. RACI, RAA)
Case Quality
Checklist
Supplier
Account Team
Shared,
documented
enforceable
commitment
(e.g. charter)
Supplier
Account Team
Supplier
Account Team
SOW
Shared
collaborative
repository
Production
Team
R/T test and
quality check
SEIT Team
A document to
list the
proposed
engineering
changes/modifi
cations that
GH side:
Supplier side
documents are
market
accordingly
(project,
Quality
Assurance
Team
Case
management
workflow tool
Configuration
Management
Team
Case
management
workflow tool
Shared,
documented
enforceable
commitment
(e.g. charter)
Engineering
Team
Engineering
Team
Final V&V
results and
comments site
viewable and
sharable to
NGC
Configuration
Management
Tool
Proposal
Manager
Quality
Assurance
Team
Program
Manager
Program
Manager
Program
Manager
Program
Manager
Program
Manager
Program
Manager
SEIT Team SEIT Team
Quality
Assurance
Team
Engineering
Team
Supplier
Account Team
Configuration
Management
Team
Supplier
Account Team
Quality
Assurance
Team
Quality
Assurance
Team
Careful
management of
dependencies
to ensure
accuracy:
version control,
Careful
management
of
dependencies
to ensure
accuracy:
Quality
Assurance
Team
SOW
Quality
Assurance
Team
Quality
Assurance
Team
Roles and
Responsibilities
(e.g. RACI, RAA)
Engineering
efforts must be
completed
before the
freeze, while
achieving
Case
management
workflow tool
Shared,
documented
enforceable
commitment
(e.g. charter)
Product
Support Team
Document
within the R/T
tool
configuration
changes that
are rejected
Review of
engineering
materials with
the SOW
represented as a
list that the GH
R/T tool pushes
the project into
verification
mode where
the Engineering
leads review the
Tool to publish
the deliverable
Case
management
workflow tool
Tool that allows
comment,
Suggested Edits
to document
issues
Roles and
Responsibilities
(e.g. RACI, RAA)
Production
Team
Flight Test
Team
Shared
customer-client
repository
Method/tool to
communicate
to all
stakeholders
when package
is delivered
Program
Manager
Configuration
Management
Team
SEIT Team SEIT Team
SEIT Team SEIT Team SEIT Team SEIT Team
Product
Support Team
Product
Support Team
Configuration
Management
Team
Case
management
workflow tool
Quality
Assurance
Team
Production
Team
Quality
Assurance
Team
Configuration
Management
Team
Configuration
Management
Team
Network list of
Dependencies
for easy
navigation and
assessment
Where inputs
are coming
from
Configuration
Management
Team
Roles and
Responsibilities
(e.g. RACI, RAA)
Configuration
Management
Team
Configuration
Management
Team
SEMP template
Configuration
freezes are not
deadline based
Document
template to
facilitate the
collaborative
writing of the
maintenance
No changes
after the
Configuration
Freeze
Engineering
Team
Configuration
changes that
are approved
are added to
the task list
within the R/T
Supplier portal
sends updates
Leads are able
to approve or
reject
w/comments
system
requirements. If
Ability to mark
within the work
where there is
confusion or
incorrect
information
Configuration
Management
Tool
Test format to
help observing
researchers
(engineers, tech
writers, etc.)
document the
SOW
Flight Test
Team
Case
management
workflow tool
Shared,
documented
enforceable
commitment
(e.g. charter)
Shared
customer-client
repository
SEIT Team
Production
Team
Product
Support Team
Product
Support Team
Product
Support Team
Product
Support Team
Product
Support Team
Product
Support Team
Engineering
Team
Engineering
Team
Production
Team
Supplier/part
database with
meta data of
past
requirements,
past quality
Configuration
Management
Team
Case
management
workflow tool
Configuration
Management
Team
Production
Team
Production
Team
Dependency
Health Status to
represent
completeness
of data - past,
present, and
Schedule
dependencies
for proposal
submittal
Production
Team
SEMP template
Production
Team
Production
Team
Case Quality
Checklist
Shared,
documented
enforceable
commitment
(e.g. charter)
Easy import of
engineering
source material
into the
document
SOW
Supplier
Account Team
Engineering
proposed
changes need
to be reviewed
to include as in
scope or as a
Schedule
mandatory
update
meetings that
are time
blocked.
Engineering
proposed
changes need
to be reviewed
to include as in
scope or as a
SOW SOW
Need to allow
for access to an
accurate
representation
of the Global
Hawk variant
Integrated
Master
Schedule
Integrated
Master
Schedule
Final V&V
results and
comments site
viewable and
sharable to
NGC
Roles and
Responsibilities
(e.g. RACI, RAA)
Shared,
documented
enforceable
commitment
(e.g. charter)
Product
Support Team
Case
management
workflow tool
Engineering
Team
Engineering
Team
Engineering
Team
Engineering
Team
Engineering
Team
Engineering
Team
Supplier
Account Team
Supplier
Account Team
Case
management
workflow tool
Evaluation tool
to assess the
Supplier's
resource
availability
against the
Production
Team
Traceability
workflow tool
captures the
acceptance
criteria and
responsible
Production
Team
Case
management
workflow tool
Enterprise
Enabler Team -
Scheduler
Dependency
database will be
properly
maintained
with demerits
for letting the
Proposal Write
ups (technical,
management or
otherwise)
Case
management
workflow tool -
to track
applicable
program
Case Quality
Checklist
Case
management
workflow tool -
to track
applicable
program
Enterprise
Enabler Team -
Scheduler
Organization
Structure
Close out the
project Design
+ Development
to prevent
further
changes. All
Supplier portal
to receive/send
documents
Case Quality
Checklist
Quality
Assurance
Team
Steps
Completed
Engineering
proposed
changes need
to be reviewed
to include as in
scope or as a
Develop an
Impact Scale
that is used to
determine if the
requested
engineering
Supplier's task
is not
accomplished
until the SOW is
100% meet.
Approval /
Rejection to
incorporate
change
SOW Steps Pass/Fail
Shared,
documented
enforceable
commitment
(e.g. charter)
Shared
customer-client
repository
SOW
Roles and
Responsibilities
(e.g. RACI, RAA)
Engineering
Team
Method/tool to
communicate
to all
stakeholders
when templates
have been
Supplier
Account Team
Supplier
Account Team
Supplier
Account Team
Supplier
Account Team
Supplier
Account Team
Supplier
Account Team
Quality
Assurance
Team
Quality
Assurance
Team
Evaluation tool
to assess the
Program
Requirements
Shared,
documented
enforceable
commitment
(e.g. charter)
Case
management
workflow tool
Criteria
Definitions
Case
management
workflow tool
Quality Metrics
Dashboard
showing
impacts of
deficiencies
Case
management
workflow tool
(The R/T
checklist
template
Make updating
the dependency
database part of
the wrap up
phase for each
project
Proposal
Presentation
manner (e.g.
demo required,
orals
discussion,
Shared,
documented
enforceable
commitment
(e.g. charter)
Agreed to Case
Quality
Checklist
Shared,
documented
enforceable
commitment
(e.g. charter)
Case
management
workflow tool -
to track
applicable
program
Who will
perform the
work -Roles and
Responsibilities
(e.g. RACI, RAA)
pertaining work
Case Quality
Checklist
Shared,
documented
enforceable
commitment
(e.g. charter)
Prepare the
final review
before
Stakeholder
Gate Review
Daily work is
tracked within
the R/T tool,
providing Sr.
Stakeholders
updates to the
Hardware
Production
Maturity Level
(also known as
volatilty or
changes made)
Engineers and
Tech Writers
assigned to the
project need be
able to co
locate for
Integrated
Master
Schedule
Incentives/Dem
otions to focus
the Supplier on
quality and
meeting
specifics within
Impacted Work
/ Deliverables
Supplier
Product Test
procedures
Steps
Completed
Roles and
Responsibilities
(e.g. RACI, RAA)
Shared,
documented
enforceable
commitment
(e.g. charter)
Configuration
Management
process that
allows for
customer
comments to
Integrated
Master
Schedule
Supplier
Account Team
Configuration
Management
Tool
Quality
Assurance
Team
Quality
Assurance
Team
Quality
Assurance
Team
Quality
Assurance
Team
Quality
Assurance
Team
Quality
Assurance
Team
Configuration
Management
Team
Configuration
Management
Team
Resource
management
tool to compare
scope (time)
and personal
Risk
Management
(e.g.
unavailable ,
discontinued
parts)
Document
template to
compare
customer ACC
criteria and
internal ACC
Template for
tests
Quality Metrics
Dashboard
showing
impacts of
deficiencies
allowing leads
Quality Metrics
Repository/Data
base that
captures the
quality inputs
allowing leads
Integrated
Master
Schedule
Software
Program
Dependencies
are
continuously
updated by
grabbing the
information
Tracking of
proposal
quality checks
being met
SOW
Impacted
Programs
SOW
Integrated
Master
Schedule
Software
Program
How will be
performed
SOW SOW
Written
ingrated test
documents
Alerts are sent
out to PM +
senior
Stakeholders
when the
accumulated
Engineering
baseline is
frozen
Slips are not
allowed to
impact the
schedule of
others.
Steps
Completed
Integrated
Master
Schedule
Integrated
Master
Schedule
Case Quality
Checklist
NGC has
worked with
end user to
ensure our test
procedures
align with their
SOW
Configuration
Management
process that
allows for
customer
comments to
Customer V&V
procedures
Delivery
Considerations
(location,
timeliness,
courier,
destination)
Quality
Assurance
Team
Repository of
Templates
Configuration
Management
Team
Configuration
Management
Team
Configuration
Management
Team
Configuration
Management
Team
Configuration
Management
Team
Configuration
Management
Team
Production
Team
Production
Team
Team
management
database -
experience and
program history
Supplier-NG-
Customer fee
charge model
Shared,
documented
enforceable
commitment
(e.g. charter)
Internal
acceptance
Criteria Baseline
Quality Metrics
Repository/Data
base that
captures the
quality inputs
Quality Metrics
Reporting it
notifies impact
resources to act
upon
Shared,
documented
enforceable
commitment
(e.g. charter)
There is a
proposal
baseline freeze
whereby all
changes need
to be
Case Quality
Checklist
Types of
Branches
Suppliers
receive all
necessary
details to
achieve quality
and complete
Shared
customer-client
repository
Quality Checks
Organization
Structure
Supplier
Schedules
Agreed to
Customer V&V
document
Shared,
documented
enforceable
commitment
(e.g. charter)
high fidelty
prototypes
Steps
Completed
Personnel
supporting the
co-creation
must support
until final
package is
# of times
alarm
raised/ECP
(change
volitatlity)
Schedule
Impact Level -
H, M, L
Steps Pass/Fail Steps Pass/Fail
Roles and
Responsibilities
(e.g. RACI, RAA)
Integrated
Master
Schedule
Version
controlling
Configuration
Management
Team
Shared,
documented
enforceable
commitment
(e.g. charter)
Production
Team
Production
Team
Manufacturing
/ Production
Team
Production
Team
Production
Team
Production
Team
Case
management
workflow tool -
hosts the
project, laid out
like case
Case
management
workflow tool
Tagging system
that collects the
effected
modules
Supplier -
Vendor
certification
Rules for
aligning criteria
(e.g. criteria
change,
acceptance,
rejection)
Tests from
each discipline
Quality Metrics
Reporting it
notifies impact
resources to act
upon
Shared,
documented
enforceable
commitment
(e.g. charter)
Roles and
Responsibilities
(e.g. RACI, RAA)
How inputs fit
together to tell
story
Required
Attendees
Steps
Completed
Case Quality
Checklist
Shared,
documented
enforceable
commitment
(e.g. charter)
Risk
Management
Deviations from
case quality
checklist
Assign tasks
between R/T
Admin and
Suppliers
Number of
redlines before
configuration
freeze
No changes
after the
Configuration
Freeze
This step may
occur in parallel
with 3.10
provided
policies are met
Standards and
specifications
are supportive
of the end user
useage
Personnel are
incentivized to
rasise the alarm
Type of Change
-
administrative,
engineering,
scope
Steps
Completed
Flight Test
Team invites
key personnel
from
Production and
Engineering
SOW
Customer
Approval/
Rejection
Acknowledgem
ent of delivery
receipt from
customer
Production
Team
Roles and
Responsibilities
(e.g. RACI, RAA)
Case
management
workflow tool
Case
management
workflow tool
Enterprise
Enabler Team
(function
supporting,
budgeting,
IT,etc)
Enterprise
Enabler Team
(function
supporting,
budgeting,
IT,etc)
Enterprise
Enabler Team
(function
supporting,
budgeting,
IT,etc)
Enterprise
Enabler Team
(function
supporting,
budgeting,
IT,etc)
Shared,
documented
enforceable
commitment
(e.g. charter)
Document
templates to
track Program
Requirements
Shared,
documented
enforceable
commitment
(e.g. charter)
Supplier parts
certifications
RAA for how
criteria
alignment is
communicated
Number of
Tests
Shared,
documented
enforceable
commitment
(e.g. charter)
Roles and
Responsibilities
(e.g. RACI, RAA)
Case Quality
Checklist
Dependencies
are
continuously
updated by
grabbing the
information
Scope of Work Total Branches
Workload
Forecast
Roles and
Responsibilities
(e.g. RACI, RAA)
Configuration
Management
Steps
Completed
Define the
schedule based
on the Supplier
Schedule and
DONE by dates
gains or losses
# of redlines
before
Configuration
freeze
SOW
Personnel
supporting the
co-creation
must support
until final
package is
Track possible
areas for
standards and
specifications
changes for
change
Alarms raised
must include
the source of
alarm and
reason for
alarm
Quality Impact
Level
Acceptance/Rej
ection of
supplier
product
NGC has
worked with
end user to
ensure our test
procedures
align with their
Integrated
Master
Schedule
Steps
completed in
checklist
New final
deliverable
Case
management
workflow tool
Impacted
Templates
Shared,
documented
enforceable
commitment
(e.g. charter)
Shared,
documented
enforceable
commitment
(e.g. charter)
Flight Test
Team
Flight Test
Team
Flight Test
Team
Flight Test
Team
CM verifies we
can commit to
the deadline
Shared,
documented
enforceable
commitment
(e.g. charter)
System models
Engineering
and Tech Pub
Assessment of
Parts needed
Customer
acceptance
criteria
Repeatable test
or template is
available
All quality
metrics are
understood and
approved by all
actors
All quality
dependencies
are understood
and approved
by all actors
Agreed to Case
Quality
Checklist
Make updating
the dependency
database part of
the wrap up
phase for each
project
Steps
Completed
All work done
will be on the
NG network
Team schedules
Case Quality
Checklist
Scope
Management
Internal
acceptance
criteria has
been aligned
and shared with
customer
Deviations from
case quality
checklist
All the work
done will be
ready for a
freeze
Case Quality
Checklist
Personnel
supporting the
co-creation
must support
until final
package is
Require alarms
be raised by
both engineer
and tech writer
for validation to
decrease false
Priority for
changes is
deteremind by
personnel have
time and
resources to
If product is
rejected,
Supplier has
[TBD] days to
find
replacement
End User
Operational
Test Procedures
Steps
passed/failed
Steps
completed in
checklist
Tool to capture
retrospective
that requires
each item to be
adjudicated
Approval to
change
templates
Roles and
Responsibilities
(e.g. RACI, RAA)
Roles and
Responsibilities
(e.g. RACI, RAA)
Tool allowing
easy reference
and "hooks" to
enforce
commitment
Tool allowing
easy reference
and "hooks" to
enforce
commitment
Tool allowing
easy reference
and "hooks" to
enforce
commitment
Tool allowing
easy reference
and "hooks" to
enforce
commitment
Cases to
meetings held
End User
Community
knowledge
Document
models
Changes
requested by
customer
Internal
acceptance
Criteria Baseline
Deadline are
established for
when
Current and/or
added SOW
Agreed to
Quality Metrics
Workload
Forecast
Customer has
awarded NGC
Budget has
been allocated
for licenses of
configuration
branching tool
as needed
Integrated
Master
Schedule
Suppliers
receive all
necessary
details to
achieve quality
and complete
Steps
Completed
low to high
fidelty
prototypes
Steps
Completed
UAT and
Customer V&V
expedcatios/rub
ric has been
completed
Stakeholder
Review
Entrance and
Exit Criteria
performance
review is
reflective of
how well the
pair increase
the quality of
Case Manager
coordinates
changes w/
minimal
disruption to
team
If product is
rejected,
Supplier has
[TBD] days to
find
replacement or
Approval/
Rejection
Engineer and
Tech Writer are
committed
until delivery
complete. They
may be on
This step is
scheduled in
the IMS
Tool or method
for soliciting
lessons learned
ahead of time,
if needed
Steps
completed in
checklist
Rules for
celebration
allowance
(company
policy)
SOW (definition
of closeout)
Shared,
documented
enforceable
commitment
(e.g. charter)
Shared,
documented
enforceable
commitment
(e.g. charter)
Shared,
documented
enforceable
commitment
(e.g. charter)
Shared,
documented
enforceable
commitment
(e.g. charter)
Can negotiate
with the
customer
Cases to
meetings held
Baseline
Requirements
Affected
requirements
(added,
modified,
removed,
waived)
Internal
Acceptance
Criteria changes
Repeatable test
or template is
available
Set standards of
quality metrics
to be used for
the program
scope and scale
Deliverables
and their
dependencies
on each "actor"
Supplier
Schedules
The checklist
can be
quantifiable/me
asurable
All resources
requiring access
to
configuration
branching tool
completed
Steps
Completed
SOW
Sign off of
SEMP from
internal
stakeholders
required
Suppliers used
have been
certified for
business
partnering
Case Manager
may authorize
and elevate
issue to ensure
this step is
being met
Accomplishme
nts to Date
Steps
completed in
checklist
This step is
scheduled in
the IMS
New final
deliverable
replaces the old
master as the
new master
version.
Shared
customer-client
method to
conduct
retrospective if
not in person
Review of
template
update
correctness
required
Invite list
Acknowledgem
ent of case
closeout from
customer
Required
Stakeholders
Required
Stakeholders
Required
Stakeholders
Required
Stakeholders
Collocate the
work in a
shared space
Number of
agreed and
rejected
changes
Affected
requirements
(added,
modified,
removed,
waived)
Proposed vs
actual cost of
change
Can negotiate
with the
customer
Collocate the
work in a
shared space
Customer
acceptance
criteria
Accuracy /
volatility of the
dependencies
identified
Team schedules
Scope of Work
may have been
modified from
what was
proposed in
proposal
github or gitlab
Allocated
Resources
Agreed to Case
Quality
Checklist
Changes to
SEMP can be
made
Supplier must
acknowledge
meeting/not
meeting
requirements
and justification
Engineering
freezes are
communicated
to end user and
customer to
ensure
Deviations from
SOW and
corrective plan
Steps
passed/failed
NGC
participation at
V&V
Enforce
notifications to
indicate
internally and
customer if
deadlines are
Shared,
documented
enforceable
commitment
(e.g. charter)
New final
template
officially
replaces the old
master as the
new master
Budget
Steps
completed in
checklist
Objections to
Commitment
Objections to
Commitment
Objections to
Commitment
Objections to
Commitment
Can negotiate
with the
customer
Proposed vs
actual cost of
change
Proposed time
to implement
vs actual time
to implement
Understand the
implications of
customer
criteria
Customer
quality
expectations
the
dependencies
can change
Integrated
Master
Schedule
Case Manager
may authorize
and elevate
issue to ensure
this step is
being met
Quality Checks
Achieved/Not
Achieved
against
Schedule
Workload
Forecast
SEMP is
deliverable to
the customer
Path forward on
changes
Engineer and
Tech Writer are
committed
until delivery
complete. They
may be on
Enforce
notifications to
indicate
internally and
customer if
deadlines are
Roles and
Responsibilities
(e.g. RACI, RAA)
Notifications to
when templates
have been
officially
updated
Agenda
Case may be re-
opened but
requires going
through a new
case
management
Actors have
authority and
influence to
enforce policy
Actors have
authority and
influence to
enforce policy
Actors have
authority and
influence to
enforce policy
Actors have
authority and
influence to
enforce policy
Collocate the
work in a
shared space
Proposed time
to implement
vs actual time
to implement
Have means to
assess
engineering
changes
Case Manager
understands
and is able to
translate
customer
criteria to our
Commitment
monitoring
metrics
common set of
quality metrics
Customer
acceptance
criteria as a
Case Quality
Checklist that
helps the
Collocate the
work in a
shared space
The checklist
can be
quantifiable/me
asurable
Supplier
Schedules
Case Manager
may authorize
and elevate
issue to ensure
this step is
being met
Accomplishme
nts to Date
This step is
scheduled in
the IMS
lessons learned
from each
internal
organization
Steps
completed in
checklist
Case is baseline
Posted
reminders
Posted
reminders
Posted
reminders
Posted
reminders
Have means to
assess
engineering
changes
Have means to
assess
document
changes
Customer
typically isn't
allowed to be in
our proposal
development
phase
ECP related
metrics
common
understanding
of why
dependencies
occur
Checklist has
tangible metrics
in each task to
help achieve
quality
Case Manager
may authorize
and elevate
issue to ensure
this step is
being met
Team schedules
SEMP is
traceable to the
case
management
workflow tool
or has
Decrease in
redlines or
scope creep
with a increase
in Customer
satisfcation at
NGC
participation at
V&V
Steps
completed in
checklist
People should
attend in
person but
virtual party
may also be
hosted (via
Case cannot be
changed (not
editable)
beyond this
point, but may
allow for
Incentives -
Social when
policy is
enacted
Incentives -
Social when
policy is
enacted
Incentives -
Social when
policy is
enacted
Incentives -
Social when
policy is
enacted
Have means to
assess
document
changes
Assessments
are captured in
the case
management
workflow tool
Having ready
context of work
increases
effectiveness
common set of
quality metrics
Collocate the
work in a
shared space
Case Quality
Checklist is
considered the
main Task list
that represents
the Client's
Utilize the test
cases to draft
the Case
Quality
Checklist's
metrics
Integrated
Master
Schedule
Case Manager
may authorize
and elevate
issue to ensure
this step is
being met
NGC has
worked with
end user to
ensure our test
procedures
align with their
Number of
items
adjudicated
Celebratory
recognition
awards given
Video recording
of commitment
Video recording
of commitment
Video recording
of commitment
Video recording
of commitment
Assessments
are captured in
the case
management
workflow tool
Collocate the
work in a
shared space
Collocate the
work in a
shared space
quality metrics
are traceable
The checklist
can be
quantifiable/me
asurable
Customer
acceptance
criteria as a
Case Quality
Checklist that
helps the
Co-locate in
one space for in
person
discussions
This step is
scheduled in
the IMS
Gift cards or
some type of
recognition for
individuals who
went above and
beyond
Collocate the
work in a
shared space
quality metrics
have authority
to be enforced
Utilize the test
cases to draft
the Case
Quality
Checklist's
metrics
Steps
Completed
Participation
mandatory
from all internal
stakeholders
Long-term,
Model-based
Engineering
quality metrics
are specific and
tangible
Actual to
Realized
dependencies
In person with
customer and
end user
retrospective in
conjunction
with case
Incentives for
meeting quality
metrics (ice
cream party)
Case Manager
may authorize
and elevate
issue to ensure
this step is
Incentivize with
closeout
celebration s
Scenario: The customer and the NGAS team agree upon the delivery of engineered systems and their necessary documentation.
Reestablish expectations in the organization Document, design, build, and test as a team using Case Management Deliver the completed package as a team
Stage
Step
Actor
Technology
Policy
Info
Metric
Assumption
Idea

18. To-be Process Overview
1. Reestablish expectations of quality in the organization
2. Prepare proposal for engineering, publications, and production
3. Document, design, build, and test as a team
4. Deliver the completed package as a team
5. [Optional] Respond to a customer-requested update
September 201818
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

19. Currently, they’re doing...
We need to fix...
Goal††
Injection for transition
[Enabling] Solution
Future process step
††http://flyinglogic.com/docs/Thinking%20with%20Flying%20Logic.pdf
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

20. ††http://flyinglogic.com/docs/Thinking%20with%20Flying%20Logic.pdf
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

21. Metrics: Hard to measure, but worthwhile
We recommend baselining and measuring the
following, in order of when they are needed:
1. The number of customer redlines.
2. The rework necessary from discoveries during
integrated flight test
and
The TRB-approved engineering changes after
integrated flight test.
3. The changes to the requirements between
program kickoff and integrated flight test.
4. The age of data modules and engineering
artifacts, since their last changes.
5. The overlap between the scope of current ECPs
and the scope of previous ECPs.
6. The delta between our internal acceptance criteria
and the customer’s acceptance criteria.
7. The usability of the tech pubs checklists with the
System Usability Scale (SUS) survey.
8. The usability of the tech pubs in their operational
environment with contextual inquiry.
►The open items on the checklist.*
►The design changes made after the original
configuration freeze.*
►What was left unfinished after delivery since the
configuration freeze.*
* Implemented at a later date
September 201821
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

22. Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

23. New RAA for New Roles
• Responsible for case kickoff, monitoring checklist completion, case
closeout, and retrospective
• Accountable for the delivery of quality for the customer’s satisfaction
• Authority to hold delivery until quality is achieved
Case Manager
• Responsible for knowing where the suppliers touch the program
portfolio
• Accountable for information, requirements, and notifications to and from
suppliers
Supplier Account Manager
• Responsible for contributing to the scope assessment of ECPsMission Engineer
September 201823
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

24. Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

25. Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

26. system and subsystem CAD
At this point we can move forward with confidence
in the picture of the final product not changing.
v
regulations, word processing,
submission
Write the
test that
must
pass
production
Write the
check-
list
Each
function
writes
req’s for
the
others
creation of
procedures
systems engineering, integration, and test
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

27. 1. Reestablish expectations in the organization.
To-be Process: Stage 1
1.1 Commit to the deliverable being defined as the a complete package that's most usable to end user.
1.2 Commit to delivering package when quality criteria is achieved.
1.3 Commit to performers being committed to a case until delivery.
1.4 Commit to case checklists being completed before delivery.
September 201827
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

28. 2. Prepare proposal for engineering, publications, and production.
To-be Process: Stage 2
2.1 Kick off the case; chaired by the case manager and including the program manager, mission engineer, and account
manager.
2.2 Conduct an external stakeholder program review, and include operators and maintainers.
2.3 Assess the scope in terms of engineering and publications before the proposal is submitted.
2.4 Assess supplier scope via account management.
2.5 Define the difference between customer acceptance criteria and our internal acceptance criteria.
2.6 Write the test that the team must pass before the proposal is submitted .
2.7 Decide quality metrics that hold responsible entities (e.g. organizations, persons, etc.) accountable.
2.8 Declare quality dependencies for each other (e.g. engineering, tech pubs, production, and suppliers) before the
proposal is submitted.
2.9 Schedule dependencies defined in IMS and coordinated across all IMS's.
2.10 Coordinate and assemble proposal inputs.
September 201828
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

29. 3. Document, design, build, and test as a team.
To-be Process: Stage 3
3.1 Organize work by case management.
3.2 Complete configuration branch ASAP after kick-off.
3.3 Generate case checklist.
3.4 Distribute and communicate expectations .
3.5 Perform system engineering design and development.
3.6 Complete configuration freeze before integrated test UAT and customer V&V.
3.7 Complete hardware production.
3.8 Pair co-creation of maintenance procedures.
3.9 Write tech docs in compliance with standards and specs.
3.10 Raise alarm when report of schedule slips in engineering, publications and production.
3.11 Conduct impact assessment for engineering changes after configuration freeze.
3.12 Sign off that supplier products satisfy the SOW.
3.13 Conduct engineering verification of satisfaction of system requirements.
3.14 Conduct integrated flight test of whole deliverable.
3.15 Conduct end-user V&V of whole deliverable in their operational environment.
3.16 Conduct final customer V&V of the whole deliverable.
September 201829
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

30. 4. Deliver the completed package as a team.
To-be Process: Stage 4
4.1 Hand off completed package.
4.2 Conduct case retrospective to encourage learning and continuous improvement.
4.3 Update templates and guides based on results retrospective.
4.4 Celebrate case completion.
4.5 Close out the case.
September 201830
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18

31. 5. [Optional] Respond to a customer-requested update.
To-be Process: Stage 5
5.1 Reopen cases using same configuration management as new cases.
5.2 Update templates and guides on a regular basis.
5.3 Repeat stages 1–2.
September 201831
Approved For Public Release #18-2000; Unlimited Distribution Dated 9/19/18