business model, business model canvas, mission model, mission model canvas, customer development, hacking for defense, H4D, lean launchpad, lean startup, stanford, startup, steve blank, pete newell, bmnt, entrepreneurship, I-Corps, Security, NSIN, NSA, cybersecurity, Joe Felter, ATO, DOD
1. Flexible Fingerprints
Lee Alpert | Vanessa Bernick | Ayelet Drazen | Kate Yeager | Corinne Zanolli
Original Problem Statement:
Identify the optimized
combination of open source
cybersecurity tools in order to
ensure organizations with limited
resources are performing security
assurance during development.
Revised Problem Statement:
Improve DoD cybersecurity
through the increased
efficiency and transparency of
the ATO accreditation
process.
107 Interviews
Problem Sponsor: George Huber, NSA’s Center for Assured Software
Defense Mentors: Donnie Hasseltine & Mike Hoeschele
Business Mentor: Katherine Tobin
Teaching Assistant: Sally Egan
B.A. in History | Honors
in Int. Security (‘21)
B.A. in Political Science |
Honors in Int. Security (‘21)
B.A. in Political Science |
M.S. in CS (‘22)
MBA (‘22) | Former Navy
Cryptology Officer
B.S. in CS | M.S. in
MS&E (‘22)
2. We conducted 107 interviews across government,
academia/non-profit, and the private sector.
68 Interviews 13 Interviews 26 Interviews
3. We started off with little understanding of our
problem...
Week 1:
Focused on OS
cybersecurity
tools. Thought
our beneficiary
was NSA’s CAS.
Getting Started
Based on an initial meeting with our sponsor and equipped
with a list of interviewees, we set out to explore the problem
of encouraging DoD agencies to employ open source tools.
5. Open Source Tool Availability Isn’t the Pain Point
“Availability of OS tools isn’t the issue. The main concern is
that we don’t know whose tools they are, not whether they
are available.”
- Anonymous at Department of Energy
“Most programs I provide for are concerned with the
supply chains of OS tools.”
- Team Lead at Naval Surface Warfare Center
“The main problems with OS tools are maintenance and
management, ulterior motives, who is building/working on it.”
- Security Strategist at Cynet Security
6. Narrowing Down the Problem
Week 1:
Focused on OS
cybersecurity
tools. Thought
our beneficiary
was NSA’s CAS.
Week 2:
Thought primary
beneficiary was
under-resourced
components in
the DoD. Shifted
away from OS.
Week 3:
Honed on
primary
beneficiary as
program
managers in
DoD.
Finding Our Beneficiaries & Our Scope
Surprised to find that open source tools played less of a
role in the problem than we thought. This helped us find
our primary beneficiaries.
7. “Where are your primary beneficiaries?
Have you actually talked to any of them?”
—The Teaching Team
10. Patricia the
Program Manager
Don the
Developer
Software developers create the software but program
managers have ownership over the software and the system.
11. Figuring Out the Real Problem
Week 1:
Focused on OS
cybersecurity
tools. Thought
our beneficiary
was NSA’s CAS.
Week 2:
Thought primary
beneficiary was
under-resourced
components in
the DoD. Shifted
away from OS.
Week 3:
Homed in on
primary
beneficiary as
program
managers in
DoD.
Week 4:
Pivoted from
focusing on an
OS toolset to
improving JFAC
accessibility,
awareness, and
retention.
AHA!
Adoption of open source tools isn’t the real
problem… Most DoD services aren’t doing enough
cybersecurity, independent of what type of tool
they use.
12. What is the Joint Federated Assurance
Center (JFAC) ?
13. United States Army
Network Enterprise
Technology Command
(NETCOM)
JFAC
DMA NRO MDA
DISA NNSA OUSD NSA
Army Communication
Electronics Command
(SEC)
Air Force- 76th Software
Maintenance Group
● Software
● Technical Support
● Additional Online Training
Resources
JFAC- Cybersecurity Assistance for DoD Customers
Customer Orgs
Steering Committee
14. MVP #1: A Marketing and Accessibility Problem
So we modeled an improved Joint Federated Assurance Center
(JFAC) portal and thought of ways to market it better.
15. No One Knows or Cares About JFAC Resources
People Are Unaware of JFAC
“A lot of people in need of the resources the JFAC
portal provides don’t know it exists.”
- Information System Security Manager (ISSM) at
Army Future Command
Resources weren’t interesting to potential beneficiaries
“To be completely honest, I’m not interested in that at all
- Marine Corp Officer Managing a Software Product
JFAC is Not Widely Utilized
“People do not go to JFAC, everyone fends for themselves.”
- Product and Innovation Officer at Army Software Factory
JFAC is a Silo
“Effectiveness is JFAC’s biggest challenge. It
operates within a little silo.”
- Security Strategist at Cynet Security
16. Our MVP focused on improving broken solutions
which weren’t working, we weren’t adding new value.
17. Finding a Real Problem to Solve
Week 1:
Focused on OS
cybersecurity
tools. Thought
our beneficiary
was NSA’s CAS.
Week 2:
Thought primary
beneficiary was
under-resourced
components in
the DoD. Shifted
away from OS.
Week 3:
Homed in on
primary
beneficiary as
program
managers in
DoD.
Week 4:
Pivoted from
focusing on an
OS toolset to
improving JFAC
accessibility,
awareness, and
retention.
Week 5:
Considering
pivot to focus on
contractual
language that
bolsters
software
assurance.
Back to the Drawing Board
Improving the JFAC portal and resources won’t
actually encourage program offices to do more
software assurance and security testing. What
will? Where’s the real pain point?
18. AHA!
Program managers need help getting their software
launched onto networks post-development.
The pain-point here is getting the Authority to
Operate (ATO)...
19. Getting Authority to Operate (ATO) approval is a MAJOR
Pain Point
“The process is incredibly painful outside of a software
factory...sh*t ton of paperwork.”
- Cloud Lead at United States Marine Corps
“‘Poorly defined would be an understatement.”
- VP of Growth at Second Front Systems
“I’m part of a Slack channel called ATO Oh No.”
- Deployment Strategist at Palantir
20. ● Fill out list of question
● Run scans/tests-
Requirements based on
software type/network
Process for Getting Software Approved to Run on DoD Servers: ATO
Software/PM
Some other entity
approved?
No one approved
Interim Authority To Test/
Interim Authority To Build
● Pull together artifacts
● Scans, ports
used/protocols
Send package to
Authorizing Official
Gather artifacts from
this ATO Process
1) Categorize Information
System (IS)
2) Select Security Controls
3) Implement Security
Controls
4) Assess Security Controls
5) Authorize Information
System (IS)
6) Monitor Security Controls
“The security authorization package consists of the
security plan, Security Assessor Representative
(SAR), Plan Of Action & Milestones, (POA&M) and
authorization decision document, and is the
minimum information necessary for the acceptance
of an Information System or PIT system by a
receiving organization.”
Official RMF Process
1) Categorize Information
System (IS)
2) Select Security Controls
3) Implement Security
Controls
4) Assess Security Controls
5) Authorize Information
System (IS)
6) Monitor Security Controls
Official Risk
Management
Framework Process
21. The Final Problem to Solve
Week 1:
Focused on OS
cybersecurity
tools. Thought
our beneficiary
was NSA’s CAS.
Week 2:
Thought primary
beneficiary was
under-resourced
components in
the DoD. Shifted
away from OS.
Week 3:
Homed in on
primary
beneficiary as
program
managers in
DoD.
Week 4:
Pivoted from
focusing on an
OS toolset to
improving JFAC
accessibility,
awareness, and
retention.
Week 5:
Considering
focus on
contractual
language that
bolsters
software
assurance.
Week 6:
Developed
contractual
language MVP
and positioning
this language on
JFAC website.
Week 7:
Realized that
harnessing JFAC
wouldn’t
necessarily work
to improve ATO
process.
Week 8:
Pivoted from
contractual
language MVP
to developing
tool to facilitate
ATO.
Another Pivot!
MVP #2 is a tool to
facilitate the ATO.
22. “If you want to start a security company,
you start a compliance company.”
— CEO at Series A Security Company
23. Integrating Cybersecurity Compliance into the Software Development
Lifecycle
Design Development
Testing/AT
O process
Deployment Maintenance
Paperwork
Produced
24. MVP - 1. Initial Survey Automatically Generates Appropriate ATO Requirements
Design
Development
Deployment
Maintenance
Testing/AT
O process
25. MVP - 2. Development Team Access a Dashboard with Required Tasks for ATO
Design
Development
Deployment
Maintenance
Testing/AT
O process
26. MVP - 2. Each Required Task Includes Additional Documentation and Detail
Design
Development
Deployment
Maintenance
Testing/AT
O process
27. MVP - 3. After Development is Completed, A Detailed Report is Presented with Included Documentation
Design
Development
Deployment
Maintenance
Testing/AT
O process
28. Implementing an Integrated Checklist is Helpful
“MITRE could be a good partner if you turn this into a
produce/company after the class..”
- Senior Systems Engineer at MITRE
Our Product Addresses a Major Pain Point
“Anything that makes the ATO process more easily
understood is a major win.”
- Former Developer at Northrop Grumman
MVP Would Be a New Value Add
“Should there be an integrated system contractor and government
should both be looking at? I would say unequivocally and
enthusiastically yes because it doesn’t exist.”
- Current Deployment Strategist at Palantir
Integrated Compliance MVP Feedback: “We want this now.”
29. Implementing an Integrated Checklist is Helpful
“MITRE could be a good partner if you turn this into a
produce/company after the class..”
- Senior Systems Engineer at MITRE
Our Product Addresses a Major Pain Point
“Anything that makes the ATO process more easily
understood is a major win.”
- Former Developer at Northrop Grumman
MVP Would Be a New Value Add
“Should there be an integrated system contractor and government
should both be looking at? I would say unequivocally and
enthusiastically yes because it doesn’t exist.”
- Current Deployment Strategist at Palantir
Integrated Compliance MVP Feedback: “We want this now.”
30. Quarter At A Glance
Week 1:
Focused on OS
cybersecurity
tools. Thought
our beneficiary
was NSA’s CAS.
Week 2:
Thought primary
beneficiary was
under-resourced
components in
the DoD. Shifted
away from OS.
Week 3:
Homed in on
primary
beneficiary as
program
managers in
DoD.
Week 4:
Pivoted from
focusing on an
OS toolset to
improving JFAC
accessibility,
awareness, and
retention.
Week 5:
Considering
pivot to focus on
contractual
language that
bolsters
software
assurance.
Week 6:
Developed
contractual
language MVP
and positioning
this language on
JFAC website.
Week 7:
Realized that
harnessing JFAC
wouldn’t
necessarily work
to improve ATO
process.
Week 8:
Pivoted from
contractual
language MVP
to developing
tool to facilitate
ATO.
Week 9:
Offered a
partnership
with MITRE to
develop MVP.
31. Gaining Traction: “Do you all want jobs?”
MITRE has expressed interest in partnering
with us. They have proposed two options:
1. Incorporate and then go through
SBIR/STTR with MITRE as a key partner
2. Offered all of us jobs to work at MITRE
and develop our product with access to
their resources
32. Lee Alpert | Vanessa Bernick | Ayelet Drazen | Kate Yeager | Corinne Zanolli
Contact Information for Our Team
lalpert@stanford.edu | vbernick@stanford.edu | adrazen@stanford.edu | kateyeager@mac.com | corinne.zanolli@gmail.com
We are seriously considering moving forward with this project, and would be extremely
interested in feedback, both positive AND negative. Please contact flexible-
fingerprints@lists.stanford.edu or any of the emails below with comments.
36. KEY PARTNERS
● JFAC leadership, JFAC
CC
● Service Provider
agencies
KEY RESOURCES
● DoD methods for
communicating with
program managers
● Current JFAC service
providers and JFAC CC
VALUE PROPOSITIONS
● Make software assurance
more accessible: Give actors
across the DoD clearer access
to existing tools and support
allowing them to do
independent SwA checks.
● Easier connectionbetween
the beneficiariesand service
providers
● More efficient diagnosis and
treatments of SwA problems
● Abilityto differentiate
between resources most
suitable for the beneficiary
and those less useful
● Less time spent looking for
suitable resources
● Reduce burdenof redirecting
PMs to SwA resources
● Clear tracking of success
metrics to understand how
the platform is being used
KEY ACTIVITIES
Increase awareness of JFAC in
the program manager
community
Improve JFAC portal
Policy Modification
● Interagency agreement to
clarify technical and
funding information flow.
MISSION ACHIEVEMENT/IMPACT FACTORS
● Number of agencies which feel they have sufficient software assurance coverage as measured by internal and external
accounts.
● Decrease the amount of hours each agency spends identifying and testing the software assurance tools they use.
Mission: To increase usage of software assurance resources availableacross
the DoD to enable software assurance throughoutthe development lifecycle.
Designedby: Flexible
Fingerprints
Date: 4/20/21 Version: 4.0
DEPLOYMENT
● Parallel pilot program
within the DoD to test
scalability within a larger
ecosystem of potential
partners
BUY-IN & SUPPORT
● Sponsor support:
NSA CAS, Acting Director of
JFAC
● Implementation support:
analysts, cyber commands,
approval from IT team &
security approval, current
JFAC portal architects
● Development support:
academia, subject matter
experts, contractors
MISSION COSTS
● Software design & engineering
● Travel
BENEFICIARIES
Primary:
● Program managers in
technically under-resourced
DoD agencies
Secondary:
● NSA Center for Assured
Software
● JFAC: (Communicates with
Primary Services)
● Department of Energy,
Department of Agriculture,
Department of Homeland
Security
THE MISSION MODEL CANVAS
37. Value Proposition Canvas
PRODUCTS
& SERVICES
Development plug-in with
embedded ATO requirements (e.g.
NIST controls); integrated into the
development environment that to
facilitate the ATO process and
increase the amount of SwA
completed
● Provide development team with
easy interface for tracking ATO
software requirements during
development
● Get PMs involved in contract
process with eye towards ATO CLIENT GOAL
Ensure software
component meets
requirement for
ATO
PAINS
GAINS
GAIN
CREATORS
PAIN
RELIEVERS
Security and software
checks conducted by those
developing the software to
speed up deployment time.
● Make the ATO process
significantly easier by ensuring
required checks are
done/documentation exists
● Increase software assurance and
security checks being done during
development process and
decrease costly fixes at the end of
the process
ATO process is long,
time consuming, and
requires frustrating
paperwork
Wasted time and money
on software checks that
should have been done by
software developer
Provide development team
with easy interface for tracking
ATO software requirements
during development
Editor's Notes
Feedback:
Jeff Decker: not sure if you mentioned, but it would be helpful to have some voiceover or slide telling why this problem is such a big deal...not getting software to dod is a major problem
Feedback:
MITRE is an FFRDC (federally funded research and development corporation)
From teaching team: slide looks good but cut the amount of time spent on presenting in half
Jeff Decker:
I like this title. It says exactly what you want people to take away. Maybe shorten: 105 interviews across...
You should consider using directional title slides throughout. tip: create directional title slides then organize each title sequentially to determine that your narrative makes logical sense. EG - Titles from slides 2, 3, 4 should all logically follow. This will help you patch the narrative holes SB pointed out
Define ATO somewhere
-highlight that ATO is a final hurdle, the ATO is a the final goal line
- it feels likes some one is constantly moving the goal line
-the ATO is the last time the security team can influence development
By Sebastian Belalcazar Lareo, CO
In the emojis faces Collection
By Mooms, AU
In the Emoji Collection
By Mooms, AU
In the Emoji Collection
Take out triangle
List more icons
Figure out if value add to traingle
Feedbac:
Maybe get quotes to back this up
- need to be able to justify pivot
- also note that thsi when we broke from teh original ask when we realized the original ask wasn’t a real problem
Define ATO somewhere
-highlight that ATO is a final hurdle, the ATO is a the final goal line
- it feels likes some one is constantly moving the goal line
-the ATO is the last time the security team can influence development
Add legend at the bottom to explain the acronyms
Add arrows to make sure people understand journey
can back this up verbally with “Government does a lot of compliance, not assurance” Enrique Oti at Second Front
Waterfall → business plan, go perfect it, then in front of customers
Software happens too fast
Agile → release as fast as possible and do fast, small iterations
Kessel Run is doing this and doing it well, but agencies using JFAC are not using it
Software assurance throughout the development vs. all at the end
Carol Lee → only when they go to fill out
You’re not going to check your oil unless you’re legally required to do so
DoD is slow to adopt the agile framework → this is what we think the worst case scenario is
Service providers at JFAC say that this is how it works, that’s our target → some are doing it better
No standardized
8510 = risk management framework https://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodi/851001p.pdf?ver=qEE2HGN_HE4Blu7161t1TQ%3D%3D
MITRE is an non profit that operates FFRDC federally funded research and development centers