Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Sea++ H4D Stanford 2018

62,821 views

Published on

business model, customer development, hacking for defense, H4D, lean launchpad, lean startup, stanford, startup, steve blank, pete newell, bmnt

Published in: Education
  • Be the first to comment

  • Be the first to like this

Sea++ H4D Stanford 2018

  1. Ruben Krueger BS CS Software Scott Pratz MS EE Strategy Meilinda Sun BS CS Software Ross Ewald BS CS Policy Sea++ Original Problem: Test engineers cannot safety or adequately cyber test ship nav systems. 99 interviews Current Problem: COs must understand impacts of cyber vulnerabilities to missions. Sponsor: MITRE (in collaboration with the Office of Naval Research)
  2. Weeks 1-3: Understanding problem scope Customer Discovery Objective: Interview cyber engineers to learn why cyber testing is unsafe and inadequate.
  3. - MITRE Systems Engineer, Former USAF “[Operational leadership] is often the barrier to cyber testing, as they worry about the safety of the crew and integrity of the asset.”
  4. Mission Model Canvas™ Week 1 Key Partners NavSea Philadelphia Cyber Command Private Cybersecurity Firms Shipbuilders Commanding officers Key Activities: Understanding hardware testing procedures Developing evaluation criteria Gaining trust and gauging expectations of ship captains Value Proposition Development of physical and electronic security testing procedures. Streamline testing procedures to minimize operational downtime Target individual systems for most useful testing applications Incremental aggression capabilities to maximize vulnerability detection. Beneficiaries Test Engineers Commanding Officers Key Resources Safety parameters of cyber- physical systems Integrated navigation system models/block diagrams User Design Input Buy-In / Support Operational commanders for conducting testing NavSea/MITRE (Navy) and C4IT (CG) to conduct testing and evaluate effectiveness of methodology Deployment Testing of methodology on sample test plans Full electronic software deployment to customers via web-app Mission Budget / Costs Fixed: Software design & engineering Testing and reporting costs Variable: Expansion of hardware testing capabilities Travel Mission Achievement More aggressive testing taking place onboard ships Adoption and continued use metrics of the methodology by engineers Increased vulnerability detection on ships User retention during testing lifecycle
  5. Develop testing procedures Target individual systems Cyber Engineers Commanding Officers Mission Model Canvas™ Week 1 Value Proposition Beneficiaries Streamline testing procedures Incremental aggression capability
  6. Initial MVP - Collaboration Tool Between COs and Engineers Ship’s Personnel CONCERNS Cyber Engineers/Testers TESTING RQMNTS Sea++ Platform Testbed Creation: Environmental/ Cyber Monitoring Requirements Integrated Navigation System Data Identification of System Vulnerabilities s Hypothesis: Better testing planning is needed to improve safety. MVP: Platform to engage COs and engineers in the planning of cyber tests and to create a corresponding testbed.
  7. “There is no routine entity that tests cyber security on ships.” - Member of National Security Council, Navy Officer
  8. What we had been doing: “Can you please direct us to the group that does integrated testing?” NAVSEA SPAWAR NAVAIR OFFICE OF NAVAL RESEARCH FLEET CYBER
  9. Weeks 4-7: Why isn’t anyone testing ships? Customer Discovery Objective: Understand why post-acquisitions integrated cyber testing does not occur.
  10. Nonstop Pivoting, Beneficiary Confusion R&D High-value assets may be damaged Why doesn’t post-acquisitions cyber testing occur? Pentesters Post-test remediation is time-consuming and inadequate. Test Engineers Inadequate documentation for a thorough test Test Engineers Integrated system of the ship is too complex to understand Government Officials No tools to understand the impact of cyber Commanding Officers Organizational issues, cultural issues
  11. Nonstop Pivoting, Beneficiary Confusion R&D High-value assets may be damaged Why doesn’t post-acquisitions cyber testing occur? Pentesters Post-test remediation is time-consuming and inadequate. Test Engineers Inadequate documentation for a thorough test Test Engineers Integrated system of the ship is too complex to understand Government Officials No tools to understand the impact of cyber Commanding Officers Organizational issues, cultural issues
  12. How can we make post-acquisitions cyber testing happen in the Navy? Commanding Officers 3 Establish Office to Manage Integrated Testing Chief of Naval Operations1 2 Naval Systems Commands / Fleet Cyber
  13. How can we make post-acquisitions cyber testing happen in the Navy? Commanding Officers 3 Establish Office to Manage Integrated Testing Chief of Naval Operations1 2 Naval Systems Commands / Fleet Cyber Empower Commanding Officers to Demand Fix
  14. Weeks 8-10: What does a commanding officer need to know about cyber threats? Commanding officers will address cyber threats once they understand them. Customer Discovery Objective: What does a commanding officer need to know about cyber?
  15. “To many commanders, a cyber threat is not as real as a missile attack.” - National Security Council Member, Naval Officer
  16. Make cyber as real as traditional threatConcrete Only relevant info about mission impactEssential Clear path to remediationActionable Communicating Cyber Impacts to Commanders
  17. “I understand everything in terms of mission impact ... I just want sh*t to work.” - Navy Commanding Officer
  18. Make cyber as real as traditional threatConcrete Only relevant info about mission impactEssential Clear path to remediationActionable Communicating Cyber Impacts to Commanders
  19. “It’s depressing when cyber people tell us that everything is vulnerable. My focus is not on assessing vulnerabilities but on fixing them.” - Naval Commanding Officer
  20. Make cyber as real as traditional threatConcrete Only relevant info about mission impactEssential Clear path to remediationActionable Communicating Cyber Impacts to Commanders
  21. Make cyber as real as traditional threatConcrete Only relevant info about mission impactEssential Clear path to remediationActionable Communicating Cyber Impacts to Commanders
  22. Current Mission Model Canvas™ Key Partners Equipment Manufacturers Naval Engineers MITRE Fleet Cyber Chief of Naval Operations NavSea Key Activities: Development of software and visualization integration Contract for acquisitions Modifications for private sector Value Proposition Understand how cyber threats degrade missions Understand how to address cyber threats Beneficiaries Ship Commanders Key Resources Access to private sector simulations Buy-In / Support Chief of Naval Operations Office of Naval Research Ship commanders - Fleet Cyber Deployment Contract for MVP written and submitted to acquisitions Mission Budget / Costs Development of MVP (materials) Travel Installation costs for MVP Mission Achievement Increased clarity of cyber-vulnerabilities leading to better decision- making and increased testing
  23. Understand how to address cyber threats Naval Operators (Commanding Officers) Current Mission Model Canvas™ Value Proposition Beneficiaries Understand how cyber threats degrade missions
  24. Filling the gap: Convey information to commanding officers Convey Information to Fleet Cyber (Navy Cyber Operational View) Present IT Tools (VRAM, Nessus, etc.) Convey information to commanding officers
  25. Filling the gap: Convey information to commanding officers Convey Information to Fleet Cyber (Navy Cyber Operational View) Our MVP Present IT Tools (VRAM, Nessus, etc.) Additional Tools/Capabilities
  26. https://missioncritical.app
  27. https://missioncritical.app
  28. “This is exactly what we needed.” - Former Program Manager, Army Cyber Command
  29. Maritime Commercial Industry $500 BILLION SAM Maritime Government $194 BILLION SOM Naval Ships and Subs $63 BILLION Commercial Factories $155 BILLION TAM Size of Opportunity Department of Defense $716 BILLION s
  30. Moving Forward Interest from: - Former program manager at Army Cyber Command - Ship commanding officers - Lockheed Martin cyber security personnel All four of us plan on continuing with Sea++
  31. Acknowledgments Our work would not have been possible without our sponsors at MITRE (Suresh Damodaran, Matt Mickelson, and Alex Schlichting) and numerous other supporters, including the teaching team, TAs (especially Will Papper), H4D military liaisons, and our mentor Daniel Bardenstein. Additionally, a special thanks to individuals at the following organizations:
  32. Questions?
  33. APPENDICES
  34. High-value assets may be damaged R&D Inadequate documentation for a thorough test Test Engineers No tools to understand the impact of cyber Commanding Officers Post-test remediation is time- consuming and inadequate. Pentesters Integrated system of the ship is too complex to understand Test Engineers Organizational issues, cultural issues Government Officials
  35. Pivot Testing of all systems (i.e., ship-wide testing) Our previous focus
  36. Pivot Many of the same problems (not damaging high value assets, organizational issues) Testing of systems (electrical, navigational, etc.) Testing of all systems (i.e., ship-wide testing)in a “red team” style
  37. Pivot Testing of all systems cannot be solved without understanding how to test individual systems Testing of systems (electrical, navigational, etc.) Testing of all systems (i.e., ship-wide testing)
  38. Problem summary
  39. MVP - “Black Box” ● During testing, replace high-risk component with MVP ● Using an AI simulation of data sent and received by the component, simulate the behavior of the component ● Virtually replicate the behavior of the system without risking damage to it ● Mitigate risk by removing at-risk and expensive components from testing
  40. Customer Discovery Hypothesis: The USS Secure is inadequate for some kinds of cyber testing. Result: Some systems, such as an engineering control system, would probably not be able to be adequately tested in a purely simulated environment (the USS Secure). Our sponsor - "Lower fidelity (constructive or virtual) simulations do not usually come with adequate attack surface."
  41. Customer Discovery Hypothesis: A machine learning model can accurately simulate the responses of a physical component (such as an engine) under a variety of conditions. Result: Confirmed, if we can get the necessary data for the model.
  42. Customer Discovery Hypothesis: The device does not currently exist. Result: Confirmed; the Navy is currently working on building out USS Secure, a virtual testing environment for ships, but a physical proxy to replace components does not exist.
  43. MVP Who What Where When Why
  44. Buy-In/Support Supporters Advocates Saboteurs Private Sector Commercial Shipping Companies Industrial Control System Operators Integrated System Testers Component-Level testing Companies Public Sector Fleet Cyber Command NavSea Crews/Captains Time/Budget Opponents (Engineering Testing, Weapons Testing, etc.)
  45. Previous Mission Model Canvas™ -Format for tracking the testing process -Determining organizational constraints - US Cyber Command -Fleet Cyber -Chief of Naval Operations Saboteur - Naval Support Organizations - compile cyber vulnerabilities into one process for sake of efficiency and security -Ensure that red-team cyber testing occurs consistently and safely on ships -improved security of naval vessels - conflicts of interest between proposed and existing budgets and schedules for cybertesting - Increased vulnerability detection on ships - Consistent, comprehensive cyber testing on naval vessels -creation of cyber- testing agency within Fleet Cyber - Testing and reporting costs - Travel -Ships to test on -Trust of ship commanders -Safety parameters of cyber-physical systems Beneficiaries Mission AchievementMission Budget/Costs Buy-In/Support Deployment Value Proposition Key Activities Key Resources Key Partners -NavSea -SPAWAR - Fleet Cyber Command -Chief of Naval Operations
  46. CAPT Jim Passarelli: Value Proposition Canvas™ Products & Services Black box that simulates a physical component Remove equipment from testing entirely Customer Jobs -Complete missions -Maintain health and safety of crew and ship-Testing which interferes with mission scheduling -Testing that damages any part of the ship 9th Intelligence Squadron Gains Pains Gain Creators Pain Relievers Increased operational readiness Fast installation and removal of tools to ensure that they are implemented. Fast removal decreases system downtime and restoration period
  47. Test Engineer Mr. Gene Lockhart: Value Proposition Canvas™ Products & Services Black box that simulates a physical component - Free up time by easy installation and interface- ability with multiple types of industrial physical systems Customer Jobs Find cyber vulnerabilities in the electrical system - High risk of Damaging system components - Tme spent mitigating safety concerns of physical equipment 9th Intelligence Squadron Gains Pains Gain Creators Pain Relievers -More secure integrated ship systems, better testing available - Removes risk of damaging equipment by entirely removing equipment from the systems testing
  48. Research Engineer Suresh Damodaran: Valu Proposition Canvas™ Products & Services Black box that simulates a physical component Removes physical equipment from the system entirely to decrease machinery damage risk Customer Jobs Find cyber vulnerabilities in the electrical systemDamaging components in sub-systems. Decreasing thoroughness of testing to accommodate risk to damage equipment 9th Intelligence Squadron Gains Pains Gain Creators Pain Relievers - Faster, more accurate testing of components that will be installed on ships Does not compromise accuracy of results due to accurate modelling of physical systems.
  49. Grow EXPAND customer base through the Coast Guard, private sector shipping (tankers, cruise ships), and private sector industry (industrial farming, utility companies) OBTAIN REFERRAL via proof of successful military adoption to commercial industry Keep MAINTAIN interest by refining and integrating the product to meet new needs IMPROVE product by adding more complex features that improve the capabilities of the product Get ACQUIRE customers via successful testing, approach potential users and market value of our solution ACTIVATE customers with pushing naval offices to adopt the cyber testing device, train users in test device usage to ensure adoption
  50. Week 1 Mission Model Canvas - NavSea Philadelphia - Cyber Command - Private cybersecurity firms - Shipbuilders - MITRE - Development of physical and electronic security testing procedures. - Streamline testing procedures to minimize operational downtime - Target individual systems for most useful testing applications - Incremental aggression capabilities to maximize vulnerability detection. - More aggressive testing taking place onboard ships - Adoption and continued use metrics of the methodology by engineers - Increased vulnerability detection on ships - User retention during testing lifecycle - Safety parameters of cyber-physical systems - Integrated navigation system models/block diagrams - User Design Input Beneficiaries Mission AchievementMission Budget/Costs Buy-In/Support Deployment Value Proposition Key Activities Key Resources Key Partners - Operational commanders for conducting testing - NavSea/MITRE (Navy) and C4IT (CG) to conduct testing and evaluate effectiveness Primary: safety of ship crewmen, safety of large assets Primary: Providing defense contractors with methodology for determining security of navigational systems Secondary: Decrease in potential for international incidents Fixed: - Software design & engineering - Testing and reporting costs Variable: - Expansion of hardware testing capabilities - Hardware testing procedures understanding - Developing evaluation criteria -Gaining trust and gauging expectations of ship captains - Testing of methodology on sample test plans - Full electronic software deployment to customers via web-app
  51. Week 2 Mission Model Canvas -Understanding hardware testing procedures - Understanding evaluation criteria -NavSea Philadelphia -Cyber Command -Private cybersecurity firms -Shipbuilders -MITRE -Operational commanders - Test engineers - Defense contractors - Streamline testing procedures to minimize operational downtime - Yield best ID of vulnerabilities for correction - Development of physical and electronic security testing procedures - Target individual systems for most useful testing applications - Incremental aggression capabilities to maximize vulnerability detection. - More aggressive testing taking place onboard ships - Adoption and continued use metrics of the methodology by engineers - Increased vulnerability detection on ships - Using methodology on ships - Full electronic software deployment to customers via web-app - Software design & engineering - Testing and reporting costs -Ships to test on -Trust of ship commanders -Safety parameters of cyber-physical systems Beneficiaries Mission AchievementMission Budget/Costs Buy-In/Support Deployment Value Proposition Key Activities Key Resources Key Partners - Operational commanders - NavSea/MITRE (Navy) and C4IT (CG) to conduct testing and evaluate effectiveness of methodology
  52. -Determining how to streamline the restoration process -Determining necessity of adversarial assessment -NavSea Philadelphia -Cyber Command -Private cybersecurity firms -MITRE -Operational commanders - Test engineers - Streamline testing restoration procedures to minimize operational downtime - Yield best ID of vulnerabilities for correction without risking damage to ship - Development of physical and electronic security testing restoration procedures - Target individual systems for most useful testing applications - Incremental aggression capabilities to maximize - More ships undergoing aggressive testing procedures - Adoption and continued use metrics of the methodology by engineers - Increased vulnerability detection on ships - Using methodology on ships - Full electronic software deployment to customers via web-app - Software design & engineering - Testing and reporting costs - Travel to naval bases -Ships to test on -Trust of ship commanders -Safety parameters of cyber-physical systems Beneficiaries Mission AchievementMission Budget/Costs Buy-In/Support Deployment Value Proposition Key Activities Key Resources Key Partners - Operational commanders - NavSea/MITRE (Navy) and C4IT (CG) to conduct testing and evaluate effectiveness of methodology Week 3 Mission Model Canvas
  53. Week 4 Mission Model Canvas -Format for tracking the testing process and constructing relevant information upon completion -Determining how to streamline the restoration process -NavSea - US Cyber Command -Private cybersecurity firms -MITRE -Operational commanders -Test engineers - Ensure that red-team cybertesting occurs consistently and safely on ships Streamline cybertesting tracking process restoration procedures to minimize operational downtime - Yield comprehensive ID of test parameters and results without risking damage to ship - Development of tool to aid in tracking of testing and restoration physical and electronic security testing restoration procedures - Reduced restoration time - Increased vulnerability detection on ships -consistent, comprehensive cybertesting on naval vessels - Full electronic software deployment to customers via web-app -creation of cyber- testing agency within Fleet Cyber - Software design & engineering - Testing and reporting costs - Travel Installation of app -Ships to test on -Trust of ship commanders -Safety parameters of cyber-physical systems Beneficiaries Mission AchievementMission Budget/Costs Buy-In/Support Deployment Value Proposition Key Activities Key Resources Key Partners - Operational commanders - NavSea / MITRE - Fleet Cyber Command
  54. Week 5 Mission Model Canvas -Format for tracking the testing process and constructing relevant information upon completion -Determining how to streamline the restoration process -NavSea - US Cyber Command -Private cybersecurity firms -MITRE -Operational commanders -Test engineers - Ensure that red-team cybertesting occurs consistently and safely on ships Streamline cybertesting tracking process restoration procedures to minimize operational downtime - Yield comprehensive ID of test parameters and results without risking damage to ship - Development of tool to aid in tracking of testing and restoration physical and electronic security testing restoration procedures - Reduced restoration time - Increased vulnerability detection on ships -consistent, comprehensive cybertesting on naval vessels - Full electronic software deployment to customers via web-app -creation of cyber- testing agency within Fleet Cyber - Software design & engineering - Testing and reporting costs - Travel Installation of app -Ships to test on -Trust of ship commanders -Safety parameters of cyber-physical systems Beneficiaries Mission AchievementMission Budget/Costs Buy-In/Support Deployment Value Proposition Key Activities Key Resources Key Partners - Operational commanders - NavSea / MITRE - Fleet Cyber Command
  55. Week 6 Mission Model Canvas -Fleet Cyber -NavSea -Risk of damaging high value assets reduced -Ability to run wider battery of tests faster and more securely - Removal of Equipment from testing plans -Near perfect simulation of physical component with the black box -More comprehensive testing and remediation of cyber- physical system -Training data for AI model -Microcontrollers with clock speeds fast enough to respond to PLC’s Beneficiaries Mission AchievementMission Budget/Costs Buy-In/Support Deployment Value Proposition Key Activities Key Resources Key Partners - Chief of Naval Operations -Office of Naval Research -Ship commanders - Fleet Cybercommand -Ship commanders -Test engineers - Research Engineers -Development of MVP (materials) -Travel -Installation costs for MVP -Build physical prototype -Develop machine- learning technology - Contract for MVP written and submitted to acquisitions
  56. Week 7 Mission Model Canvas -Fleet Cyber -NavSea -Risk of damaging high value assets reduced and increased clarity on impact of results -Ability to run wider battery of tests faster and more securely - Removal of Equipment from testing plans -loss of monopoly over diagnostics of their product -Simulation of physical component with black box - Integration of existing private sector technologies into testing process and increased clarity of results - Metrics are impossible to know from security issues -More comprehensive testing and remediation of cyber- physical system -Training data for AI model access to private sector simulations -Microcontrollers with clock speeds fast enough to respond to PLC’s Beneficiaries Mission AchievementMission Budget/Costs Buy-In/Support Deployment Value Proposition Key Activities Key Resources Key Partners - Chief of Naval Operations -Office of Naval Research -Ship commanders - Fleet Cybercommand -Ship commanders -Test engineers - Research Engineers -Saboteur: equipment manufacturers -Development of MVP (materials) -Travel -Installation costs for MVP -Build physical prototype and reporting tool -Develop machine- learning technology -Contract for MVP written and submitted to acquisitions
  57. Week 8 Mission Model Canvas -Equipment Manufacturers -Naval Engineers -MITRE -Fleet Cyber -Chief of Naval Operations -Risk of damaging high value assets reduced and increased clarity on impact of results -Ability to run wider battery of tests and convey importance of vulnerabilities to COs - Removal of Equipment from testing plans -loss of monopoly over diagnostics of their product - Integration of existing private sector technologies into testing process and increased clarity of results -Test engineers implement as primary tracking tool for testing; COs utilize as reference point for cyber-threats -access to private sector simulations -Microcontrollers with clock speeds fast enough to respond to PLC’s Beneficiaries Mission AchievementMission Budget/Costs Buy-In/Support Deployment Value Proposition Key Activities Key Resources Key Partners - Chief of Naval Operations -Office of Naval Research -Ship commanders - Fleet Cyber -Ship commanders -Test engineers - Research Engineers -Saboteur: equipment manufacturers -Development of MVP (materials) -Travel -Installation costs for MVP -development of software and visualization integration -contract for acquisitions -modifications for private sector -Contract for MVP written and submitted to acquisitions
  58. Week 9 Mission Model Canvas

×