Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Omniscient H4D 2020 Lessons Learned

17,978 views

Published on

business model, business model canvas, mission model, mission model canvas, customer development, hacking for defense, H4D, lean launchpad, lean startup, stanford, startup, steve blank, pete newell, bmnt, entrepreneurship, I-Corps, JTAC, Computer vision, NSIN

Published in: Education
  • Be the first to comment

  • Be the first to like this

Omniscient H4D 2020 Lessons Learned

  1. Team Omniscient Original Problem DCGS operators need an automated way to review a larger quantity of collected imaging data in order to surface actionable intelligence to leadership. Sponsor Organization: US Air Force Tactical Exploitation of National Capabilities (AF TENCAP) 108 Interviews Supported By: Maj Rose (Sponsor), COL Smith-Heys (Military Mentor), Kevin Ray (Business Mentor), Gus Hernandez (Advisor) Final Problem Analysts lack the computer vision tools to augment their ability to rapidly locate, identify, and analyze objects of interest, which would allow them to focus their time on higher order analysis tasks. Nick Mirda | GSB ‘21 Prior Army Intelligence Officer Summer: BCG Jon Braatz | MS CS ‘20 Computer Vision Research Summer: ! Andrew Fang | BS CS ‘22 Computer Vision Products Summer: Anduril
  2. 90+% of images never reviewed! There’s too much data!
  3. KEY PARTNERS We will liaison with two DCGS (Distributed Common Ground/Surface System) centers, located at Langley AFB and Beale AFB Other Potential Partners: - Intelligence Analysts - USAF Weapons School - DARPA - KesselRun - Air Force Research Lab - Sandia National Labs - MIT Lincoln Lab - DIU - NASIC KEY RESOURCES VALUE PROPOSITIONSKEY ACTIVITIES MISSION ACHIEVEMENT/IMPACT FACTORS ● We will measure mission achievement by: the usability and accuracy of our model on data provided by the Air Force ● Our beneficiaries will measure mission achievement by: the adoption/ease-of-use for analysts whose jobs we will simplify and ability to surface exploitable info in collected data. DEPLOYMENT BUY-IN & SUPPORT MISSION BUDGET/COST ● Budget for training costs (AWS/Azure GPU time) ● Potential costs for proprietary software BENEFICIARIES 1. Operators: Intelligence analysts who parse through images (NASIC, CENTCOM, etc) 2. Decision Makers: High level decision makers who need actionable intelligence quickly and efficiently 1. Narrow problem space 2. Access Data 3. Build model by augmenting YOLO/existing models 4. Test model 1. Analysts who manually identify actionable intel (to provide insight on what is considered valuable data + provide robustly labeled data) 2. AWS/Azure for training 3. Lots of data 1. Workload Reduction: Reduce human hours currently spent on identifying actionable information in images 1. Decrease Intelligence Processing Timeline: Capture information from images faster than a human analyst 1. Reduce Data Backlogs: Parse image database backlogs to surface exploitable images 1. End Users: Analysts who parse through images of AF pilots who are willing to test our software 2. Leadership: Budget authority and operational policy experts 1. Initial deployment: command line service 2. Future deployments: product with a UI/clear instructions for analysts Mission Model Canvas: initial expectations
  4. KEY PARTNERS We will liaison with two DCGS (Distributed Common Ground/Surface System) centers, located at Langley AFB and Beale AFB Other Potential Partners: - Intelligence Analysts - USAF Weapons School - DARPA - KesselRun - Air Force Research Lab - Sandia National Labs - MIT Lincoln Lab - DIU - NASIC KEY RESOURCES VALUE PROPOSITIONSKEY ACTIVITIES MISSION ACHIEVEMENT/IMPACT FACTORS ● We will measure mission achievement by: the usability and accuracy of our model on data provided by the Air Force ● Our beneficiaries will measure mission achievement by: the adoption/ease-of-use for analysts whose jobs we will simplify and ability to surface exploitable info in collected data. DEPLOYMENT BUY-IN & SUPPORT MISSION BUDGET/COST ● Budget for training costs (AWS/Azure GPU time) ● Potential costs for proprietary software BENEFICIARIES 1. Operators: Intelligence analysts who parse through images (NASIC, CENTCOM, etc) 2. Decision Makers: High level decision makers who need actionable intelligence quickly and efficiently 1. Narrow problem space 2. Access Data 3. Build model by augmenting YOLO/existing models 4. Test model 1. Analysts who manually identify actionable intel (to provide insight on what is considered valuable data + provide robustly labeled data) 2. AWS/Azure for training 3. Lots of data 1. Workload Reduction: Reduce human hours currently spent on identifying actionable information in images 1. Decrease Intelligence Processing Timeline: Capture information from images faster than a human analyst 1. Reduce Data Backlogs: Parse image database backlogs to surface exploitable images 1. End Users: Analysts who parse through images of AF pilots who are willing to test our software 2. Leadership: Budget authority and operational policy experts 1. Initial deployment: command line service 2. Future deployments: product with a UI/clear instructions for analysts Mission Model Canvas: initial expectations Initial Thoughts: 1. Help analysts look for “needles in haystacks”
  5. KEY PARTNERS We will liaison with two DCGS (Distributed Common Ground/Surface System) centers, located at Langley AFB and Beale AFB Other Potential Partners: - Intelligence Analysts - USAF Weapons School - DARPA - KesselRun - Air Force Research Lab - Sandia National Labs - MIT Lincoln Lab - DIU - NASIC KEY RESOURCES VALUE PROPOSITIONSKEY ACTIVITIES MISSION ACHIEVEMENT/IMPACT FACTORS ● We will measure mission achievement by: the usability and accuracy of our model on data provided by the Air Force ● Our beneficiaries will measure mission achievement by: the adoption/ease-of-use for analysts whose jobs we will simplify and ability to surface exploitable info in collected data. DEPLOYMENT BUY-IN & SUPPORT MISSION BUDGET/COST ● Budget for training costs (AWS/Azure GPU time) ● Potential costs for proprietary software BENEFICIARIES 1. Operators: Intelligence analysts who parse through images (NASIC, CENTCOM, etc) 2. Decision Makers: High level decision makers who need actionable intelligence quickly and efficiently 1. Narrow problem space 2. Access Data 3. Build model by augmenting YOLO/existing models 4. Test model 1. Analysts who manually identify actionable intel (to provide insight on what is considered valuable data + provide robustly labeled data) 2. AWS/Azure for training 3. Lots of data 1. Workload Reduction: Reduce human hours currently spent on identifying actionable information in images 1. Decrease Intelligence Processing Timeline: Capture information from images faster than a human analyst 1. Reduce Data Backlogs: Parse image database backlogs to surface exploitable images 1. End Users: Analysts who parse through images of AF pilots who are willing to test our software 2. Leadership: Budget authority and operational policy experts 1. Initial deployment: command line service 2. Future deployments: product with a UI/clear instructions for analysts Mission Model Canvas: initial expectations Initial Thoughts: 1. Help analysts look for “needles in haystacks” 2. What do those “needles” look like?
  6. KEY PARTNERS We will liaison with two DCGS (Distributed Common Ground/Surface System) centers, located at Langley AFB and Beale AFB Other Potential Partners: - Intelligence Analysts - USAF Weapons School - DARPA - KesselRun - Air Force Research Lab - Sandia National Labs - MIT Lincoln Lab - DIU - NASIC KEY RESOURCES VALUE PROPOSITIONSKEY ACTIVITIES MISSION ACHIEVEMENT/IMPACT FACTORS ● We will measure mission achievement by: the usability and accuracy of our model on data provided by the Air Force ● Our beneficiaries will measure mission achievement by: the adoption/ease-of-use for analysts whose jobs we will simplify and ability to surface exploitable info in collected data. DEPLOYMENT BUY-IN & SUPPORT MISSION BUDGET/COST ● Budget for training costs (AWS/Azure GPU time) ● Potential costs for proprietary software BENEFICIARIES 1. Operators: Intelligence analysts who parse through images (NASIC, CENTCOM, etc) 2. Decision Makers: High level decision makers who need actionable intelligence quickly and efficiently 1. Narrow problem space 2. Access Data 3. Build model by augmenting YOLO/existing models 4. Test model 1. Analysts who manually identify actionable intel (to provide insight on what is considered valuable data + provide robustly labeled data) 2. AWS/Azure for training 3. Lots of data 1. Workload Reduction: Reduce human hours currently spent on identifying actionable information in images 1. Decrease Intelligence Processing Timeline: Capture information from images faster than a human analyst 1. Reduce Data Backlogs: Parse image database backlogs to surface exploitable images 1. End Users: Analysts who parse through images of AF pilots who are willing to test our software 2. Leadership: Budget authority and operational policy experts 1. Initial deployment: command line service 2. Future deployments: product with a UI/clear instructions for analysts Mission Model Canvas: initial expectations Initial Thoughts: 1. Help analysts look for “needles in haystacks” 2. What do those “needles” look like? 3. Get our hands on imagery data to build a solution
  7. BENEFICIARIES BUYERS PARTNERS We interviewed 108 people all holding a different piece of the puzzle. EXPERTS
  8. How we’re feeling: we got this!
  9. Everyone we talked to had a different problem. Detect Changes We were overwhelmed.
  10. Everyone we talked to had a different problem. Image Clarity Rating (NIIRS) Detect Changes We were overwhelmed.
  11. Everyone we talked to had a different problem. Image Clarity Rating (NIIRS) North Korean MissilesDetect Changes We were overwhelmed.
  12. Everyone we talked to had a different problem. Full-Motion Video (Maven) Image Clarity Rating (NIIRS) North Korean MissilesDetect Changes We were overwhelmed.
  13. Everyone we talked to had a different problem. Full-Motion Video (Maven) Image Clarity Rating (NIIRS) North Korean MissilesDetect Changes And there were a lot of imagery options. We were overwhelmed.
  14. Crisis!
  15. We lost two teammates!
  16. How we’re feeling: yikes
  17. Air Force is shifting to higher level analysis. Imagery Analysis ● Recording object position ● Annotating observations Imagery Understanding ● Situational analysis ● Deep understanding
  18. “Instead of counting objects that can be automatically detected, my analysts can ask why those vehicles are there, really unleashes analytic horsepower.” -Director of Operations @ 13th Intel Squadron Machine learning can automate the drudge work.
  19. ● Tracking all aircraft in flight (NRO). ○ Unsuccessful. ● Project Maven: automatic full-motion video analysis. ○ Mixed results. ● Automatic airfield layout change detection (NGA). ○ Ongoing. ● Identify groupings of tanks (NGA). ○ Ongoing. Previous efforts stumbled due to overambitious goals and improperly labelled data. Many DoD programs to automate imagery analysis, but most are still work-in-progress. ? ? ?
  20. First MVP: a generic computer vision tool. ● it processes analyst imagery to detect objects. ● it runs in the background. ● it uses computer vision. Feedback: 1) “I’ve heard this dozens of times.” 2) “I care less about innovation, more about integration.” We need a specific use case & a way to get in.
  21. A bad pivot: we jumped on the first computer vision solution we saw (computer vision to help bandwidth). • We thought RQ-4 Global Hawks had significant bandwidth limitations that hampered SAR imagery delivery to the base, after speaking with a pilot. We pivoted too early, deviated from beneficiary insights, and were invalidated with further interviews.
  22. How we’re feeling: demoralized :/
  23. Breakthrough!
  24. Insight: The AN-2 is a proxy for what our solution can deliver. ● This is a discrete, strategically relevant problem we can sink our teeth into ● If we can track these, we expand to tracking other equipment ● it’s not easy, but we think we could build it. How do we get in? How do we execute?
  25. 3 meter resolution 50 centimeter resolution Our algorithms would require a lot of good data. We scraped a few thousand images. For free!
  26. Taechon Airfield AN-2 Image #B4993 Date: 05MAY2020 Asset Quantity: 9 Asset Coordinates: 1. 51SYE1354920036 2. 51SYE1347120046 ... 9. 51SYE1319020263c First validated MVP automates AN-2 detection & feeds into analyst workflow. .KML Outputs
  27. KEY PARTNERS Air Force DGS-3 AETs that will be our customers/users. Commercial satellite imagery companies to acquire data to train on. (Maxar, Planet). Innovation and Research organizations to accelerate classified data/system access. (DIU, CRADA, SBIR). KEY RESOURCES VALUE PROPOSITIONSKEY ACTIVITIES MISSION ACHIEVEMENT/IMPACT FACTORS ● We will measure mission achievement by: accuracy of AN-2 classifications from still imagery and anomaly alerts. ● Our beneficiaries will measure mission achievement by: the ability to identify meaningful activity/objects of interest from large data sets and adoption/ease-of-use for analysts. DEPLOYMENT BUY-IN & SUPPORT MISSION BUDGET/COST ● Budget for training costs (AWS/Azure GPU time). ● Unclassified EO data/labels from commercial companies. ● Potential costs for proprietary software. BENEFICIARIES 1. Operators: DGS-3 AET Analysts (Phase 1), ISR Pilots, Sensor Operators, Collection Managers 1. Decision Makers: US Forces and Korea leadership need actionable intel fast. 1. Analysts who manually analyze still imagery (to provide insight on what is data + provide robustly labeled data). 2. Lots of commercial satellite data (Maxar). 1. Decrease AN-2 Imagery Process Time: Quickly scan large images to extract quantity/location of assets of interest. 1. Get AN-2 Anomalies to Analysts/Leaders Fast: Alert analysts to enable them to verify suspicious activity ASAP. 1. Up-to-Date AN-2 Activity/Locations: Know where all AN-2s are and what they do with reliable information. 1. End Users: Imagery analysts & AETs at DGS-3. 1. Leadership: DGS-3 Collections Managers, Combatant Commanders setting ISR priorities. 1. Image Processing on unclass computer. 2. Future deployments: plugin for existing analysis software 3. Continuously-running anomaly notification System 1. Access commercial satellite data of NK airfields. 2. Build and test model on unclassified satellite imagery. 3. Identify integration pathway. Mission Model Canvas: use case identified! Allowed us to: 1. Articulate specific value proposition
  28. KEY PARTNERS Air Force DGS-3 AETs that will be our customers/users. Commercial satellite imagery companies to acquire data to train on. (Maxar, Planet). Innovation and Research organizations to accelerate classified data/system access. (DIU, CRADA, SBIR). KEY RESOURCES VALUE PROPOSITIONSKEY ACTIVITIES MISSION ACHIEVEMENT/IMPACT FACTORS ● We will measure mission achievement by: accuracy of AN-2 classifications from still imagery and anomaly alerts. ● Our beneficiaries will measure mission achievement by: the ability to identify meaningful activity/objects of interest from large data sets and adoption/ease-of-use for analysts. DEPLOYMENT BUY-IN & SUPPORT MISSION BUDGET/COST ● Budget for training costs (AWS/Azure GPU time). ● Unclassified EO data/labels from commercial companies. ● Potential costs for proprietary software. BENEFICIARIES 1. Operators: DGS-3 AET Analysts (Phase 1), ISR Pilots, Sensor Operators, Collection Managers 1. Decision Makers: US Forces and Korea leadership need actionable intel fast. 1. Analysts who manually analyze still imagery (to provide insight on what is data + provide robustly labeled data). 2. Lots of commercial satellite data (Maxar). 1. Decrease AN-2 Imagery Process Time: Quickly scan large images to extract quantity/location of assets of interest. 1. Get AN-2 Anomalies to Analysts/Leaders Fast: Alert analysts to enable them to verify suspicious activity ASAP. 1. Up-to-Date AN-2 Activity/Locations: Know where all AN-2s are and what they do with reliable information. 1. End Users: Imagery analysts & AETs at DGS-3. 1. Leadership: DGS-3 Collections Managers, Combatant Commanders setting ISR priorities. 1. Image Processing on unclass computer. 2. Future deployments: plugin for existing analysis software 3. Continuously-running anomaly notification System 1. Access commercial satellite data of NK airfields. 2. Build and test model on unclassified satellite imagery. 3. Identify integration pathway. Mission Model Canvas: use case identified! Allowed us to: 1. Articulate specific value proposition 2. Focus on the appropriate end users
  29. KEY PARTNERS Air Force DGS-3 AETs that will be our customers/users. Commercial satellite imagery companies to acquire data to train on. (Maxar, Planet). Innovation and Research organizations to accelerate classified data/system access. (DIU, CRADA, SBIR). KEY RESOURCES VALUE PROPOSITIONSKEY ACTIVITIES MISSION ACHIEVEMENT/IMPACT FACTORS ● We will measure mission achievement by: accuracy of AN-2 classifications from still imagery and anomaly alerts. ● Our beneficiaries will measure mission achievement by: the ability to identify meaningful activity/objects of interest from large data sets and adoption/ease-of-use for analysts. DEPLOYMENT BUY-IN & SUPPORT MISSION BUDGET/COST ● Budget for training costs (AWS/Azure GPU time). ● Unclassified EO data/labels from commercial companies. ● Potential costs for proprietary software. BENEFICIARIES 1. Operators: DGS-3 AET Analysts (Phase 1), ISR Pilots, Sensor Operators, Collection Managers 1. Decision Makers: US Forces and Korea leadership need actionable intel fast. 1. Analysts who manually analyze still imagery (to provide insight on what is data + provide robustly labeled data). 2. Lots of commercial satellite data (Maxar). 1. Decrease AN-2 Imagery Process Time: Quickly scan large images to extract quantity/location of assets of interest. 1. Get AN-2 Anomalies to Analysts/Leaders Fast: Alert analysts to enable them to verify suspicious activity ASAP. 1. Up-to-Date AN-2 Activity/Locations: Know where all AN-2s are and what they do with reliable information. 1. End Users: Imagery analysts & AETs at DGS-3. 1. Leadership: DGS-3 Collections Managers, Combatant Commanders setting ISR priorities. 1. Image Processing on unclass computer. 2. Future deployments: plugin for existing analysis software 3. Continuously-running anomaly notification System 1. Access commercial satellite data of NK airfields. 2. Build and test model on unclassified satellite imagery. 3. Identify integration pathway. Mission Model Canvas: use case identified! Allowed us to: 1. Articulate specific value proposition 2. Focus on the appropriate end users 3. Outline a specific integration pathway
  30. How we’re feeling: Yes! On the right track!
  31. Unclassified Satellite Imagery NIPRnet (Unclassified) Unclassified Tool, Integrates with Classified Systems
  32. Unclassified Satellite Imagery Entity Detection Algorithms NIPRnet (Unclassified) Unclassified Integration into Existing Analyst Workflow
  33. Unclassified Satellite Imagery Entity Detection Algorithms NIPRnet (Unclassified) Unclassified Integration into Existing Analyst Workflow Historical Baseline Comparison
  34. Unclassified Satellite Imagery Entity Detection Algorithms NIPRnet (Unclassified) Unclassified Integration into Existing Analyst Workflow Historical Baseline Comparison .KML Output Files
  35. Unclassified Satellite Imagery Entity Detection Algorithms NIPRnet (Unclassified) Unclassified Integration into Existing Analyst Workflow Historical Baseline Comparison Date Location Quantity SIPRnet (Classified) .KML Output Files
  36. Data Uploaded to SIPRnet Unclassified Satellite Imagery Entity Detection Algorithms NIPRnet (Unclassified) Unclassified Integration into Existing Analyst Workflow Historical Baseline Comparison Date Location Quantity SIPRnet (Classified) .KML Output Files .KMLs Uploaded
  37. Data Uploaded to SIPRnet Unclassified Satellite Imagery Entity Detection Algorithms Data integrates with analyst tools NIPRnet (Unclassified) Unclassified Integration into Existing Analyst Workflow Historical Baseline Comparison Date Location Quantity SIPRnet (Classified) .KML Output Files .KMLs Uploaded
  38. Data Uploaded to SIPRnet Unclassified Satellite Imagery Entity Detection Algorithms Data integrates with analyst tools Intel products are built more quickly and effectively. NIPRnet (Unclassified) Unclassified Integration into Existing Analyst Workflow Historical Baseline Comparison Date Location Quantity SIPRnet (Classified) .KML Output Files .KMLs Uploaded
  39. Credibility: Computer Vision: Working Towards Unofficial Demo with Air Combat Command (In 2-3 Weeks) TENCAP Mentorship to TENCAP Letter of Endorsement
  40. Help us build omniscience. MILITARY: Collaboration- how can we help you? PRIMEs: Platforms and Partnerships- how can we work together? NEXT STEPS: COMMERCIAL: Dual use | Venture Funding- let’s chat! Email: hello@omniscientlabs.io $20k | Now $50k | 6 Months Working over summer: ● Develop AN-2 algorithms ● Scope out dual use applications

×