Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Gutenberg H4D Stanford 2019

48,476 views

Published on

business model, business model canvas, mission model, mission model canvas, customer development, hacking for defense, H4D, lean launchpad, lean startup, stanford, startup, steve blank, pete newell, bmnt, AI, Machine Learning, ML

Published in: Education
  • Be the first to comment

  • Be the first to like this

Gutenberg H4D Stanford 2019

  1. Modern deep learning models require extensive labeled training data. This quickly becomes a hard and expensive problem in the Intelligence Community because crowdsourcing data labeling is not possible for classified data. As a result, intelligence analysts are pulled off mission to support data labeling efforts. Large amounts of intel currently are under-analyzed, and analysts are burdened with tedious manual processes to analyze data. AI methods would help rectify this, but efforts to adopt them have yet to succeeded. GUTENBERG: LESSONS LEARNED Remmelt Ammerlaan Alexander Krey Lawrence Moore Nini Moorhead Kelly Shen THEN NOW 114 Interviews MS Comp. Math MBA / MPA MS Comp. Math MBA MS Comp. Science
  2. HOW WE GOT STARTED NORMAL H4D PROCESS Sponsor Problem Statement H4D Teams
  3. HOW WE GOT STARTED GUTENBERG H4D PROCESS: CREATE OUR OWN! H4D Data Labeling in IC
  4. WHERE WE BEGAN: MMC Key goal: automate labeling to improve analyst lives
  5. INITIAL SCOPE Machine Learning Crucial Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future
  6. INITIAL SCOPE Crowdsourcing Would Help Machine Learning is Crucial + Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future
  7. INITIAL SCOPE Crowdsourcing Would Help Machine Learning is Crucial + BUT That Isn’t Possible in the IC Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future
  8. INITIAL “SOLUTION” “Let’s use semi-supervised learning !” - Gutenberg Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future
  9. INITIAL “SOLUTION” “Let’s use semi-supervised learning !” --Gutenberg Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future
  10. TURNED TO SOLUTIONS TOO DEEP, TOO FAST Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future “Just getting access to data is a huge problem” - IC Program Manager
  11. TURNED TO SOLUTIONS TOO DEEP, TOO FAST “Just getting access to data is a huge problem” - IC Program Manager Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future “We don’t know even know what to Google to start learning ML” - IC Program Manager
  12. TURNED TO SOLUTIONS TOO DEEP, TOO FAST “Just getting access to data is a huge problem” - IC Program Manager “We don’t know even know what to Google to start learning ML” - IC Program Manager Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future
  13. KEY LEARNINGS AFTER WEEK 0 ❏ DON’T ALWAYS LISTEN TO NINI ❏ CUSTOMER DISCOVERY REQUIRES BREADTH ❏ SOLVE FOR THE END USER
  14. PIVOT “Data labeling is just the tip of the iceberg” - MAVEN Data Administrator
  15. STAGE 1: DEFINE ANALYST WORKFLOW + PROBLEMS MVP 1: Map of IC text analytics pipeline Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future Data Labeling Train Model Entity Extraction
  16. STAGE 1: DEFINE ANALYST WORKFLOW + PROBLEMS “You may also be interested in…” MVP 2: “Apple News” for daily intelligence MVP 1: Map of IC text analytics pipeline Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future Data Labeling Train Model Entity Extraction
  17. STAGE 1: WHAT WE HEARD Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future Welcome to the graveyard of failed analytic tool pilots… “We’ve tried integrating automated tools before, and they mostly have failed” - DARPA Program Manager
  18. STAGE 1: WHAT WE HEARD Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future Welcome to the graveyard of failed analytic tool pilots… “The higher-ups want one thing, and the analysts want something very different” - Former IC Analyst
  19. KEY LEARNINGS AFTER STAGE 1 ❏ LOW ADOPTION OF ML TOOLS IN IC ❏ DISCONNECT BETWEEN WHAT ANALYSTS AND LEADERS WANT FROM AUTOMATION ❏ ANY TECH SOLUTION MUST BE INTEGRATED INTO ANALYST WORKFLOW
  20. PIVOT Our reaction: “The uncertainty around what AI can do is the single biggest deterrent for adoption” - Data Scientist in IC
  21. STAGE 2: FOSTERING COMMON UNDERSTANDING Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future MVP 3: “AI playbook” to identify capabilities and necessary steps Boolean search Central data repo Automatic doc curation Semantic understanding Automatic insight from text
  22. STAGE 2: FOSTERING COMMON UNDERSTANDING MVP 4: First chapter in “playbook” on full-motion video analysis Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future MVP 3: “AI playbook” to identify capabilities and necessary steps Boolean search Central data repo Automatic doc curation Semantic understanding Automatic insight from text
  23. STAGE 2: WHAT WE HEARD Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future TRIP TO BEALE AFB (PROJECT MAVEN TEST SITE) + Chick-Fil-A!
  24. STAGE 2: WHAT WE HEARD Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future TRIP TO BEALE AFB (PROJECT MAVEN TEST SITE) “The single biggest issue is the iteration time of algorithms” - ISR Analyst at Beale
  25. STAGE 2: WHAT WE HEARD Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future TRIP TO BEALE AFB (PROJECT MAVEN TEST SITE) “We need a standardized process that specifies how to label and train models” - ISR Analyst at Beale
  26. KEY LEARNINGS AFTER STAGE 2 ❏ CURRENT BEST-IN-CLASS SOLUTION DOESN’T MEAN BEST IT COULD BE ❏ THE DOD / IC NEED TO STANDARDIZE TOP- DOWN BEST PRACTICES OF AI
  27. STAGE 3: DEVELOPING A ML PIPELINE Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future MVP 6+7: schema administration train testdeploy declassify organize labelling
  28. STAGE 3: WHAT WE HEARD Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future “Data centralization is world hunger for us” - Colonel, USAF TRIP TO DIU (FORWARD DEPLOYED SOCOM TEAM)
  29. STAGE 3: WHAT WE HEARD Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future “The best structure for data currently means grouped into folders on a shared drive” - Forward Deployed Data Scientist, SOCOM TRIP TO DIU (FORWARD DEPLOYED SOCOM TEAM)
  30. STAGE 3: WHAT WE HEARD Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future “Moving data between security enclaves is our number one blocker” - Deployed data lead, SOCOM TRIP TO DIU (FORWARD DEPLOYED SOCOM TEAM)
  31. KEY LEARNINGS AFTER STAGE 3 ❏ FORMAT STANDARDIZATION IS ESSENTIAL TO USING FULL POTENTIAL OF DATA ❏ DATA SHARING IS HINDERED BY NETWORK SEPARATION AND SECURITY ISSUES
  32. STAGE 4: THE ROAD TO DC Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future Here are the issues we know you are facing Here are the amazing things you can do Here’s the research and open source software to get there
  33. STAGE 4: THE ROAD TO DC Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future We curated a list of senior AI players in the IC to brief
  34. FINAL MVP: DC BRIEFING Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future To adapt to the IC’s needs, a machine learning pipeline must be distributed and non-labor intensive ❏ Three recent trends in academia could make this a reality: ● Semi-supervised learning ● Active / online learning ● Federated learning
  35. STAGE 4: WHAT WE HEARD “Data access is on a need to know basis” - CIA Data Scientist Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future TRIP TO DC (BRIEF DOD AND IC SENIORS)
  36. STAGE 4: WHAT WE HEARD “The infrastructure in place was never designed with automated methods in mind” - CIA Data Scientist Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future TRIP TO DC (BRIEF DOD AND IC SENIORS)
  37. STAGE 4: WHAT WE HEARD “What would make you want to join the IC?” - Deputy Director of National Intelligence Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future TRIP TO DC (BRIEF DOD AND IC SENIORS)
  38. DC TAKEAWAYS ❏ THE IC NEEDS TO CREATE AN ELITE CULTURE AROUND DATA SCIENCE ❏ THE IC MUST RETHINK ITS RELATIONSHIP WITH DATA ❏ THE IC AND DOD NEED TO BE ON THE SAME PAGE FOR AI
  39. SUMMARIZING OUR JOURNEY Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future We came in ready to build...
  40. SUMMARIZING OUR JOURNEY Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future We came in ready to build... ...but that isn’t what the IC needed in the end...
  41. SUMMARIZING OUR JOURNEY Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future We came in ready to build... ...LISTEN TO THE CUSTOMER!! ...but that isn’t what the IC needed in the end...
  42. NEXT STEPS Partners: Follow-Ups Sponsor: Final Deck Gutenberg: New Roles Week 0 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Future ❏ ODNI: Thoughts on IC vs industry differences ❏ NSCAI: Thoughts on ML use in legacy systems ❏ ODNI ❏ IARPA ❏ USAF, 70th ISR ❏ Nini: Vannevar Labs ❏ Kelly: Apple ❏ Lawrence: Waymo ❏ Remmelt: Microsoft ❏ Krey: AWS
  43. THANK YOU Teaching Team Col Pete Newell, Ret Steve Blank Steve Weinstein Tom Bedecarre Jeff Decker Diego Solorzano Cervantes Nate Simon Mentors LtCol Kevin Childs Lisa Wallace Sponsor John Beiler Dean Souleles Key Partners ODNI IARPA SOCOM USAF IQT And all others who helped us along the way!

×