THE CIA’S “GRAND CHALLENGES” WITH BIG DATA from Structure:Data 2013

  • 75,417 views
Uploaded on

Presentation from Ira "Gus" Hunt, CIA …

Presentation from Ira "Gus" Hunt, CIA
#dataconf
More at http://event.gigaom.com/structuredata/

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
75,417
On Slideshare
0
From Embeds
0
Number of Embeds
32

Actions

Shares
Downloads
124
Comments
1
Likes
5

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. 1
  • 2. Beyond Big Data Riding theTechnology Wave Ira A. (Gus) Hunt ! Chief Technology Officer
  • 3. Our MissionWe are the nations first line of defense. We accomplishwhat others cannot accomplish and go where otherscannot go. We carry out our mission by: Collecting information that reveals the plans, intentions and capabilities of our adversaries and provides the basis for decision and action. Producing timely analysis that provides insight, warning and opportunity to the President and decisionmakers charged with protecting and advancing Americas interests. Conducting covert action at the direction of the President to preempt threats or achieve US policy objectives.
  • 4. 4 Big Bets1   Revolutionize Big Data Exploitation –  Acquire, federate, secure and exploit. Grow the haystack, magnify the needles.2   Accelerate Operational Excellence –  Innovate IT operations and run IT like a business.3   Serve CIA by supporting the IC –  Assume a leadership role in IC activities that matter to CIA; Build to share4   Drive Performance through Talent Management –  Focus on continuous learning and diversity of thought, experience, background
  • 5. 6 Key Technology Enablers0   Secure Mobility –  Immediate, secure and appropriate access to people, data and tools from anywhere at anytime1   Advanced Mission Analytics—Analytics as a Service –  World-class abilities to discover patterns, correlate information, understand plans and intentions, and find and identify operational targets in a sea of data. Big Data analytics as a service2   Enterprise Widgets and Services –  A customizable, integrated and adaptive webtop that lets analysts, ops officers, and targeters to “have it their way”. Personalization in context.3   Security as a Service –  One environment, all data, protected and secure.--ubiquitous encryption, enterprise authentication, audit, DRM, secure ID propagation, and Gold Version C&A.4   Data Harbor—Data as a Service –  An ultra-high performance data environment that enables CIA missions to acquire, federate, and position and securely exploit huge volumes data. Data in context.5   Cloud Computing—Infrastructure as a Service –  Capacity ahead of demand. Large scale, elastic, commodity hosting, storage, and compute
  • 6. It’s aBig Data World 6
  • 7. Google > 100 PB > 1T indexed URLs > 3 million servers> 7.2B page-views/day 7
  • 8. FaceBook > 1 billion users > 300PB; +> 500TB/day> 35% of world’s photographs 8
  • 9. YouTube > 1000PB +>72 hours/minute>37 million hours/year > 4 billion views/day 9
  • 10. World Population > 7,057,065,162 10
  • 11. Twitter> 124B tweets/year > 390M/day ~4500/sec 11
  • 12. Global Text Messages > 6.1T per year > 193,000 per second > 876 per person per year 12
  • 13. US Cell Calls > 2.2 T minutes/year > 19 minutes / person / day(uncompressed < 1 YouTube/year) 13
  • 14. 3Driving Forces 14
  • 15. Social Mobile Cloud 15
  • 16. + + =Big Data 16
  • 17. + +Increases the velocity of innovation 17
  • 18. + +Accelerates social Change 18
  • 19. 19
  • 20. + + Altered the Flowof Information 20
  • 21. 3Emerging Forces
  • 22. Nano Bio Sensors 22
  • 23. Mobile Sensor Platform Microphone Image 3-axis accelerometer Touch Light Proximity GeolocationCommunicator, Tricorder, Transporter 23
  • 24. Mobile Health PlatformPacemakerBlood sugar testerInsulin controllerHealth monitorExercise coachRemote tune-upsEarly warning system 24
  • 25. Mobile Sensor PlatformIdentity by 3-axis accelerometer Gender (71%) Height--tall or short (80%) Weight--heavy or light (80%) You by your gait (100%) Actitracker—Android App 25
  • 26. + + + + + =The inanimate becomes sentient 26
  • 27. + + + + + = Smarter Planet Cars drive themselvesMachines know your needs 27
  • 28. + + + + + =Drive radical efficienciesEnhance social engagementImprove information sharingEnables global reachGreen (automatic routing)Improve our healthStop/prevent crime… 28
  • 29. Sensors are Really Big1   Sensors are unbounded2   Sensors are promiscuous3   Sensors are indiscriminate
  • 30. The Internet of Things is Bigger1   Everything is Connected2   Everything Communicates3   Everything is a Sensor
  • 31. That’s theReally Big Data Challenge of the future 31
  • 32. Why We Care 32
  • 33. Why We Care 33
  • 34. Why We Care 34
  • 35. Why We Care 35
  • 36. Impact of Big Data1   Know what we know2   Discover the gaps in our knowledge3   Focus targeting to fill the gaps4   More effective use of expensive or long lead collection assets5   Better global coverage to limit surprise6   Enhance understanding and improve analysis
  • 37. Implications 37
  • 38. 4 Rules of Big Data1   It’s the data… - Apologies to James Carville2   Power to the people - Apologies to the Black Panthers3   Latency breeds contempt - Apologies to Aesop4   Context, context, context - Apologies to Lord Harold Samuel
  • 39. It’s the Data… 39
  • 40. Data vs Tools—A History Lesson•  Sophisticated tools without the data are useless•  Mediocre tools with the data are frustrating•  Analysts will always opt for frustration over futility, if that is their only option
  • 41. Our Job1   Leverage the Big Data world2   Find the Information that Matters3   Connect the Dots4   Understand the Plans of our Adversaries Safeguard our national security
  • 42. TheProblem 42
  • 43. Our Problem: Which 5K1   Don’t know the future value of data2   We cannot connect dots we don’t have3   Traditional, requirements driven, collection fails in the Big Data world - Can’t task for data you don’t know you do need - The few cannot know the needs of the many - Global Coverage requires Global Data
  • 44. Characteristics of Big Data1   More is always better2   Signal to noise only gets worse3   Enumeration not modeling4   Requirements are usually hindsight
  • 45. Data as a Service•  Analysts and operators are not data engineers•  Need insight and understanding•  Ask a question and get a coherent answer•  Cannot know what data sets contain information of value to them•  Imbue data services and tools with those smarts•  Smart Data, smart tools, smarter intelligence 45
  • 46. Power to the People 46
  • 47. Today•  Analytics and tools are hard to use•  Specialists are required to derive value•  Skilled people are in short supply•  Algorithms are dense and arcane•  Require a lot of hand curation•  Built for business not for intelligence 47
  • 48. New Fields of Expertise Data ScientistInformation Engineer 48
  • 49. Data Science* Data science combines elements from manyfields: Math Statistics Data Engineering Pattern Recognition and Learning Advanced Computing Visualization Uncertainty Modeling Data Warehousing High performance computing * Wikipedia
  • 50. Big Data Democracy Wins The power of big datacan only be fully realized when it is in the hands of the average user 50
  • 51. Tomorrow•  Elegant, powerful and easy to use tools and visualizations•  Machines to do more of the heavy lifting•  Intelligent systems that learn from the user•  Correlation not search•  “Curiosity layer”– machines that are curious on your behalf
  • 52. 7 Universal Constructs for AnalyticsPeople EventsPlaces ConceptsOrganizations ThingsTime 52
  • 53. User Built Recipes 53
  • 54. Keep it Simple•  Data Scientists focus on hard problems•  Build reusable components that anyone can apply—Recipes•  Share them widely—Apps Store/Apps Mall —Recipe Book•  Let users assemble components their way •  Experiment and fail quickly to succeed faster
  • 55. Latency Breeds Contempt 55
  • 56. Its All About Speed•  Hadoop/Map Reduce—batch •  Flexible, powerful, slow•  Equivalent of Real-Time Map/Reduce •  Flexible, powerful and fast •  Demel, Caffeine, Impala, Apache Drill, Spanner…•  Recursive Streams processing w/ complex analytics•  In-memory—peta-scale RAM architectures •  Distributed, in-memory analytics
  • 57. Tectonic Technology ShiftsTraditional Processing Mass Analytics/Big Data Data on SAN Data at processor Move Data to Question Move Question to Data Backup Replication management Vertical scaling Horizontal scaling Capacity after demand Capacity ahead of demand DR COOP Size to peak load Dynamic/elastic provisioning Tape SAN SAN Disk Disk SSD RAM limited Peta-scale RAM
  • 58. New Computing Architectures•  Data close to compute•  Power at the edge•  Optical Computing/Optical Bus•  End of the motherboard—shared pools of everything•  Software defined everything—compute, storage, networking, data center•  Network is the bottleneck and constraint
  • 59. Context, Context, Context 59
  • 60. Everything in Your Frame of Reference•  Widgets—Webtop in context to business•  Schema on Read—Data in context to your question•  User assembled analytics—answers in context to your questions•  Elastic computing—computing in context to your demand
  • 61. ClosingThoughts 61
  • 62. High Noon in theInformation Age 62
  • 63. It is nearly within our grasp to compute on all human generated information 63
  • 64. FaceBook > 1 billion users> 35% of all photographs 64
  • 65. The inanimate is rapidly becoming sentientSmarter Planet Cars drive themselves Machines know your needs 65
  • 66. 3 Wave of rd ComputingCognitive Machines Watson 66
  • 67. + + + + + =Moving faster than government can keep upThe legal system is woefully behindWhat are your rights? Who owns your data?Driving the pace of social changeExponentially increasing cyber threats 67
  • 68. 68