Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.



Published on

Presentation from Ira "Gus" Hunt, CIA
More at

Published in: Technology


  1. 1. 1
  2. 2. Beyond Big Data Riding theTechnology Wave Ira A. (Gus) Hunt ! Chief Technology Officer
  3. 3. Our MissionWe are the nations first line of defense. We accomplishwhat others cannot accomplish and go where otherscannot go. We carry out our mission by: Collecting information that reveals the plans, intentions and capabilities of our adversaries and provides the basis for decision and action. Producing timely analysis that provides insight, warning and opportunity to the President and decisionmakers charged with protecting and advancing Americas interests. Conducting covert action at the direction of the President to preempt threats or achieve US policy objectives.
  4. 4. 4 Big Bets1   Revolutionize Big Data Exploitation –  Acquire, federate, secure and exploit. Grow the haystack, magnify the needles.2   Accelerate Operational Excellence –  Innovate IT operations and run IT like a business.3   Serve CIA by supporting the IC –  Assume a leadership role in IC activities that matter to CIA; Build to share4   Drive Performance through Talent Management –  Focus on continuous learning and diversity of thought, experience, background
  5. 5. 6 Key Technology Enablers0   Secure Mobility –  Immediate, secure and appropriate access to people, data and tools from anywhere at anytime1   Advanced Mission Analytics—Analytics as a Service –  World-class abilities to discover patterns, correlate information, understand plans and intentions, and find and identify operational targets in a sea of data. Big Data analytics as a service2   Enterprise Widgets and Services –  A customizable, integrated and adaptive webtop that lets analysts, ops officers, and targeters to “have it their way”. Personalization in context.3   Security as a Service –  One environment, all data, protected and secure.--ubiquitous encryption, enterprise authentication, audit, DRM, secure ID propagation, and Gold Version C&A.4   Data Harbor—Data as a Service –  An ultra-high performance data environment that enables CIA missions to acquire, federate, and position and securely exploit huge volumes data. Data in context.5   Cloud Computing—Infrastructure as a Service –  Capacity ahead of demand. Large scale, elastic, commodity hosting, storage, and compute
  6. 6. It’s aBig Data World 6
  7. 7. Google > 100 PB > 1T indexed URLs > 3 million servers> 7.2B page-views/day 7
  8. 8. FaceBook > 1 billion users > 300PB; +> 500TB/day> 35% of world’s photographs 8
  9. 9. YouTube > 1000PB +>72 hours/minute>37 million hours/year > 4 billion views/day 9
  10. 10. World Population > 7,057,065,162 10
  11. 11. Twitter> 124B tweets/year > 390M/day ~4500/sec 11
  12. 12. Global Text Messages > 6.1T per year > 193,000 per second > 876 per person per year 12
  13. 13. US Cell Calls > 2.2 T minutes/year > 19 minutes / person / day(uncompressed < 1 YouTube/year) 13
  14. 14. 3Driving Forces 14
  15. 15. Social Mobile Cloud 15
  16. 16. + + =Big Data 16
  17. 17. + +Increases the velocity of innovation 17
  18. 18. + +Accelerates social Change 18
  19. 19. 19
  20. 20. + + Altered the Flowof Information 20
  21. 21. 3Emerging Forces
  22. 22. Nano Bio Sensors 22
  23. 23. Mobile Sensor Platform Microphone Image 3-axis accelerometer Touch Light Proximity GeolocationCommunicator, Tricorder, Transporter 23
  24. 24. Mobile Health PlatformPacemakerBlood sugar testerInsulin controllerHealth monitorExercise coachRemote tune-upsEarly warning system 24
  25. 25. Mobile Sensor PlatformIdentity by 3-axis accelerometer Gender (71%) Height--tall or short (80%) Weight--heavy or light (80%) You by your gait (100%) Actitracker—Android App 25
  26. 26. + + + + + =The inanimate becomes sentient 26
  27. 27. + + + + + = Smarter Planet Cars drive themselvesMachines know your needs 27
  28. 28. + + + + + =Drive radical efficienciesEnhance social engagementImprove information sharingEnables global reachGreen (automatic routing)Improve our healthStop/prevent crime… 28
  29. 29. Sensors are Really Big1   Sensors are unbounded2   Sensors are promiscuous3   Sensors are indiscriminate
  30. 30. The Internet of Things is Bigger1   Everything is Connected2   Everything Communicates3   Everything is a Sensor
  31. 31. That’s theReally Big Data Challenge of the future 31
  32. 32. Why We Care 32
  33. 33. Why We Care 33
  34. 34. Why We Care 34
  35. 35. Why We Care 35
  36. 36. Impact of Big Data1   Know what we know2   Discover the gaps in our knowledge3   Focus targeting to fill the gaps4   More effective use of expensive or long lead collection assets5   Better global coverage to limit surprise6   Enhance understanding and improve analysis
  37. 37. Implications 37
  38. 38. 4 Rules of Big Data1   It’s the data… - Apologies to James Carville2   Power to the people - Apologies to the Black Panthers3   Latency breeds contempt - Apologies to Aesop4   Context, context, context - Apologies to Lord Harold Samuel
  39. 39. It’s the Data… 39
  40. 40. Data vs Tools—A History Lesson•  Sophisticated tools without the data are useless•  Mediocre tools with the data are frustrating•  Analysts will always opt for frustration over futility, if that is their only option
  41. 41. Our Job1   Leverage the Big Data world2   Find the Information that Matters3   Connect the Dots4   Understand the Plans of our Adversaries Safeguard our national security
  42. 42. TheProblem 42
  43. 43. Our Problem: Which 5K1   Don’t know the future value of data2   We cannot connect dots we don’t have3   Traditional, requirements driven, collection fails in the Big Data world - Can’t task for data you don’t know you do need - The few cannot know the needs of the many - Global Coverage requires Global Data
  44. 44. Characteristics of Big Data1   More is always better2   Signal to noise only gets worse3   Enumeration not modeling4   Requirements are usually hindsight
  45. 45. Data as a Service•  Analysts and operators are not data engineers•  Need insight and understanding•  Ask a question and get a coherent answer•  Cannot know what data sets contain information of value to them•  Imbue data services and tools with those smarts•  Smart Data, smart tools, smarter intelligence 45
  46. 46. Power to the People 46
  47. 47. Today•  Analytics and tools are hard to use•  Specialists are required to derive value•  Skilled people are in short supply•  Algorithms are dense and arcane•  Require a lot of hand curation•  Built for business not for intelligence 47
  48. 48. New Fields of Expertise Data ScientistInformation Engineer 48
  49. 49. Data Science* Data science combines elements from manyfields: Math Statistics Data Engineering Pattern Recognition and Learning Advanced Computing Visualization Uncertainty Modeling Data Warehousing High performance computing * Wikipedia
  50. 50. Big Data Democracy Wins The power of big datacan only be fully realized when it is in the hands of the average user 50
  51. 51. Tomorrow•  Elegant, powerful and easy to use tools and visualizations•  Machines to do more of the heavy lifting•  Intelligent systems that learn from the user•  Correlation not search•  “Curiosity layer”– machines that are curious on your behalf
  52. 52. 7 Universal Constructs for AnalyticsPeople EventsPlaces ConceptsOrganizations ThingsTime 52
  53. 53. User Built Recipes 53
  54. 54. Keep it Simple•  Data Scientists focus on hard problems•  Build reusable components that anyone can apply—Recipes•  Share them widely—Apps Store/Apps Mall —Recipe Book•  Let users assemble components their way •  Experiment and fail quickly to succeed faster
  55. 55. Latency Breeds Contempt 55
  56. 56. Its All About Speed•  Hadoop/Map Reduce—batch •  Flexible, powerful, slow•  Equivalent of Real-Time Map/Reduce •  Flexible, powerful and fast •  Demel, Caffeine, Impala, Apache Drill, Spanner…•  Recursive Streams processing w/ complex analytics•  In-memory—peta-scale RAM architectures •  Distributed, in-memory analytics
  57. 57. Tectonic Technology ShiftsTraditional Processing Mass Analytics/Big Data Data on SAN Data at processor Move Data to Question Move Question to Data Backup Replication management Vertical scaling Horizontal scaling Capacity after demand Capacity ahead of demand DR COOP Size to peak load Dynamic/elastic provisioning Tape SAN SAN Disk Disk SSD RAM limited Peta-scale RAM
  58. 58. New Computing Architectures•  Data close to compute•  Power at the edge•  Optical Computing/Optical Bus•  End of the motherboard—shared pools of everything•  Software defined everything—compute, storage, networking, data center•  Network is the bottleneck and constraint
  59. 59. Context, Context, Context 59
  60. 60. Everything in Your Frame of Reference•  Widgets—Webtop in context to business•  Schema on Read—Data in context to your question•  User assembled analytics—answers in context to your questions•  Elastic computing—computing in context to your demand
  61. 61. ClosingThoughts 61
  62. 62. High Noon in theInformation Age 62
  63. 63. It is nearly within our grasp to compute on all human generated information 63
  64. 64. FaceBook > 1 billion users> 35% of all photographs 64
  65. 65. The inanimate is rapidly becoming sentientSmarter Planet Cars drive themselves Machines know your needs 65
  66. 66. 3 Wave of rd ComputingCognitive Machines Watson 66
  67. 67. + + + + + =Moving faster than government can keep upThe legal system is woefully behindWhat are your rights? Who owns your data?Driving the pace of social changeExponentially increasing cyber threats 67
  68. 68. 68