1
Beyond Big Data   Riding theTechnology Wave                  Ira A. (Gus) Hunt      !           Chief Technology Officer
Our MissionWe are the nations first line of defense. We accomplishwhat others cannot accomplish and go where otherscannot ...
4 Big Bets1	     Revolutionize Big               Data Exploitation          –  Acquire, federate, secure and exploit. Grow...
6 Key Technology Enablers0	     Secure Mobility           –    Immediate, secure and appropriate access to people, data an...
It’s aBig Data  World           6
Google       > 100 PB  > 1T indexed URLs  > 3 million servers> 7.2B page-views/day                        7
FaceBook      > 1 billion users   > 300PB; +> 500TB/day> 35% of world’s photographs                          8
YouTube        > 1000PB  +>72 hours/minute>37 million hours/year > 4 billion views/day                         9
World Population  > 7,057,065,162                    10
Twitter> 124B tweets/year    > 390M/day     ~4500/sec                     11
Global Text Messages     > 6.1T per year   > 193,000 per second  > 876 per person per year                              12
US Cell Calls   > 2.2 T minutes/year  > 19 minutes / person / day(uncompressed < 1 YouTube/year)                          ...
3Driving Forces                 14
Social   Mobile         Cloud                 15
+   +      =Big Data               16
+       +Increases the velocity of  innovation                        17
+      +Accelerates social   Change                     18
19
+      + Altered the   Flowof Information                 20
3Emerging Forces
Nano   Bio       Sensors                 22
Mobile Sensor Platform       Microphone       Image       3-axis accelerometer       Touch       Light       Proximity    ...
Mobile Health PlatformPacemakerBlood sugar testerInsulin controllerHealth monitorExercise coachRemote tune-upsEarly warnin...
Mobile Sensor PlatformIdentity by 3-axis accelerometer    Gender (71%)    Height--tall or short (80%)    Weight--heavy or ...
+       +          +     +       +          =The inanimate becomes       sentient                        26
+         +          +      +         +          =     Smarter Planet  Cars drive themselvesMachines know your needs      ...
+          +       +           +          +       =Drive radical efficienciesEnhance social engagementImprove information ...
Sensors are Really Big1	     Sensors are unbounded2	     Sensors are promiscuous3	     Sensors are indiscriminate
The Internet of Things is Bigger1	     Everything is Connected2	     Everything   Communicates3	     Everything is a Sensor
That’s theReally Big Data Challenge of the future                           31
Why We  Care         32
Why We  Care         33
Why We  Care         34
Why We  Care         35
Impact of Big Data1	     Know what we know2	     Discover the gaps in our knowledge3	     Focus targeting to fill the gaps...
Implications               37
4 Rules of Big Data1	     It’s the data…                         - Apologies to James Carville2	     Power to the people  ...
It’s the Data…                 39
Data vs Tools—A History                            Lesson•  Sophisticated tools without the data are   useless•  Mediocre ...
Our Job1	     Leverage the Big Data world2	     Find the Information that Matters3	     Connect the Dots4	     Understand ...
TheProblem          42
Our Problem: Which 5K1	     Don’t know the future value of data2	     We cannot connect dots we don’t have3	     Tradition...
Characteristics of Big Data1	     More is always better2	     Signal to noise only gets worse3	     Enumeration not modeli...
Data as a Service•  Analysts and operators are not data engineers•  Need insight and understanding•  Ask a question and ge...
Power to the  People               46
Today•  Analytics and tools are hard to use•  Specialists are required to derive value•  Skilled people are in short suppl...
New Fields of Expertise    Data ScientistInformation Engineer                       48
Data Science* Data science combines elements from manyfields:      Math      Statistics      Data Engineering      Pattern...
Big Data Democracy Wins The power of big datacan only be fully realized when it is in the hands  of the average user      ...
Tomorrow•    Elegant, powerful and easy to use tools and     visualizations•    Machines to do more of the heavy lifting• ...
7 Universal Constructs for                  AnalyticsPeople          EventsPlaces          ConceptsOrganizations   ThingsT...
User Built Recipes                53
Keep it Simple•  Data Scientists focus on hard problems•  Build reusable components that anyone   can apply—Recipes•  Shar...
Latency Breeds   Contempt                 55
Its All About Speed•  Hadoop/Map Reduce—batch  •    Flexible, powerful, slow•  Equivalent of Real-Time Map/Reduce  •    Fl...
Tectonic Technology ShiftsTraditional Processing       Mass Analytics/Big Data             Data on SAN     Data at process...
New Computing Architectures•  Data close to compute•  Power at the edge•  Optical Computing/Optical Bus•  End of the mothe...
Context,    Context,         Context                   59
Everything in Your Frame of                       Reference•  Widgets—Webtop in context to business•  Schema on Read—Data ...
ClosingThoughts           61
High Noon     in theInformation Age                  62
It is nearly within our grasp  to compute on all human    generated information                            63
FaceBook    > 1 billion users> 35% of all photographs                       64
The inanimate is rapidly    becoming sentientSmarter Planet  Cars drive themselves     Machines know your needs           ...
3 Wave of   rd ComputingCognitive Machines        Watson                     66
+           +            +                   +           +            =Moving faster than government can keep upThe legal ...
68
Upcoming SlideShare
Loading in …5
×

THE CIA’S “GRAND CHALLENGES” WITH BIG DATA from Structure:Data 2013

76,336 views

Published on

Presentation from Ira "Gus" Hunt, CIA
#dataconf
More at http://event.gigaom.com/structuredata/

Published in: Technology
1 Comment
6 Likes
Statistics
Notes
No Downloads
Views
Total views
76,336
On SlideShare
0
From Embeds
0
Number of Embeds
71,580
Actions
Shares
0
Downloads
145
Comments
1
Likes
6
Embeds 0
No embeds

No notes for slide

THE CIA’S “GRAND CHALLENGES” WITH BIG DATA from Structure:Data 2013

  1. 1. 1
  2. 2. Beyond Big Data Riding theTechnology Wave Ira A. (Gus) Hunt ! Chief Technology Officer
  3. 3. Our MissionWe are the nations first line of defense. We accomplishwhat others cannot accomplish and go where otherscannot go. We carry out our mission by: Collecting information that reveals the plans, intentions and capabilities of our adversaries and provides the basis for decision and action. Producing timely analysis that provides insight, warning and opportunity to the President and decisionmakers charged with protecting and advancing Americas interests. Conducting covert action at the direction of the President to preempt threats or achieve US policy objectives.
  4. 4. 4 Big Bets1   Revolutionize Big Data Exploitation –  Acquire, federate, secure and exploit. Grow the haystack, magnify the needles.2   Accelerate Operational Excellence –  Innovate IT operations and run IT like a business.3   Serve CIA by supporting the IC –  Assume a leadership role in IC activities that matter to CIA; Build to share4   Drive Performance through Talent Management –  Focus on continuous learning and diversity of thought, experience, background
  5. 5. 6 Key Technology Enablers0   Secure Mobility –  Immediate, secure and appropriate access to people, data and tools from anywhere at anytime1   Advanced Mission Analytics—Analytics as a Service –  World-class abilities to discover patterns, correlate information, understand plans and intentions, and find and identify operational targets in a sea of data. Big Data analytics as a service2   Enterprise Widgets and Services –  A customizable, integrated and adaptive webtop that lets analysts, ops officers, and targeters to “have it their way”. Personalization in context.3   Security as a Service –  One environment, all data, protected and secure.--ubiquitous encryption, enterprise authentication, audit, DRM, secure ID propagation, and Gold Version C&A.4   Data Harbor—Data as a Service –  An ultra-high performance data environment that enables CIA missions to acquire, federate, and position and securely exploit huge volumes data. Data in context.5   Cloud Computing—Infrastructure as a Service –  Capacity ahead of demand. Large scale, elastic, commodity hosting, storage, and compute
  6. 6. It’s aBig Data World 6
  7. 7. Google > 100 PB > 1T indexed URLs > 3 million servers> 7.2B page-views/day 7
  8. 8. FaceBook > 1 billion users > 300PB; +> 500TB/day> 35% of world’s photographs 8
  9. 9. YouTube > 1000PB +>72 hours/minute>37 million hours/year > 4 billion views/day 9
  10. 10. World Population > 7,057,065,162 10
  11. 11. Twitter> 124B tweets/year > 390M/day ~4500/sec 11
  12. 12. Global Text Messages > 6.1T per year > 193,000 per second > 876 per person per year 12
  13. 13. US Cell Calls > 2.2 T minutes/year > 19 minutes / person / day(uncompressed < 1 YouTube/year) 13
  14. 14. 3Driving Forces 14
  15. 15. Social Mobile Cloud 15
  16. 16. + + =Big Data 16
  17. 17. + +Increases the velocity of innovation 17
  18. 18. + +Accelerates social Change 18
  19. 19. 19
  20. 20. + + Altered the Flowof Information 20
  21. 21. 3Emerging Forces
  22. 22. Nano Bio Sensors 22
  23. 23. Mobile Sensor Platform Microphone Image 3-axis accelerometer Touch Light Proximity GeolocationCommunicator, Tricorder, Transporter 23
  24. 24. Mobile Health PlatformPacemakerBlood sugar testerInsulin controllerHealth monitorExercise coachRemote tune-upsEarly warning system 24
  25. 25. Mobile Sensor PlatformIdentity by 3-axis accelerometer Gender (71%) Height--tall or short (80%) Weight--heavy or light (80%) You by your gait (100%) Actitracker—Android App 25
  26. 26. + + + + + =The inanimate becomes sentient 26
  27. 27. + + + + + = Smarter Planet Cars drive themselvesMachines know your needs 27
  28. 28. + + + + + =Drive radical efficienciesEnhance social engagementImprove information sharingEnables global reachGreen (automatic routing)Improve our healthStop/prevent crime… 28
  29. 29. Sensors are Really Big1   Sensors are unbounded2   Sensors are promiscuous3   Sensors are indiscriminate
  30. 30. The Internet of Things is Bigger1   Everything is Connected2   Everything Communicates3   Everything is a Sensor
  31. 31. That’s theReally Big Data Challenge of the future 31
  32. 32. Why We Care 32
  33. 33. Why We Care 33
  34. 34. Why We Care 34
  35. 35. Why We Care 35
  36. 36. Impact of Big Data1   Know what we know2   Discover the gaps in our knowledge3   Focus targeting to fill the gaps4   More effective use of expensive or long lead collection assets5   Better global coverage to limit surprise6   Enhance understanding and improve analysis
  37. 37. Implications 37
  38. 38. 4 Rules of Big Data1   It’s the data… - Apologies to James Carville2   Power to the people - Apologies to the Black Panthers3   Latency breeds contempt - Apologies to Aesop4   Context, context, context - Apologies to Lord Harold Samuel
  39. 39. It’s the Data… 39
  40. 40. Data vs Tools—A History Lesson•  Sophisticated tools without the data are useless•  Mediocre tools with the data are frustrating•  Analysts will always opt for frustration over futility, if that is their only option
  41. 41. Our Job1   Leverage the Big Data world2   Find the Information that Matters3   Connect the Dots4   Understand the Plans of our Adversaries Safeguard our national security
  42. 42. TheProblem 42
  43. 43. Our Problem: Which 5K1   Don’t know the future value of data2   We cannot connect dots we don’t have3   Traditional, requirements driven, collection fails in the Big Data world - Can’t task for data you don’t know you do need - The few cannot know the needs of the many - Global Coverage requires Global Data
  44. 44. Characteristics of Big Data1   More is always better2   Signal to noise only gets worse3   Enumeration not modeling4   Requirements are usually hindsight
  45. 45. Data as a Service•  Analysts and operators are not data engineers•  Need insight and understanding•  Ask a question and get a coherent answer•  Cannot know what data sets contain information of value to them•  Imbue data services and tools with those smarts•  Smart Data, smart tools, smarter intelligence 45
  46. 46. Power to the People 46
  47. 47. Today•  Analytics and tools are hard to use•  Specialists are required to derive value•  Skilled people are in short supply•  Algorithms are dense and arcane•  Require a lot of hand curation•  Built for business not for intelligence 47
  48. 48. New Fields of Expertise Data ScientistInformation Engineer 48
  49. 49. Data Science* Data science combines elements from manyfields: Math Statistics Data Engineering Pattern Recognition and Learning Advanced Computing Visualization Uncertainty Modeling Data Warehousing High performance computing * Wikipedia
  50. 50. Big Data Democracy Wins The power of big datacan only be fully realized when it is in the hands of the average user 50
  51. 51. Tomorrow•  Elegant, powerful and easy to use tools and visualizations•  Machines to do more of the heavy lifting•  Intelligent systems that learn from the user•  Correlation not search•  “Curiosity layer”– machines that are curious on your behalf
  52. 52. 7 Universal Constructs for AnalyticsPeople EventsPlaces ConceptsOrganizations ThingsTime 52
  53. 53. User Built Recipes 53
  54. 54. Keep it Simple•  Data Scientists focus on hard problems•  Build reusable components that anyone can apply—Recipes•  Share them widely—Apps Store/Apps Mall —Recipe Book•  Let users assemble components their way •  Experiment and fail quickly to succeed faster
  55. 55. Latency Breeds Contempt 55
  56. 56. Its All About Speed•  Hadoop/Map Reduce—batch •  Flexible, powerful, slow•  Equivalent of Real-Time Map/Reduce •  Flexible, powerful and fast •  Demel, Caffeine, Impala, Apache Drill, Spanner…•  Recursive Streams processing w/ complex analytics•  In-memory—peta-scale RAM architectures •  Distributed, in-memory analytics
  57. 57. Tectonic Technology ShiftsTraditional Processing Mass Analytics/Big Data Data on SAN Data at processor Move Data to Question Move Question to Data Backup Replication management Vertical scaling Horizontal scaling Capacity after demand Capacity ahead of demand DR COOP Size to peak load Dynamic/elastic provisioning Tape SAN SAN Disk Disk SSD RAM limited Peta-scale RAM
  58. 58. New Computing Architectures•  Data close to compute•  Power at the edge•  Optical Computing/Optical Bus•  End of the motherboard—shared pools of everything•  Software defined everything—compute, storage, networking, data center•  Network is the bottleneck and constraint
  59. 59. Context, Context, Context 59
  60. 60. Everything in Your Frame of Reference•  Widgets—Webtop in context to business•  Schema on Read—Data in context to your question•  User assembled analytics—answers in context to your questions•  Elastic computing—computing in context to your demand
  61. 61. ClosingThoughts 61
  62. 62. High Noon in theInformation Age 62
  63. 63. It is nearly within our grasp to compute on all human generated information 63
  64. 64. FaceBook > 1 billion users> 35% of all photographs 64
  65. 65. The inanimate is rapidly becoming sentientSmarter Planet Cars drive themselves Machines know your needs 65
  66. 66. 3 Wave of rd ComputingCognitive Machines Watson 66
  67. 67. + + + + + =Moving faster than government can keep upThe legal system is woefully behindWhat are your rights? Who owns your data?Driving the pace of social changeExponentially increasing cyber threats 67
  68. 68. 68

×