Big Data,      New Physics, and    Geospatial Super-Food     Tristan Sternson, InfoReady Managing Director1               ...
My Background – Tristan Sternson    Past 12 years focussed purely on IM / BI Solutions    Started InfoReady in 2008    Pri...
Who is InfoReady?        Pure-Play Information Management and Business Intelligence        Consulting firm        Team Inf...
Big Data Definition    Datasets that grow so large that they become difficult to work with,     including; capture, storag...
The Big Data Opportunity        V35                          © 2012 Infoready Pty Ltd
Big Data – Why the hype?    By 2015, nearly 3B people will be online, pushing the data    created and shared to nearly 8 z...
Business Value                  Business leaders frequently make        1 in 3    decisions based on information they     ...
Big Data Trends        20%       80%8                       © 2012 Infoready Pty Ltd
What the Industry Analysts say       Gartner predicts Big Data to be    one of the top-10 strategic initiatives           ...
What the Industry Analysts say     Key take-aways from Analyst perspectives          Gartner TDWI     Data will grow expon...
Enterprise Intelligence         vs. Enterprise Amnesia11                                © 2012 Infoready Pty Ltd
Trend: Organizations Are Getting Dumber                               Available                              Observation  ...
Trend: Organizations Are Getting Dumber                               Available                              Observation  ...
Algorithms at Dead End.           You Can’t       Squeeze Knowledge         Out of a Pixel.14                         © 20...
No Context              scrila34@msn.com15                               © 2012 Infoready Pty Ltd
Context, definition     Better understanding      something by taking into      account the things around it.16           ...
Information in Context … and Accumulating                 scrila34@msn.com       Job     Applicant                        ...
The Puzzle Metaphor     Imagine an ever-growing pile of puzzle pieces of varying sizes,     shapes and colors     What it ...
Puzzling           1000 pieces   100 pieces           100%          10%                         (duplicates)           12 ...
20   © 2012 Infoready Pty Ltd
21   © 2012 Infoready Pty Ltd
First Discovery – “we found Dora?”22                                   © 2012 Infoready Pty Ltd
Sorting Algorithm23                  © 2012 Infoready Pty Ltd
Another Puzzle …24                 © 2012 Infoready Pty Ltd
10 Mins – Completed Dora Puzzels25                                 © 2012 Infoready Pty Ltd
Data Finds Data26                © 2012 Infoready Pty Ltd
Obvious Duplicates in Front Of Your Eyes27                                         © 2012 Infoready Pty Ltd
Incremental Context – Incremental Discovery     10:00am   START      1min    “I can see Dora”     1min      “How many puzz...
Lots of Sorted Pieces29                      © 2012 Infoready Pty Ltd
Pieces in Context30                  © 2012 Infoready Pty Ltd
Quickly we find meaning (90mins)            66 pieces            of            1190 pieces            only 5.5%31         ...
Wow 1%         11 pieces         of         1190 pieces         only 1%32                     © 2012 Infoready Pty Ltd
Koala, Possum or Monkey?33                         © 2012 Infoready Pty Ltd
Foundation34           © 2012 Infoready Pty Ltd
More Data Finds Data35                     © 2012 Infoready Pty Ltd
Out of Tablespace…36                   © 2012 Infoready Pty Ltd
Incremental Context – Incremental Discovery     55min    “Second puzzle is definitely a motorbike – I can see a           ...
How Context Accumulates     With each new observation … one of three assertions are made:     1) Un-associated; 2) placed ...
Overstated Population     Unique Identities                                        True Population                        ...
Counting Is Difficult                          Mark R Smith                        (614) 13-123-123                       ...
The Rise and Fall of a Population     Unique Identities                                        True Population            ...
Data Triangulation                      New Record                                           Mark R Smith                 ...
Big Data [in context]. New Physics.     More data: better the predictions      – Lower false positives      – Lower false ...
Big Data           Pile of ____   In Context44                                © 2012 Infoready Pty Ltd
One Form of Context: “Expert Counting”     Is it 5 people each with 1 account … or is it 1     person with 5 accounts?    ...
Expert Counting: Degrees of Difficulty                                                                   Deceit           ...
Key Features Enable Expert Counting     People          Cars                Router     Name            Make               ...
Consider Lying Identical Twins     PASSPORT   #123                Sue                      PASSPORT   #123                ...
The same thing cannot be in     two places … at the same     time.     Two different things cannot     occupy the same spa...
Space  Time Enables Absolute Disambiguation     People          Cars                Router     Name            Make       ...
“Life Arcs” Are Also Telling           Bill Smith                      Bill Smith            13/4/67                      ...
OMG52         © 2012 Infoready Pty Ltd
Space-Time-Travel     Cell phones are generating a staggering amount of     geo-locational data – 600B transactions per da...
Powerful Predictions     Prediction with 87% certainty where you will be     next Thursday at 5:35pm     Names of the top ...
Consequences     Space-time-travel data is the ultimate     biometric     It will enable enormous opportunity     It will ...
Macro Trends56                  © 2012 Infoready Pty Ltd
The Greater the Context, the Greater the Value                                           Data                             ...
Time Is Of The Essence                                                        The better the                              ...
Enterprise Intelligence         One Plausible Journey         One Plausible Journey59                               © 2012...
Sense and Respond       Observation         Space        New     Observations                         What you know60     ...
Sense and Respond       Observation         Space                                  Data Finds                             ...
Sense and Respond                                  Explore and Reflect       Observation         Space                    ...
Sense and Respond                                    Explore and Reflect       Observation         Space                  ...
Sense and Respond                                     Explore and Reflect       Observation         Space                 ...
Closing Thoughts65                      © 2012 Infoready Pty Ltd
The most competitive organizations     are going to make sense of what they are observing            fast enough to do som...
Wish This On The Enemy                               Available                              Observation     Computing Powe...
The Way Forward: Enterprise Intelligence                               Available                              Observation ...
Questions?     Email:   tristan.sternson@infoready.com.au     Twitter: http://www.twitter.com/tsternson     Blog:    www.i...
Upcoming SlideShare
Loading in …5
×

Australian CIO Summit 2012: Big Data, New Physics, and Geospatial Super-Food by Tristan Sternson, Managing Director, InfoReady

602 views
527 views

Published on

Australian CIO Summit 2012: Big Data, New Physics, and Geospatial Super-Food by Tristan Sternson, Managing Director, InfoReady

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
602
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Australian CIO Summit 2012: Big Data, New Physics, and Geospatial Super-Food by Tristan Sternson, Managing Director, InfoReady

  1. 1. Big Data, New Physics, and Geospatial Super-Food Tristan Sternson, InfoReady Managing Director1 © 2012 Infoready Pty Ltd
  2. 2. My Background – Tristan Sternson Past 12 years focussed purely on IM / BI Solutions Started InfoReady in 2008 Prior Roles – Accenture Data Management & Architecture / IM Lead, PWC Consulting / IBM Personally designed and deployed and led many large IM and DW application in Australia and UK Thought leader in Information Management in Australia and APAC Early adopter Data Governance, Big Data, Industry Data Models, Appliance DW solutions2 © 2012 Infoready Pty Ltd
  3. 3. Who is InfoReady? Pure-Play Information Management and Business Intelligence Consulting firm Team InfoReady career IM and BI Experts • One of the fastest growing consulting firms in Australia. • IM Focused Tier One Consulting capability. • Focus - people, process and technology • Assisting companies turn valuable information into actionable intelligence. • Strategy, Architecture, Solution Design & Delivery3 © 2012 Infoready Pty Ltd
  4. 4. Big Data Definition Datasets that grow so large that they become difficult to work with, including; capture, storage, search, sharing, analytics, and visualization. Benefits of working with larger and larger datasets allowing analysts to "spot business trends, prevent diseases, combat crime.” We haven’t seen anything yet, as more devices come online, eg; mobile, airborn, logs, cameras, microphones etc…4 Wikipedia - 2012 © 2012 Infoready Pty Ltd
  5. 5. The Big Data Opportunity V35 © 2012 Infoready Pty Ltd
  6. 6. Big Data – Why the hype? By 2015, nearly 3B people will be online, pushing the data created and shared to nearly 8 zettabytes. 30 billion pieces of content were added to Facebook this past month by 600M plus users. More than 2B videos were watched on YouTube … yesterday. In the US mobile phone users between the ages of 18 and 24 send an incredible 110 text messages per day. 32B searches were performed last month … on Twitter. Worldwide IP traffic will quadruple by 2015.6 © 2012 Infoready Pty Ltd
  7. 7. Business Value Business leaders frequently make 1 in 3 decisions based on information they don’t trust, or don’t have 1 in 2 access to leaders say theythey ’need to Business the information don’t have do their jobs of CIOs cited “Business intelligence & 83% Analytics” as part of their visionary plans to enhance competitiveness of CEOs recognise they need to better 60% understand information more rapidly in order to make swift decisions7 7 © 2012 Infoready Pty Ltd
  8. 8. Big Data Trends 20% 80%8 © 2012 Infoready Pty Ltd
  9. 9. What the Industry Analysts say Gartner predicts Big Data to be one of the top-10 strategic initiatives for 20129 © 2012 Infoready Pty Ltd
  10. 10. What the Industry Analysts say Key take-aways from Analyst perspectives Gartner TDWI Data will grow exponentially Fusion of structured and unstructured data The connection between big data and advanced analytics will get even stronger Future users will not be able to put all useful information into a single data warehouse 10 © 2012 Infoready Pty Ltd
  11. 11. Enterprise Intelligence vs. Enterprise Amnesia11 © 2012 Infoready Pty Ltd
  12. 12. Trend: Organizations Are Getting Dumber Available Observation Computing Power Growth Space Context Enterprise Amnesia Sensemaking Algorithms Time12 © 2012 Infoready Pty Ltd
  13. 13. Trend: Organizations Are Getting Dumber Available Observation Computing Power Growth Space WHY? Context Sensemaking Algorithms Time13 © 2012 Infoready Pty Ltd
  14. 14. Algorithms at Dead End. You Can’t Squeeze Knowledge Out of a Pixel.14 © 2012 Infoready Pty Ltd
  15. 15. No Context scrila34@msn.com15 © 2012 Infoready Pty Ltd
  16. 16. Context, definition Better understanding something by taking into account the things around it.16 © 2012 Infoready Pty Ltd
  17. 17. Information in Context … and Accumulating scrila34@msn.com Job Applicant Top 200 Customer Criminal Investigation Identity Thief17 © 2012 Infoready Pty Ltd
  18. 18. The Puzzle Metaphor Imagine an ever-growing pile of puzzle pieces of varying sizes, shapes and colors What it represents is unknown – there is no picture on hand Is it one puzzle, 15 puzzles, or 1,500 different puzzles? Some pieces are duplicates, missing, incomplete, low quality, or have been misinterpreted Some pieces may even be professionally fabricated lies Until you take the pieces to the table and attempt assembly, you don’t know what you are dealing with18 © 2012 Infoready Pty Ltd
  19. 19. Puzzling 1000 pieces 100 pieces 100% 10% (duplicates) 12 pieces 12 pieces 100% 100% (pure noise) 66 pieces 66%19 © 2012 Infoready Pty Ltd
  20. 20. 20 © 2012 Infoready Pty Ltd
  21. 21. 21 © 2012 Infoready Pty Ltd
  22. 22. First Discovery – “we found Dora?”22 © 2012 Infoready Pty Ltd
  23. 23. Sorting Algorithm23 © 2012 Infoready Pty Ltd
  24. 24. Another Puzzle …24 © 2012 Infoready Pty Ltd
  25. 25. 10 Mins – Completed Dora Puzzels25 © 2012 Infoready Pty Ltd
  26. 26. Data Finds Data26 © 2012 Infoready Pty Ltd
  27. 27. Obvious Duplicates in Front Of Your Eyes27 © 2012 Infoready Pty Ltd
  28. 28. Incremental Context – Incremental Discovery 10:00am START 1min “I can see Dora” 1min “How many puzzles are there?” 8min “Are there 1000 pieces and 3 or 4 puzzles?” 10min 2 x Dora puzzles complete 12min “I have blue sky and an animal” 18mins “The other puzzle is more colourful – maybe a red motorbike” 23min “we’ve found Jenny Sanders – can I search google on my iPhone for the picture?” 35min “How can we have 2 pieces the same?”28 © 2012 Infoready Pty Ltd
  29. 29. Lots of Sorted Pieces29 © 2012 Infoready Pty Ltd
  30. 30. Pieces in Context30 © 2012 Infoready Pty Ltd
  31. 31. Quickly we find meaning (90mins) 66 pieces of 1190 pieces only 5.5%31 © 2012 Infoready Pty Ltd
  32. 32. Wow 1% 11 pieces of 1190 pieces only 1%32 © 2012 Infoready Pty Ltd
  33. 33. Koala, Possum or Monkey?33 © 2012 Infoready Pty Ltd
  34. 34. Foundation34 © 2012 Infoready Pty Ltd
  35. 35. More Data Finds Data35 © 2012 Infoready Pty Ltd
  36. 36. Out of Tablespace…36 © 2012 Infoready Pty Ltd
  37. 37. Incremental Context – Incremental Discovery 55min “Second puzzle is definitely a motorbike – I can see a wheel and seat” 65min Motorcycle coming together very quickly 70min “It’s definitely a koala” 75min “The koala has a baby” 83min “The middle piece of the bike is missing – do I really need it, I know what it is” 88min “These are both Australian puzzles” 114min One of the kids starts isolating pieces that are causing her “noise” 130min 7 chunks emerge from 7 piles of SORTED pieces 165min Pieces beginning to come together quite quickly and picture starts to really emerge37 © 2012 Infoready Pty Ltd
  38. 38. How Context Accumulates With each new observation … one of three assertions are made: 1) Un-associated; 2) placed near like neighbors; or 3) connected Must favor the false negative New observations sometimes reverse earlier assertions Some observations produce novel discovery As the working space expands, computational effort increases Given sufficient observations, there can come a tipping point Thereafter, confidence improves while computational effort decreases!38 © 2012 Infoready Pty Ltd
  39. 39. Overstated Population Unique Identities True Population Observations39 © 2012 Infoready Pty Ltd
  40. 40. Counting Is Difficult Mark R Smith (614) 13-123-123 DL: 00001234 Mark Smith 6/12/1978 0413123123 File 2 File 140 © 2012 Infoready Pty Ltd
  41. 41. The Rise and Fall of a Population Unique Identities True Population Observations41 © 2012 Infoready Pty Ltd
  42. 42. Data Triangulation New Record Mark R Smith (614) 13-123-123 DL: 00001234 Mark Smith 6/12/1978 0413123123 Mark Randy Smith 0413123123 DL: 00001234 File 2 File 142 © 2012 Infoready Pty Ltd
  43. 43. Big Data [in context]. New Physics. More data: better the predictions – Lower false positives – Lower false negatives More data: bad data good – Suddenly glad your data is not perfect More data: less compute43 © 2012 Infoready Pty Ltd
  44. 44. Big Data Pile of ____ In Context44 © 2012 Infoready Pty Ltd
  45. 45. One Form of Context: “Expert Counting” Is it 5 people each with 1 account … or is it 1 person with 5 accounts? Is it 20 cases of H1N1 in 20 cities … or one case reported 20 times? If one cannot count … one cannot estimate vector or velocity (direction and speed). Without vector and velocity … prediction is nearly impossible.45 © 2012 Infoready Pty Ltd
  46. 46. Expert Counting: Degrees of Difficulty Deceit Bob Jones Ken Wells 123455 550119 Incompatible Features Bob Jones bjones@hotmail Fuzzy 123455 Bob Jones Robert T Jonnes Exactly 123455 000123455 Same Bob Jones Bob Jones 123455 12345546 © 2012 Infoready Pty Ltd
  47. 47. Key Features Enable Expert Counting People Cars Router Name Make Device ID Address Model Make Date of Birth Year Model Phone License Plate No. Firmware Vers. Passport VIN Asset ID Nationality Owner Etc. Biometric Etc. Etc.47 © 2012 Infoready Pty Ltd
  48. 48. Consider Lying Identical Twins PASSPORT #123 Sue PASSPORT #123 3/3/84 Sue Uberstan 3/3/84 Exp 2011 Uberstan Exp 2011 “Same person – trust me.” Fingerprint DNA Most Trusted Authority48 © 2012 Infoready Pty Ltd
  49. 49. The same thing cannot be in two places … at the same time. Two different things cannot occupy the same space … at the same time.49 © 2012 Infoready Pty Ltd
  50. 50. Space Time Enables Absolute Disambiguation People Cars Router Name Make Device ID When When When Address Model Make Where Where Where Date of Birth Year Model Phone License Plate No. Firmware Vers. Passport VIN Asset ID Nationality Owner Etc. Biometric Etc. Etc.50 © 2012 Infoready Pty Ltd
  51. 51. “Life Arcs” Are Also Telling Bill Smith Bill Smith 13/4/67 13/4/67 Melbourne, Victoria Brisbane, Queensland Address History Address History Melbourne, Vic 2008-2008 Carina, QLD 2005-2009 St Kilda, Vic 2005-2008 Brisbane, QLD 2005-2005 Hampton, Vic 1996-2005 Bondi, NSW 1990-2005 Brighton, Vic 1984-1996 Carina, QLD 1982-199051 © 2012 Infoready Pty Ltd
  52. 52. OMG52 © 2012 Infoready Pty Ltd
  53. 53. Space-Time-Travel Cell phones are generating a staggering amount of geo-locational data – 600B transactions per day being created in the US alone This data is being “de-identified” and shared with third parties – in volume and in real-time Your movement quickly reveals where you spend your time (e.g., evenings vs. working hours) Re-identification (figuring out who is who) is somewhat trivial53 © 2012 Infoready Pty Ltd
  54. 54. Powerful Predictions Prediction with 87% certainty where you will be next Thursday at 5:35pm Names of the top 10 people you co-locate with, not at home and not at work Intelligence service preempts the next mass protest in real-time Robbery of a convenience store is about to happen at 10:42pm54 © 2012 Infoready Pty Ltd
  55. 55. Consequences Space-time-travel data is the ultimate biometric It will enable enormous opportunity It will unravel one’s secrets It will challenge existing notions of privacy And, it’s here now and more to come55 © 2012 Infoready Pty Ltd
  56. 56. Macro Trends56 © 2012 Infoready Pty Ltd
  57. 57. The Greater the Context, the Greater the Value Data in Context Value of Data Pile of Data (Big) Records Managed (Ludicrous Big)57 © 2012 Infoready Pty Ltd
  58. 58. Time Is Of The Essence The better the predictions … the Batch faster they will be wanted. Day Willingness to Wait “Why did we have to wait until the Hour end of the day for the smart answer?” 200ms Real-Time (Iffy) Relevance (Totally)58 © 2012 Infoready Pty Ltd
  59. 59. Enterprise Intelligence One Plausible Journey One Plausible Journey59 © 2012 Infoready Pty Ltd
  60. 60. Sense and Respond Observation Space New Observations What you know60 © 2012 Infoready Pty Ltd
  61. 61. Sense and Respond Observation Space Data Finds Data Relevance Finds the Sensor (200ms) ? Decide61 © 2012 Infoready Pty Ltd
  62. 62. Sense and Respond Explore and Reflect Observation Space Deep Reflection Curated Data Data Finds Pattern Data Discovery Directed Attention Relevance Finds the Sensor (200ms) ? Relevance Decide Find You62 © 2012 Infoready Pty Ltd
  63. 63. Sense and Respond Explore and Reflect Observation Space Deep Reflection Curated Data Data Finds Pattern Data Discovery Directed Attention Relevance NEW Finds the Sensor (200ms) ? INTERESTS Decide63 © 2012 Infoready Pty Ltd
  64. 64. Sense and Respond Explore and Reflect Observation Space Deep Reflection Curated Data Data Finds Pattern Data Discovery Directed Attention Relevance NEW Finds the Sensor (200ms) ? INTERESTS Decide64 Report and Manage © 2012 Infoready Pty Ltd
  65. 65. Closing Thoughts65 © 2012 Infoready Pty Ltd
  66. 66. The most competitive organizations are going to make sense of what they are observing fast enough to do something about it while they are observing it.66 © 2012 Infoready Pty Ltd
  67. 67. Wish This On The Enemy Available Observation Computing Power Growth Space Context Enterprise Amnesia Sensemaking Algorithms Time67 © 2012 Infoready Pty Ltd
  68. 68. The Way Forward: Enterprise Intelligence Available Observation Computing Power Growth Space Context Sensemaking Algorithms Time68 © 2012 Infoready Pty Ltd
  69. 69. Questions? Email: tristan.sternson@infoready.com.au Twitter: http://www.twitter.com/tsternson Blog: www.infoready.com.au LinkedIn: http://www.linkedin.com/in/tristansternson69 © 2012 Infoready Pty Ltd

×