Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Big Data & the Enterprise


Published on

Given at the European Trading Architecture Summit 2012

  • Be the first to comment

  • Be the first to like this

Big Data & the Enterprise

  1. 1. Welcome
  2. 2. It used to be easy…
  3. 3. they all looked pretty much alike
  4. 4. NoSQL BigData MapReduce Graph Document Shared Column EventualBigTable CAP Nothing Oriented Consistency ACID BASE Mongo Coudera HadoopVoldemort Cassandra Dynamo Marklogic Redis Velocity Hbase Hypertable Riak BDB
  5. 5. Now it’s downrightc0nfuZ1nG!
  6. 6. What Happened?
  7. 7. we changed scale
  8. 8. tack ge dwe ch an
  9. 9. thebig dataconundrum
  10. 10. thebig dataconundrum ?
  11. 11. The Internet
  12. 12. Which isn’t mostly textEverything (500,000) Web Pages (40) ~0.01% is web pages Words (0.6) ~1% of that is text Sizes in Petabytes
  13. 13. And there is lots of other stuff out there mobile weather sensors Social Logs data audio video
  14. 14. Gartner80% of business is conducted on unstructured information
  15. 15. Big Data is now a new class of economic asset* *World economic forum 2012
  16. 16. Yet 80% Enterprise Databases < 1TB Reference from 2009
  17. 17. so what does Big Data mean forthe enterprise?
  18. 18. Insight α data >Data beats Algorithms
  19. 19. Backing up a bit…
  20. 20. We live in a world of, largely, private data Where data is often changed and forwarded on
  21. 21. Sometimes we’re a bit more organized!
  22. 22. But most of our data is not generally accessible Core Operational Exposed Data
  23. 23. Sharing is often an afterthought Core Operational Exposed Data
  24. 24. How do we process, acquire, reason about and act upon information?
  25. 25. The Brain Reptilian Primitive operations: balance, temperature regulation, breathing Mammalian Emotion, short-term memory, flee of fight etc (limbic) Neocortex Plan, innovate, problem solve etc
  26. 26. Our intelligence is segregated in disparate worlds
  27. 27. could our corporations bemore intelligent?
  28. 28. Siloed, closed, bespoke data makes our organisations opaque and unresponsive
  29. 29. What if we exposed it all?
  30. 30. So what might that look like?•  Single data store•  Federated, homogenous stores•  Federated, heterogeneous stores
  31. 31. The Google Approach MapReduce Google Filesystem BigTable Tenzing Megastore F1 Dremel Spanner
  32. 32. The Ebay Approach
  33. 33. so is one approachbetter?
  34. 34. Data Volume?TB 0 1 10 100 1000 10,000 We live well within the overlap region
  35. 35. Academic acumen?
  36. 36. Performance Trade-Off Curve•  Volume (pure physical size)•  Velocity (rate of change)•  Variety (number of different types of data, formats and sources)•  Static & Dynamic Complexity (do you need to interpret the affect one message has on another)
  37. 37. ProblemOur ability to model data is much more of a gating factor than raw size, particularly when considering new forms of data Dave Campbell (Microsoft – VLDB Keynote)
  38. 38. Gravitate around a single data model Globally Accessible Application Specific Core Data Models Core Data Model ModelViews /linkages
  39. 39. The data itself follows a similar pattern Globally Accessible Application Specific Core Data Core Data Model dataViews /linkages
  40. 40. Compose Solutions (for now)
  41. 41. Big Data is more than the opportunity for better insight over new data sources
  42. 42. It is the opportunity to make the organisation smarter, simply by making data more accessible
  43. 43. But the harder job, for us, is unifyingthe various domains to make all that data intelligible
  44. 44. Thanks @benstopford