Big Data overview


Published on

Subset of overview course I gave for ESCEN, Silicon Valley

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Big Data overview

  1. 1. Big Data overview ESCEN Alexis Roos Senior Sales Engineer / Architect © Copyright 2013, Alexis Roos,
  2. 2. Course objectives ● Give you a map / big picture and pointers to be able to drill down as you need ● Will cover business side but will also cover technology as without good technical understanding; it is not possible to grasp business side ● Will go over landscape and possibilities and illustrate with a good number of use cases
  3. 3. Proposed Agenda ● What is Big Data? ● Big Data landscape (Tech heavy) ● Business / Use cases ● Discussion
  4. 4. Proposed Agenda ● What is Big Data? ● Big Data landscape ● Business / Use cases ● Discussion
  5. 5. Big Data
  6. 6. Data and Big Data ● Data is the basis for Information Economics are now allowing to store virtually unlimited data ● "“Big data” is high -volume, -velocity and -variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making." Gartner's definition. Year Cost of 1GB 1980 $3,000,000 1990 $8000 2000 $30 2010 $0.08
  7. 7. Data – Information processing 1/2 ● Through processing data becomes Information (knowledge) and knowledge creates insight and insight = success. ● Transaction processing: A sequence of information exchange and related work that is treated as a unit for the purposes of satisfying a request (usually human but not exclusively) aka Online Transaction processing or OLTP Example: you buy an item on Amazon: . Item is placed on hold in Inventory system . Item is placed in shopping cart . System requests CC payment authorization for item . If payment is approved, CC is charged, item is removed from inventory and shipped. -> all of the above or nothing (roll back)
  8. 8. Data – Information processing 2/2 ● Real Time processing Perceived as "immediate" from the originator Ex: trading, payment, online booking, "right" ad delivery, gaming, etc. ● Batch processing: Delayed Execution of a series of programs ("jobs") on a computer without manual intervention. Ex: billing, virus scanning, web indexing, data mining, analytics, etc.
  9. 9. Data – ACID Transaction ● Technical definition: ● Atomicity: each transation is all or nothing ● Consistency: transaction will stay consistent with data rules ● Isolation: Ensures that each transaction is kept isolated from others ● Durability: Once a transaction has been committed, it will remain so, even in the event of power loss, crashes, or errors
  10. 10. Big Data - Applications ● Find deeper insight in data: customers, partners and business. All Industries will be affected. "The software is eating the world" ● Retail: buying patterns, store traffic, etc ● Logistics: track and optimize shipments, etc ● Healthcare: preventive medecine, disease management, etc. ● Social media: optimize usage, ads, etc. ● Finance: buying patterns, portfolio optimization
  11. 11. Big Data – Three dimensions ● Volume ● Amount of data ● Velocity ● Speed at which it arrives ● Variety ● Types of data
  12. 12. Big Data – Volume/Size matters Name Value Example kilobyte (kB) 10^3 Email (7KB), Images, web pages megabyte (MB) 10^6 Ebooks, MP3, SD video etc. gigabyte (GB) 10^9 HD movie terabyte (TB) 10^12 For a single journey across the Atlantic Ocean, a four-engine jumbo jet can create 640 terabytes of data petabyte (PB) 10^15 FB has over 1.5 PB of stored photos exabyte (EB) 10^18 Seagate Technology reported selling 330 exabytes worth of hard drives during the 2011 Fiscal Year zettabyte (ZB) 10^21 WW production and consumption of data. According to International Data Corporation, the total amount of global data is expected to grow to 2.7 zettabytes during 2012 yottabyte (YB) 10^24 Not there yet .. ...
  13. 13. Big Data - Speed ● How fast is new data coming? ● How does this data need to be used or correlated? ● How long is data valuable? ● How fast does data need to be processed? ● This dimension in particular will affect the system architecture
  14. 14. Big Data - Variety ● What type(s) / format(s)? ● Human or machine generated ● Text, location, document, picture, video, click streams, log file, event, etc. ● Is it structured or unstructured? ● Static vs dynamic ● What are relationships/dependencies within data elements?
  15. 15. Proposed Agenda ● What is Big Data? ● Big Data landscape ● Business / Use cases ● Discussion
  16. 16. Big Data landscape Big Data applications are roughly built using three technology layers: ● Storage ● Analytics ● Visualization
  17. 17. For whom?
  18. 18. Big Data landscape ● Storage ● Analytics ● Visualization
  19. 19. Big Data – Storage ● Main logical data models: ● Tabular (represented by rows and columns) - Relational model ● Tree (a set of nodes with parent-children relationship) ● Graph structure (a set of interconnected nodes) ● Document (free structure / unstructured / schema less)
  20. 20. Big Data – Storage ● Physical data models: ● Relational Data Base Mananement Systems (RDBMS) support ACIDity and joins are considered relational. Use SQL language as API. ● Key-value systems basically support get, put, and delete operations based on a primary key. ● Column-oriented systems still use tables but have no joins (joins must be handled within your application). Obviously, they store data by column as opposed to traditional row- oriented databases. This makes aggregations much easier. ● Document-oriented systems store structured "documents" such as JSON or XML but have no joins (joins must be handled within your application). It's very easy to map data from object-oriented software to these systems.
  21. 21. Big Data - Storage ● Not practical to store data on 1 system, but distributing data creates complexity: ● Consistency: means that each client always has the same view of the data. ● Availability: means that all clients can always read and write. ● Partition: tolerance means that the system works well across physical network partitions. ● If system is partitioned, it is only possible to achieve 2 out of 3 properties (known as CAP theorem): CA, AP or CP.
  22. 22. Big data - Storage Source:
  23. 23. Big Data - Storage ATKearny
  24. 24. Big Data landscape ● Storage ● Analytics ● Visualization
  25. 25. Big Data – Analytics ● Process of examining large amounts of data of a variety of types to uncover hidden patterns, unknown correlations and other useful information resulting in business benefits, such as more effective marketing or increased revenue. ● Can work on all forms of data as described before ● Can involve Transactions, Real Time and/or Batch Oriented
  26. 26. Big Data – Analytics ● "Stages" of analytics: ● Business monitoring: traditional BI, Charting, Key Performance Indicator, etc. ● Business insights: uses statistics, data mining, predictive analysis to generate actionable insights: "Intelligent dashboards". Leverages trending, classification, optimization, simulations. ● Business transformation based on data
  27. 27. Big Data – Analytics ● Anticipate and predict
  28. 28. Big Data – Analytics ● Traditional predictive analytics and data mining are designed for relational data or structured data so a whole set of new technologies have evolved for unstructured data. ● Hadoop (batch oriented): "brute force" ● Real Time processing (new trend): optimized for specific use cases ● Machine learning: data intensive
  29. 29. Big Data – Hadoop ● Designed for large scale (100's of terabytes of data) batch oriented information processing: archiving, transformation, exploration, etc. ● Reliable while using commodity HW and open source ● Main components: ● Distributed File System (HDFS) ● Map Reduce: distributed data processing ● Associated infrastructure components, query mechanisms and machine learning
  30. 30. Big Data – Hadoop Example ● Derive meaning from logs: ● Who is using the web site? IP, location, device, etc. ● What pages are they looking at? How long, how often? ● Are they buying? Adding products to cart? Checking out? ● What are the trends?
  31. 31. Big Data – Real-Time ● Goal is to process data from highly dynamic sources in real time ● Data is typically streaming to the processing system and stored / processed directly into memory ● Complex Event Processing has been there for years but need new architecture for Big Data scale and distributed processing: Storm/Kafta are one of the frameworks that could become "Hadoop" of Real-Time
  32. 32. Big Data – Real-Time Example ● Derive meaning from tweets: ● How well brand is trending? ● By time, category? ● Compared to competitors ● Sentiment? ● etc
  33. 33. Big Data – Machine learning ● What is Machine learning?
  34. 34. Big Data – Machine learning ● "A branch of artificial intelligence, that is about the construction and study of systems that can learn from data." Supports Predictive Analytics ● Can perform tasks that are too difficult to specify algorithmically ● Example of applications: ● Computer vision, Natural language processing, Fraud detection, Game playing, Robot locomotion, Sentiment analysis, Adaptive systems, scientific applications, anomaly detection, recommendation engine, personal assistant, etc
  35. 35. Big Data – Example ● Handwritten recognition ● Handcrafted rules will result in large number of rules and exceptions. Best to have a machine that learns from a large training set.
  36. 36. Big Data – Example ● Computer vision: car detection ● First Learning ● Then Testing: Is this a car? Not a carCars
  37. 37. Big Data – Machine learning ● Supervised or unsupervised learning: whether we train the model or the system learns on its own ● Types of information processing: ● Supervised – Classification (discrete) – Regression (continuous) ● Unsupervised – Clustering (discrete)
  38. 38. Big Data – Machine learning Supervised – Classification / Regression ● First teach the model ● Then verify against the model
  39. 39. Big Data – Machine learning Classification ● Classifier (single or multi class): given some set of features with corresponding labels, learn a function to predict the labels from the features x x x x x x x x o o o o o x2 x1
  40. 40. Big Data – Machine learning Classification Many algorithms to choose from: ● SVM ● Neural networks ● Naïve Bayes ● Bayesian network ● Logistic regression ● Randomized Forests ● Boosted Decision Trees ● K-nearest neighbor ● RBMs ● Etc.
  41. 41. Big Data – Machine learning Regression ● Regression allows to fit an equation to a dataset to be able to predict values for new data Example: calculate price of a house: in reality much more than 1 variable: size, number of floors, # of rooms, age, location, etc
  42. 42. Big Data – Machine learning Clustering ● Clustering allows to place data elements into related groups without advance knowledge of the group definitions. ● Example: social network aka similar profiles ● K-means is a popular algorithm for clustering
  43. 43. Big Data – Machine learning ● Predictive analytics techniques usage
  44. 44. Big Data – Machine learning ● Designing a high accuracy learning system “It’s not who has the best algorithm that wins. It’s who has the most data.” Ex: Classify between confusable words. {to, two, too}, {then, than} For breakfast I ate _____ eggs. ● Algorithms ● Perceptron (Logistic regression) ● Winnow ● Memory-based ● Naïve Bayes Training set size (millions) Accuracy
  45. 45. Big Data landscape ● Storage ● Analytics ● Visualization
  46. 46. Big Data – Visualization ● Help overcome information overload ● Allows to see patterns and connections: instantly and overtime ● Focus on specific parts of data but also in relation to other parts: data is relative ● Many different tools and techniques can be used based on data sets
  47. 47. Big Data – Visualization ● Many differents types available: ● 1D, 2D, 3D ● Temporal: timeline, time series, etc ● Advanced types: cloud tag, bubble chart, network graph, rose chart, , spider chart, heatmap, tree map, dependency graph, etc. ● Can allow interactivity (navigate, zoom in/out, slide and dice, etc).
  48. 48. Big Data – Visualization Examples
  49. 49. Proposed Agenda ● What is Big Data? ● Big Data landscape ● Business / Use cases ● Discussion
  50. 50. Big Data – A Word on Privacy ● Currently mostly ignored: Big Brother? ● Everything is being stored (data retention) – Location, calls, SMS, searches, web access, transactions, applications used, contacts, calendar, etc. ● Data doesn't belong to you (Facebook, etc) and may be resold (based on privacy policy) ● Apps can read your calendar, contacts, etc. and upload data on their server ● For now users do not seem to care: they care about service and free (as in $ ). Your phone company is watching Google's drive privacy article Who's afraid of the bad, big data?
  51. 51. Big Data – And Social Media An opportunity to: ● Identify trends: tweets, likes, blogs, page views, etc ● Pinpoint problems: social media data can be used to get sentiment / feedback on products / brands / events (even real-time) ● Predict behavior: what is trend over time and how does it correlate to particular events?
  52. 52. Big Data – Not just 1 device
  53. 53. Big Data – Mobile is growing faster
  54. 54. Big Data – Shopping habits
  55. 55. Big Data - Business models ● Data is the "new oil" ● Every day, 2.5 quintillion bytes of data are created, with 90 percent of the world's data created in the past two years alone. ● Data production will be 44 times greater in 2020 than in 2009.
  56. 56. Big Data - Business models ● Data is the new business model as: ● Cost of HW, SW and networks requires to produce and transport data continues to approach an effective cost of zero ● Even in the physical manufacturing world, cost will go down: robotics, 3D printing, etc. ● Data creates insight which allows to enhance and disrupt existing business models
  57. 57. Big Data - Business models ● Opportunities for: ● Web businesses To increase ARPU ● Enterprises Serve their customers better and improve management of suppliers and partners ● IoT Internet of Things (IoT) or M2M (Machine To Machine) for instance will allow brand new capabilities and services
  58. 58. Big Data - Business models ● Already used by web business (Google, Facebook,etc and moving to Enterprises)
  59. 59. Big Data - Web ● More data can derive more insight which lead to increase ARPU ● Ex: Ad platform Advertisers define ads and campaigns available across web, mobile, TV, etc. On Google properties, Google makes money each time an ad is clicked (CPC). On Network members and content providers, Google makes money each time an ad is clicked or is displayed (CPM) -> Increase relevance and knowledge on the user lead to increased revenues
  60. 60. Big Data - Enterprises ● All Industries are being disrupted
  61. 61. Big Data - Enterprises
  62. 62. Big Data - Enterprises ● Differentiation: satisfy customers, improve existing services and create new service offerings ● Improve processes: merchandising, forecasting, and purchasing to distribution, allocation, and transportation, etc. ● Data as a service: resell information, analysis and insights
  63. 63. Big Data - Enterprises
  64. 64. Big Data - IoT More and more machine are connecting and generating data
  65. 65. Big Data - IoT
  66. 66. Big Data - IoT
  67. 67. Big Data – IoT and Healthcare Home Healthcare / Tele-Health ● Business and Technology trends ● Aging Population ● Increase in Chronic Illnesses ● Demand from patients for home environment and independence ● Costs pressure and scarcity for hospital beds ● Affordable and available telecommunications ● Computing advances: cost, size, power, performance, imaging, etc.
  68. 68. Proposed Agenda ● What is Big Data? ● Big Data landscape ● Business / Use cases ● Discussion