Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Big Data World


Published on

A comprehensive introduction to big data concepts and related technologies.

Published in: Data & Analytics

Big Data World

  1. 1. Big Data World Hossein Zahed 1
  2. 2. Table of Contents • Definitions • Big Data 3V's • Internet Stats • Applications & Examples • Data Science Areas • Identities and Skills • Data Work Flow • Challenges • Data Generation • Data Structure • Cloud Service Providers 2 • Hadoop Ecosystem • Data Visualization • Data Analytics Methods • Data Trends • Programming Languages • NoSQL Databases • Interesting Facts • Interesting Insights • Data Sources • Keywords & Glossary • References
  3. 3. Big Data - Definitions 1. The first documented use of the term “big data” appeared in a 1997 paper by scientists at NASA, describing the problem they had with visualization (i.e. computer graphics) which “provides an interesting challenge for computer systems: data sets are generally quite large, taxing the capacities of main memory, local disk, and even remote disk. We call this the problem of big data. When data sets do not fit in main memory (in core), or when they do not fit even on local disk, the most common solution is to acquire more resources.” (NASA) 2. Data of a very large size, typically to the extent that its manipulation and management present significant logistical challenges. (Oxford English Dictionary) 3. Big data usually includes data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process data within a tolerable elapsed time. (Wikipedia) 3
  4. 4. Big Data – Every 60 Seconds on the Internet 4
  5. 5. Big Data – Basic 3V’s Big Data: Extremely large data sets that may be analyzed computationally to reveal patterns, trends, and associations, especially relating to human behavior and interactions. (Google) 5 Velocity Variety Volume
  6. 6. Big Data – Basic 3V’s 1. Volume: Huge amount of data (Terabytes of Records, Transactions, Tables, Files) 2. Velocity: High rate of data and information flowing into and out of our systems (Batch, Real-time, Streams, Near-time) 3. Variety: Complexity, thousands or more features per data item (Structured, Unstructured, Semi-Structured) 6
  7. 7. Big Data – MoreV’s • Veracity: Accuracy and uncertainty of data • Validity: Data quality, clean/unclean data • Variability: Constantly changing/dynamic data • Value: The potential business value/ROI of data • Venue: Distributed, heterogeneous data from multiple platforms • Vocabulary: Schema, data models, semantics, ontologies, taxonomies, context based • Vagueness: Confusion over the meaning of data • Visibility: Open/Secure data • Visualization: Presentation of data in a readable and accessible way 7
  8. 8. Big Data – Moore’s Law Physical capacity and performance of computers double about every two years! 8
  9. 9. Big Data – Gartner’s EmergingTechnology (2015) 9
  10. 10. Big Data – Internet Stats • The data volumes are exploding, more data has been created in the past two years than in the entire previous history of the human race. • Data is growing faster than ever before and by the year 2020, about 1.7 megabytes of new information will be created every second for every human being on the planet. • By then, our accumulated digital universe of data will grow from 4.4 zettabytes (1021) today to around 44 zettabytes, or 44 trillion gigabytes. • Every second we create new data. For example, we perform 40,000 search queries every second (on Google alone), which makes it 3.5 billion searches per day and 1.2 trillion searches per year. • In Aug 2015, over 1 billion people used Facebook FB +2.39% in a single day. 10
  11. 11. Big Data – Internet Stats – Continued • Facebook users send on average 31.25 million messages and view 2.77 million videos every minute. • We are seeing a massive growth in video and photo data, where every minute up to 300 hours of video are uploaded to YouTube alone. • In 2015, a staggering 1 trillion photos will be taken and billions of them will be shared online. By 2017, nearly 80% of photos will be taken on smart phones. • This year, over 1.4 billion smart phones will be shipped – all packed with sensors capable of collecting all kinds of data, not to mention the data the users create themselves. • By 2020, we will have over 6.1 billion smartphone users globally (overtaking basic fixed phone subscriptions). 11
  12. 12. Internet Stats - Continued • Within five years there will be over 50 billion smart connected devices in the world, all developed to collect, analyze and share data. • By 2020, at least a third of all data will pass through the cloud (a network of servers connected over the Internet). • Distributed computing (performing computing tasks using a network of computers in the cloud) is very real. Google GOOGL +0.63% uses it every day to involve about 1,000 computers in answering a single search query, which takes no more than 0.2 seconds to complete. • The Hadoop (open source software for distributed computing) market is forecast to grow at a compound annual growth rate 58% surpassing $1 billion by 2020. • Estimates suggest that by better integrating big data, healthcare could save as much as $300 billion a year — that’s equal to reducing costs by $1000 a year for every man, woman, and child. 12
  13. 13. Internet Stats - Continued • Estimates suggest that by better integrating big data, healthcare could save as much as $300 billion a year — that’s equal to reducing costs by $1000 a year for every man, woman, and child. • The White House has already invested more than $200 million in big data projects. • For a typical Fortune 1000 company, just a 10% increase in data accessibility will result in more than $65 million additional net income. • Retailers who leverage the full power of big data could increase their operating margins by as much as 60%. • 73% of organizations have already invested or plan to invest in big data by 2016 • Favorite fact: At the moment less than 0.5% of all data is ever analyzed and used, just imagine the potential here. • More stats: 13
  14. 14. Big Data – Consumer Applications • Google Search! • IPhone Siri • Microsoft Cortana • Amazon Suggestions • Spotify Suggestions • Yelp Recommendations • Netflix Recommendations • Google Now! 14
  15. 15. Big Data – Business Applications • Google Ads Searches: Showing relevant ads to users • Predictive Marketing: consumer behavior, users demographic info • Banking: Fraud detection, risk reporting, customer data analysis • Financial: Stocks prediction, Forex • Fraud Detection: spam filtering, online payments • Health: self-aware medics, sports analysis, genomics, health records • Smart Cities: IoT, transportation, traffic, governance, energy, economy • Social Media: friends, topics, videos recommendations • Education: LMS tracks & logs, time spent on subjects 15
  16. 16. Big Data – ResearchApplications • Google Trends: Flu, Zika & Ebola virus, racial justice, supporting refugees and migrant crisis • National Institute of Health: Brain Innovative Neurotechnologies to create a full map of brain functionalities • NASA: Kepler space telescope searching for exoplanets/planets out side of our solar system • Facebook Graphs: Revealing relationships, six-degrees of separation, psychological and personality data • Google Books: Ngram Viewer, History of words, their usage, different meanings 16
  17. 17. Big Data – Example 1 – UPS Post • Insight: Optimize the routing again, predict the maintenance requirements of vehicles. • System: ORION database: engine performance, speed, number of stops, mileage, miles per gallon, GPS, driver behavior, safety habits, emissions, fuel consumption, deliveries, customers, addresses, routes. 250 million+ data points. • Analysis: Advanced mathematical models that provide additional optimization and navigational capabilities to make drivers more efficient. • Result: Saved over 39 million gallons of fuel, avoided 364 million miles, reduced engine idle time by 10 million minutes. 17
  18. 18. Big Data – Example 2 –Walmart • Insight: Customers stock up on certain products in the days leading up to predicted hurricanes. • System: RetailLinksystem records sale, triggers reordering, scheduling, and delivery. Back-office scanners track shipments. Partners use RFID technology to track and coordinate inventories. Data includes daily sales, shipments, returns, purchase orders, invoices. • Analysis: Mines data to get its product mix right under all sorts of varying environmental conditions. • Result: Revenues greater thananyfirm in the US. RFID boosted sales 20%. Gillette increased sales 19%. 18
  19. 19. Big Data – Example 3 – Fraud at eBay • Insight: Fraud spikes mid-week, enabling fraudsters to receive goods by the weekend. Basic fraud pattern= long-distance, high-dollar, expedited shipping. • System: Names, email, addresses, device fingerprinting, IP address, geolocation lookups, time zones, countries in Oracle database of 1.3 billion entries. • Analysis: Run transactions against 600 rules, 20- plus machine learning algorithms. Regularly tweak the fraud rules. • Result: In 2014, prevented $55-million worth of fraudulent transactions. 19
  20. 20. Big Data – Example 4 – Kaiser Permanente • Insight: Kaiser Permanente: HealthConnectexchanges data across all facilities, promotes electronic records. Improved outcomes in cardiovascular disease and saved $1 billion from reduced office visits and lab tests. • System: Pharmaceutical companies have aggregated years of research and development data into medical databases, payorsand providers have digitized patient records, public stakeholders have opened data from clinical trials. 4 billion petabytes. • Analysis: Determine whether standard protocol for a disease produces optimal results. • Result: $300 billion to $450 billion in reduced health- care spending. 20
  21. 21. Big Data vs Small Data 21 Aspect Small Data Big Data Goals Have specific goal May have a goal Location On a single computer On the cloud (multiple servers) Structure Highly structured Semi-structured/unstructured File Types SQL, Excel Documents, multimedia, graphs, tables Data Preparation Prepared by one user Prepared, analyzed, used by different group of users Longevity Short time period Continues for a long time Measurements Single unit (cm) Multiple units (cm, inch,…) Reproducibility Usually reproducible Rarely reproducibility Lost Costs Limited Huge amount Introspection Clear meaning Complex meaning, meaningless Analysis Can be analyzed at once Needs an analysis procedure
  22. 22. Big Data – Data ScienceVenn Diagram 22
  23. 23. Big Data – Professional Identities 23 Data Developer Developer Engineer Data Researcher Researcher Scientist Statistician Data Creative Jack of All Trades Artist Hacker Data Businessperson Leader Businessperson Entrepreneur
  24. 24. Big Data – Five Skill Groups 24 Business ML / Big Data Math / OR Programming Statistics Product Development Unstructured Data Optimization System Administration Visualization Business Structured Data Math Back-End Programming Temporal Statistics Machine Learning Graphic Models Frond-End Programming Surveys and Marketing Big and Distributed Data Bayesian / Monte Carlo Statistics Spatial Statistics Algorithms Science Simulation Data Manipulation Classical Statistics
  25. 25. Big Data – Crossed Identities and Skills 25
  26. 26. Big Data – Scientific Data • Genetic Data (1V): High Volume of data in a structured way • Earthquake Prediction (1V): High Velocity of data, almost real-time • Facial Recognition (1V): High Variety of data • Jet Engine Sensors (2Vs): High Volume + High Velocity (20TB/hour data) • Surveillance Video (2Vs): High Velocity + High Variety of data streaming • Google Books (2Vs): High Volume + High Variety of data (30 Million books) 26
  27. 27. Big Data – Data ScienceWork Flow 27 Start
  28. 28. Big Data – Common Challenges • Anonymity: danger of de-anonymizing public data, social network graphs, medical data,… • Confidentiality: trying to protect data and access levels, storing unimportant data and it’s responsibility • Data Quality: Nearly 95% of spreadsheets have errors • Incomplete or corrupted data • Duplicate records • Typographical errors • Data without context/missing context • Incomplete transformations • Data conversion errors 28
  29. 29. Big Data – Security Challenges • Secure computations in distributed programming frameworks • Security best practices for non-relation data stores • Secure data storage and transaction logs • End-point input validation/filtering • Real-time security/compliance monitoring • Scalable and composable privacy-preserving data mining and analytics • Cryptographically enforced access control and secure communication • Granular access control • Granular audits • Data provenance 29
  30. 30. Big Data – Human Generated Data • Intentional Data: Chats, photos, videos, comments, likes, web searches, emails, cell phone call, text messages, online purchases,… • Meta Data: Data about data, second order data • Photo metadata taken by cameras • Cell phones time and location • Emails To, From, CC, BCC • Social networks connectivity's • Twitter collects 150 pieces of metadata for each tweet 30
  31. 31. Big Data – IPhone 4s Photo EXIF Metadata 31 ExifToolVersion Number : 8.68 File Name : IMG_1031.JPG Directory : . File Size : 3.1 MB File Modification Date/Time : 2011:10:05 01:43:44-07:00 File Permissions : rw-r--r-- FileType : JPEG MIME Type : image/jpeg Exif Byte Order : Big-endian (Motorola, MM) Make : Apple Camera Model Name : iPhone 4S Orientation : Rotate 180 X Resolution : 72Y Resolution : 72 ResolutionUnit : inches Software : 5.0 Modify Date : 2011:08:24 13:13:33YCb Cr Positioning : Centered ExposureTime : 1/286 F Number : 2.4 Exposure Program : Program AE ISO : 64 ExifVersion : 0221 Date/TimeOriginal : 2011:08:24 13:13:33 Create Date : 2011:08:24 13:13:33 ComponentsConfiguration :Y,Cb, Cr, - Shutter SpeedValue : 1/286 ApertureValue : 2.4 BrightnessValue : 6.992671928 Metering Mode : Multi-segment Flash : Auto, Did not fire Focal Length : 4.3 mm SubjectArea : 1631 1223 881 881 FlashpixVersion : 0100 Color Space : sRGB Exif ImageWidth : 3264 Exif Image Height : 2448 Sensing Method : One-chip color area Exposure Mode : AutoWhite Balance : Auto Focal Length In 35mm Format : 35 mm SceneCaptureType : Standard Sharpness : NormalGPS Latitude Ref : North GPS Longitude Ref : West GPSAltitude Ref : Above Sea Level GPSTime Stamp : 21:08:30 GPS Img Direction Ref :True NorthGPS Img Direction : 346.4727273 Compression : JPEG (old-style) ThumbnailOffset : 908Thumbnail Length : 12311 ImageWidth : 3264 Image Height : 2448 Encoding Process : Baseline DCT, Huffman coding Bits Per Sample : 8 Color Components : 3YCb Cr Sub Sampling :YCbCr4:2:0 (2 2) Aperture : 2.4 GPS Altitude : 1222 m Above Sea LevelGPS Latitude : 37 deg 44' 10.80" N GPS Longitude : 119 deg 35' 58.80" W GPS Position : 37 deg 44' 10.80" N, 119 deg 35' 58.80"W Image Size : 3264x2448 Scale FactorTo 35 mm Equivalent: 8.2 Shutter Speed : 1/286Thumbnail Image : (Binary data 12311 bytes, use -b option to extract)CircleOf Confusion : 0.004 mm FieldOfView : 54.4 deg Focal Length : 4.3 mm (35 mm equivalent: 35.0 mm) Hyperfocal Distance : 2.08 m LightValue : 11.3
  32. 32. Big Data – Computer Generated Data • Sources: Cell phones connecting to towers, Satellite radio, GPS connecting, Wi-Fi connections, Web Crawlers,… • Internet of Things (IoT): Information collected an transmitted via IoT devices, Production Lines, Smart Meters, Environmental Monitoring, Industrial Applications, Infrastructure Management, Energy Management, Medical and Healthcare Systems, Smart Buildings,… • Machine to Machine: Server to Server connections, Web Services, Cloud Computations, Real-Time Analytics, Network Monitoring, Routing and Switching,… 32
  33. 33. Big Data – Structured vs. Unstructured Data 33
  34. 34. Big Data – Structured vs. Unstructured Data 34 Features Structured Data Unstructured Data Representation Discrete rows and columns Less defined boundaries and easily addressable Storage Rational Databases or Spreadsheets Unmanaged file structured Metadata Syntax Semantics Integration Tools ETL or ELT Batch processing or manual data entry that involves codes Standard SQL, ADO.NET, ODBC,... OpenXML, JSON, SMTP, SMS, CSV,... Databases MSSQL, Oracle, Excel,… Hadoop, HDInsight, MongoDB,… Content Typically Text Text, Images, Audio, Video, Documents
  35. 35. Big Data – Cloud Computing Services 35 SaaS / DaaS PaaS IaaS
  36. 36. Big Data – Cloud Computing Services Continued • IaaS: Infrastructure as a Service • Servers, Virtual Machines, Storage, Load Balancers, Firewalls, Network • PaaS: Platform as a Service • Web Servers, Databases, Development Tools, Execution Runtime • SaaS: Software as a Service • CRM, ERP, Email, Virtual Desktop, Communications, Games • DaaS: Data as a Service (Free or Commercial) • Stocks, Forex, Google Map, Reddit, Twitter Demographic Data 36
  37. 37. Big Data – Cloud Service Providers • Google Big Data Solutions • Amazon Public Elastic Cloud • Microsoft Azure • OpenStack by Rackspace and NASA • IBM Big Data Solutions • Cloudera • Oracle Cloud Platform • Hortonworks • SAP Big Data 37
  38. 38. Big Data – Cloud Providers Comparison 38
  39. 39. Big Data – Hadoop • Apache Hadoop (pronunciation: /həˈduːp/) is an open-source software framework for distributed storage and distributed processing of very large data sets on computer clusters built from commodity hardware. All the modules in Hadoop are designed with a fundamental assumption that hardware failures are common and should be automatically handled by the framework. (Wikipedia) • History: Doug Cutting, Mike Cafarella and team took the solution provided by Google and started an Open Source Project called HADOOP in 2005 and Doug named it after his son's toy elephant. Now Apache Hadoop is a registered trademark of the Apache Software Foundation. • Hadoop is a Free and Open Source Project 39
  40. 40. Big Data – Hadoop EcosystemArchitecture 40
  41. 41. Big Data – Hadoop Components • HDFS: The Hadoop distributed File System, used to store files across many computers • MapReduce: • Map splits a task into pieces • Reduce combines the output • Has been replaced by YARN (Known as MapReduce 2) • YARN: Can do Batch Processing like MapReduce and also Stream Processing and Graph Processing unlike MapReduce 41
  42. 42. Big Data – Hadoop Components Continued • Pig: Writes MapReduce programs, uses the Pig Latin programming language • Hive: Summarizes queries, Analyzes data, uses the HiveQL programming language • HBase: A NoSQL, not relational, not only SQL database • Storm: Processing and Streaming data • Spark: In Memory Processing (HDD to RAM) • Giraph: Graph Processing for Social Networks data 42
  43. 43. Big Data – Landscape 43
  44. 44. Big Data – Microsoft HDInsight 44
  45. 45. Big Data – Google Big Data Cloud 45
  46. 46. Big Data – Amazon AWS Big Data 46
  47. 47. Big Data –Who Uses Hadoop • Google • Yahoo! • LinkedIn • Facebook • Quantcast • Amazon • IBM 47 • ISI • Spotify • Twitter • Adobe • Ebay • Alibaba • Many others
  48. 48. Big Data – ETL Definition • ETL: Stands for Extract, Transform, Load • Extract: The process of pulling data from storage such as a database • Transform: The process of putting data into a common format • Load: The process of loading data into software for analysis 48 Extract Transform Load
  49. 49. Big Data – ETL in Hadoop • ETL in Hadoop works differently from common databases • Data starts and ends in Hadoop • Hadoop can handle different formats • It doesn’t require as much inspection • No need to be aware of or worry about ETL processes in Hadoop • Make it a point to inspect data 49
  50. 50. Big Data – Monitoring & Anomaly • Monitoring • Detects specific events • Needs specific criterion in advance • Triggers automatic response • Anomaly • Notifies of “unusual activity” • Based on flexible criterial • Doesn’t trigger a response • Instead, invites inspection 50
  51. 51. Big Data –Visualization – Human vs. Computers • Computers spot certain patterns • Computers excel at predictive models • Computers excel at data mining • Humans perceive and interpret better • Humans vision still plays and important role • Humans identify visual patterns • Humans identify anomalies • Humans seeing patterns across groups • Humans interpret content of images better • Humans identify Gestalt Test better 51
  52. 52. Big Data –Visualization – GestaltTest 52
  53. 53. Big Data –Visualization – Best Practices • Prettier graphs are not always better • Never use a false third dimension • Animated and interactive graphs can be distracting • The goal of data visualization is insight • Use proper chart formats for visualization • Choosing the right color scheme (Qualitative, Sequential, Diverging) • Make sure chart alone can tell your story 53
  54. 54. Big Data – Microsoft Excel Role • Excel is the most common data tool • Millions of people use it and know how to deal with it • Professional data miners use it • Excel can do real data science on its own • ODBC interfaces can connect Excel directly to Hadoop • Excel is great for sharing data results • Excel includes interactive PivotTables, Sortable Worksheets, Graphics and Charts 54
  55. 55. Big Data – Data Analytics (DA) Methods • Machine Learning (ML) • Pattern Recognition (PR) • Data Mining (DM) • Natural Language Processing (NLP) • Information Retrieval (IR) • Text Mining (TM) • Predictive Analytics • Business Intelligence (BI) • Prescriptive Analytics 55
  56. 56. Big Data – Machine Learning (ML) • Definition: Machine Learning (LM) is a subfield of computer science (more particularly soft computing) that evolved from the study of pattern recognition and computational learning theory in artificial intelligence. In 1959, Arthur Samuel defined machine learning as a "Field of study that gives computers the ability to learn without being explicitly programmed". (Wikipedia) • Examples: Recommendations, Classifications, Line Regression, Clustering, Neural Networks 56
  57. 57. Big Data – Pattern Recognition (PR) • Definition: Pattern Recognition (PR) is a branch of machine learning that focuses on the recognition of patterns and regularities in data, although it is in some cases considered to be nearly synonymous with machine learning. Pattern recognition systems are in many cases trained from labeled "training" data (supervised learning), but when no labeled data are available other algorithms can be used to discover previously unknown patterns (unsupervised learning). (Wikipedia) • Examples: Face detection, fingerprint verification, screening for tumors and cancers, shape recognition, navigation systems 57
  58. 58. Big Data – Data Mining (DM) • Definition: Data Mining (DM) is an interdisciplinary subfield of computer science. It is the computational process of discovering patterns in large data sets involving methods at the intersection of artificial intelligence, machine learning, statistics, and database systems. The overall goal of the data mining process is to extract information from a data set and transform it into an understandable structure for further use. (Wikipedia) • Examples: Anomaly Detection, Association Rule Learning, Clustering, Classification, Regression, Summarization 58
  59. 59. Big Data – Natural Language Processing (NLP) • Definition: Natural Language Processing (NLP) is a field of computer science, artificial intelligence, and computational linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. (Wikipedia) • Examples: Natural language understanding, enabling computers to derive meaning from human or natural language input; and others involve natural language generation. (SIRI, Cortana) 59
  60. 60. Big Data – Information Retrieval (IR) • Definition: Information Retrieval (IR) is the activity of obtaining information resources relevant to an information need from a collection of information resources. Searches can be based on or on full-text (or other content-based) indexing. (Wikipedia) • Examples: Automated information retrieval systems are used to reduce what has been called "information overload". Many universities and public libraries use IR systems to provide access to books, journals and other documents. Web search engines (Google & Bing) are the most visible IR applications. 60
  61. 61. Big Data –Text Mining (TM) • Definition: Text Mining (TM) also referred to as text data mining, roughly equivalent to text analytics, refers to the process of deriving high-quality information from text. High-quality information is typically derived through the devising of patterns and trends through means such as statistical pattern learning. Text mining usually involves the process of structuring the input text (usually parsing, along with the addition of some derived linguistic features and the removal of others, and subsequent insertion into a database), deriving patterns within the structured data, and finally evaluation and interpretation of the output. (Wikipedia) • Examples: Enterprise Business Intelligence/Data Mining, Competitive Intelligence, National Security/Intelligence, Publishing, Social Media Monitoring, Search/Information Access, Natural Language/Semantic Toolkit or Service, Sentiment Analysis Tools, Listening Platforms 61
  62. 62. Big Data – Predictive Analytics • Definition: Predictive Analytics encompasses a variety of statistical techniques from predictive modeling, machine learning, and data mining that analyze current and historical facts to make predictions about future or otherwise unknown events. In business, predictive models exploit patterns found in historical and transactional data to identify risks and opportunities. Models capture relationships among many factors to allow assessment of risk or potential associated with a particular set of conditions, guiding decision making for candidate transactions. (Wikipedia) • Examples: Actuarial Science, Marketing, Financial Services, Insurance, Telecommunications, Retail, Travel, Healthcare, Child Protection, Pharmaceuticals, Capacity Planning 62
  63. 63. Big Data – Business Intelligence (BI) • Definition: Business Intelligence (BI) can be described as "a set of techniques and tools for the acquisition and transformation of raw data into meaningful and useful information for business analysis purposes". The term "data surfacing" is also more often associated with BI functionality. BI technologies are capable of handling large amounts of unstructured data to help identify, develop and otherwise create new strategic business opportunities. The goal of BI is to allow for the easy interpretation of these large volumes of data. Identifying new opportunities and implementing an effective strategy based on insights can provide businesses with a competitive market advantage and long-term stability. (Wikipedia) • Examples: Measurement, Analytics, Enterprise Reporting, Collaboration Platform, Knowledge management 63
  64. 64. Big Data – Prescriptive Analytics • Definition: Prescriptive analytics is the third and final phase of business analytics (BA) which includes descriptive, predictive and prescriptive analytics. Predictive analytics answers the question what will happen. This is when historical performance data is combined with rules, algorithms, and occasionally external data to determine the probable future outcome of an event or the likelihood of a situation occurring. The final phase is prescriptive analytics, which goes beyond predicting future outcomes by also suggesting actions to benefit from the predictions and showing the implications of each decision option. (Wikipedia) 64
  65. 65. Big Data – Prescriptive Analytics Continued 65
  66. 66. Big Data – Prescriptive Analytics Continued 66
  67. 67. Big Data – InterestTrends 67
  68. 68. Big Data – InterestTrends 68
  69. 69. Big Data – Programming Languages 69 6.3% 8.1% 8.5% 8.8% 12.4% 30.6% 35.0% 36.4% 49.0% MATLAB SPSS PIG / HIVEQL UNIX SHELL JAVA SQL PYTHON SAS R
  70. 70. Big Data – NoSQL Databases 70 Database Type Vendors Wide Column Store Hadoop HBase, Cassandra, Hortonworks, Cloudera, Amazon SimpleDB, IBM Informix Document Store Elastic, MongoDB, Azure DocumentDB, Terrastore, JSON ODM Key Value / Tuple Store Azmazon DynamoDB, Azure Table Storage, Oracle NoSQL Database, Genomu Graph Databases Neo4J, Infinite Graph, Sparksee, InfoGrid, GraphBase Multimodel Databases ArangoDB, OrientDB, RockallDB, FoundationDB Object Databases Versant, db4o, Objectivity, Startcounter, Perst, HSS Database, Magma, EyeDB, NDatabase, ObjectDB
  71. 71. Big Data – NoSQL Databases Continued 71 Database Type Vendors Grid & Cloud Database Solutions Crate Data, Oracle Coherence, GigaSpaces, Infinispan XML Databases EMC Documentum xDB, eXist, Senda, BaseX, QizX, Berkeley DB XML Multidimensional Databases Globals, SciDB, MiniM DB, DaggerDB Multivalue Databases U2, OpenInsight, Reality, OpenQM, ESENT Event Sourcing Event Store, ES4J Time Series / Streaming Databases Axibase, Influxdata, kdb+ Other NoSQL Databases IBM Lutos, eXteremeDB, Yserial, BayesDB, GPUdb, CodernityDB
  72. 72. Big Data – 10 Interesting Facts 1. Every 2 days we create as much information as we did from the beginning of time until 2003. 2. Over 90% of all the data in the world was created in the past 2 years. 3. It is expected that by 2020 the amount of digital information in existence will have grown from 3.2 zettabytes today to 40 zettabytes. 4. The total amount of data being captured and stored by industry doubles every 1.2 years. 5. Every minute we send 204 million emails, generate 1.8 million Facebook likes, send 278 thousand Tweets, and upload 200 thousand photos to Facebook. 72
  73. 73. Big Data – 10 Interesting Facts 6. Google alone processes on average over 40 thousand search queries per second, making it over 3.5 billion in a single day. 7. Around 100 hours of video are uploaded to YouTube every minute and it would take you around 15 years to watch every video uploaded by users in one day. 8. Facebook users share 30 billion pieces of content between them every day. 9. AT&T is thought to hold the world’s largest volume of data in one unique database – its phone records database is 312 terabytes in size, and contains almost 2 trillion rows. 10.The amount of data transferred over mobile networks increased by 81% to 1.5 Exabyte’s (1.5 billion gigabytes) per month between 2012 and 2014. Video accounts for 53% of that total. 73
  74. 74. Big Data – 10 Interesting Insights 1. “The world is one big data problem.” – Andrew McAfee 2. “In God we trust. All others must bring data.” – W. Edwards Deming 3. “Torture the data, and it will confess to anything.” – Ronald Coase 4. “Information is the oil of the 21st century, and analytics is the combustion engine.” - Peter Sondergaard 5. “It’s easy to lie with statistics. It’s hard to tell the truth without statistics.” – Andrejs Dunkels 74
  75. 75. Big Data – 10 Interesting Insights 6. “The goal is to turn data into information, and information into insight.” – Carly Fiorina 7. “The most valuable commodity I know of is information.” – Gordon Gekko 8. “Data really powers everything that we do.” – Jeff Weiner 9. “Numbers have an important story to tell. They rely on you to give them a voice.” – Stephen Few 10.“Data beats emotions.” – Sean Rad 75
  76. 76. Big Data – Free Data Sources • Google Trends: • Google Finance: • Google Freebase: • Wikipedia Content: • U.S. Government Open Data: • Quandl: • World Health Organization: • Amazon Public Datasets: • Facebook Graph: • UNICEF: 76
  77. 77. Big Data – KeyTerms & Glossary • Algorithm • Analytics Platform • Apache Hive • Behavioral Analytics • Big Data Analytics • Business Intelligence • Cascading • Cloud Computing • Concurrency / Concurrent computing • Cluster Analysis • Comparative Analysis 77 • Internet of Things (IOT) • Machine Learning • Metadata • Natural Language Processing • Pattern Recognition • Petabyte • Predictive Analytics • Prescriptive Analytics • Semi-structured Data • Sentiment Analysis • Terabyte • Connection Analytics • Correlation Analysis • Data Analyst • Data Cleansing • Data Mining • Data Model / Data Modeling • Data Warehouse • Descriptive Analytics • ETL • Hadoop • Exabyte
  78. 78. References • online-60-seconds/ • serious-look-10-big-data-v%E2%80%99s • • mind-boggling-facts-everyone-must-read/#7f2504de6c1d • hype-cycle-for-emerging-technologies-2015/ • 78
  79. 79. References - Continued • • question.html • language-for-data-science • • • • insightful-and-thought-provoking-quotes-about-big-data 79
  80. 80. References - Continued • data_revolution_in_us_health_care • big-data • • systems-v1.3/s15-07-data-asset-in-action-technolog.html • • • • Analyzing the Analyzers Book – Harris, Murphy, Vaisman 80