Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Big dataproposal


Published on

Published in: Technology
  • Login to see the comments

Big dataproposal

  2. 2. WHAT’S NEW ABOUT BIG DATA YOU SAY…  Traditionally, analytics on data internal to an organization  Customer data  ERP data  Some pre-digested external data like market research  Sophisticated analytics using new data sources  Social data  Website data Low density, fine grained and massive “Most EDWs are < 2TB”
  3. 3. LOW DENSITY, HIGH VOLUME DATA Amul comment data: 18000 * 140 * 60 * 24 * 30 = 100 GB per month Category Unique visitors Retail – Luxury goods 20 million Retail – consumer goods 30 million Retail – tickets 26 million Social Media Website data Traditional technologies cannot handle this low-density, high volume data
  4. 4. SKELETON OF A BIG DATA PROJECT Internal Data External Data TB - PBs Actionable report Analytics Workflow
  5. 5. HOW DO THE BIG GUYS DO IT?  Build data centers  Buy or build custom big-data software  Hire ETL engineers who manage bringing data into the system  Hire admins to keep it all running  Hire data scientists to come up with interesting questions  Hire developers who can translate questions into programs
  6. 6. Lots of upfront investment Long time to get started Lots of risks BIG DATA PROJECT ENTAILS
  7. 7. LANDSCAPE IS CHANGING  Advent of public clouds  Cheap, reliable storage  Provision 10-1000s of machines in a couple of minutes  Pay as you go, grow as you please  Free / inexpensive big-data software  Hadoop, Hive, Presto
  8. 8. CLOUD PRIMITIVES  Persistent object store e.g. AWS S3  Reliability is basically solved for you (*)  Ability to provision clusters with pre-built images in a couple of minutes  Pay by the hour (or by the minute)  Spot instances (AWS)  Relational DB as a Service  MySQL, PostgreSQL etc
  10. 10. CLOUD’S COMPUTE FLEXIBILITY  Analytics workloads tend to be bursty  Most orgs struggle to predict usage 2-3 months down the line  Tend to overprovision compute  Result: < 30% utilization of their hardware  Cloud allows you to scale up and down  Trickier for a big data system, but possible Chen et al, VLDB 2012 Provision for peak workload
  11. 11. BIG DATA SOFTWARE  Many open source projects  Hadoop based on Google’s MR paper (Yahoo)  Hive (SQL-on-Hadoop)  Presto (Fast SQL)  Production ready, running at scale at Yahoo, FB and many other environments
  12. 12. ENTER HADOOP  Open-source implementation of Map-reduce used by Google to index trillions of web pages  Allows programmers to write distributed programs using map and reduce abstractions  Ability to run these programs on large amounts of data  Uses bunch of cheap hardware, can tolerate failures
  13. 13. HADOOP SCALES!
  14. 14. HIVE: SQL ON HADOOP  Facebook had a Multi Petabyte Warehouse  Had 80+ engineers writing Hadoop jobs  Files are insufficient data abstractions  Need tables, schemas, partitions, indices  SQL is highly popular  So, implement SQL on top of Hadoop  Allowed non-programmers to process all the data  FB open-sourced it  Production ready  Processes 25PB of data in FB  Processes 20PB of data at Qubole
  15. 15. HIVE ALLOWS YOU TO DESCRIBE DATA  Example  My data lives in Amazon S3 in a specific location  It is in delimited text format  Please create a virtual table for me  Number of data formats: JSON, Text, Binary, Avro, ProtoBuf, Thrift  Analytics is often a downstream process  Conversion of data is time consuming and not productive create external table nation (N_NATIONKEY INT, N_NAME STRING, N_REGIONKEY INT, N_COMMENT STRING) ROW FORMAT DELIMITED STORED AS TEXTFILE LOCATION 's3n://public-qubole/datasets/tpch5G/nation';
  16. 16. HIVE EXTENSIBILITY  Connect to external data sources like MongoDB  Write code to understand new data formats - serdes  Custom UDFs in Java  Plug in custom code in python or any other language SELECT TRANSFORM (hosting_ids, user_id, d) USING 'python' AS (hosting_ranks_array, user_id, d) FROM s_table;
  17. 17. HIVE ALLOWS YOU TO QUERY THE DATA  SQL-Like  Query is parallelized using Hadoop as execution engine Select count(*) from nation; Count(*) Count(*) Sum()
  18. 18. HIVE EXECUTION  Split Hive query into multiple Hadoop/MR jobs  Run Job 1, save intermediate output to HDFS  Run Job 2..  Return results  Data parallel because every hadoop job runs on number of machines T11 100MB T12 100MB 10 files 5 files 5 files
  19. 19. TASK PARALLELISM T1 T2 T3 100MB 100MB 100MB10 files
  20. 20. EXECUTION MODEL 1 T1 100MB T2 100MB T3 100MB10 files • Only 100MB of memory required • Can stop and resume • Allows for multiplexing multiple pipelines • Can tolerate failures • Spilling can be expensive • Time to first result is high
  21. 21. EXECUTION MODEL 2 T1 100MB T2 100MB T3 100MB10 files • Task parallelism • Needs 3X memory • No spilling, hence much faster • Early first results • Stop and resume is trickier • Multiplexing is more difficult • Cannot tolerate failures
  22. 22. ENTER PRESTO  Hive was EM1 and had associated disadvantages  Internal project at Facebook to implement EM2 (Presto)  Use case was interactive queries over the same data  Open sourced late 2013  Promised much faster query performance  In-memory processing, aggressive pipelining  Supports all the data formats that Hive does  Can’t plug in user code at this point, vanilla SQL
  23. 23. CONTRASTING HIVE AND PRESTO Hive Presto Uses Hadoop MR for execution (EM1) Pipelined execution model (EM2) Spills intermediate data to FS Intermediate data in memory Can tolerate failures Does not tolerate failures Automatic join ordering User-specified join ordering Can handle joins of two large tables One table needs to fit in memory Supports grouping sets Does not support GS Plug in custom code Cannot plug in custom code More data types Limited data types Hive 0.11 vs Presto 0.60
  24. 24. PERFORMANCE COMPARISON • Presto is 2.5-7x faster • But, some queries just run out of memory • Contrasts the execution models
  25. 25. IN A NUTSHELL
  26. 26. SAMPLE SETUP Cloud Storage Sqoop Application Sync Heavy duty queries Interactive queries
  27. 27. CRYSTAL BALL  Hive is actively working on task parallelism as part of the Stinger Initiative  Presto is also making rapid progress in bridging some of its gaps  There are other open source projects:  Impala, Shark, Drill, Tajo  Lots of goodies for users
  28. 28. CONCLUSION  Big Data Analytics is becoming accessible and affordable  Public clouds give flexibility and change economics  Hive and Presto provide intuitive and powerful ways to interact with your data