SlideShare a Scribd company logo
1 of 28
HIVE AND PRESTO
FOR BIG DATA
ANALYTICS IN THE
CLOUD
WHAT’S NEW ABOUT BIG DATA YOU
SAY…
 Traditionally, analytics on data internal to an organization
 Customer data
 ERP data
 Some pre-digested external data like market research
 Sophisticated analytics using new data sources
 Social data
 Website data
Low density, fine grained and massive
“Most EDWs are < 2TB”
LOW DENSITY, HIGH VOLUME
DATA
Amul comment data: 18000 * 140 * 60 * 24 * 30 = 100 GB per month
Category Unique visitors
Retail –
Luxury goods
20 million
Retail –
consumer
goods
30 million
Retail – tickets 26 million
Social Media Website data
Traditional technologies cannot handle this low-density, high volume data
SKELETON OF A BIG DATA
PROJECT
Internal
Data
External
Data
TB - PBs
Actionable report
Analytics Workflow
HOW DO THE BIG GUYS DO IT?
 Build data centers
 Buy or build custom big-data software
 Hire ETL engineers who manage bringing data into the system
 Hire admins to keep it all running
 Hire data scientists to come up with interesting questions
 Hire developers who can translate questions into programs
Lots of upfront
investment
Long time to get started
Lots of risks
BIG DATA PROJECT ENTAILS
LANDSCAPE IS CHANGING
 Advent of public clouds
 Cheap, reliable storage
 Provision 10-1000s of machines in a couple of minutes
 Pay as you go, grow as you please
 Free / inexpensive big-data software
 Hadoop, Hive, Presto
CLOUD PRIMITIVES
 Persistent object store e.g. AWS S3
 Reliability is basically solved for you (*)
 Ability to provision clusters with pre-built images in a couple of
minutes
 Pay by the hour (or by the minute)
 Spot instances (AWS)
 Relational DB as a Service
 MySQL, PostgreSQL etc
THE CLOUD CAN HANDLE YOUR DATA
CLOUD’S COMPUTE FLEXIBILITY
 Analytics workloads tend to be bursty
 Most orgs struggle to predict usage 2-3 months down the line
 Tend to overprovision compute
 Result: < 30% utilization of their hardware
 Cloud allows you to scale up and down
 Trickier for a big data system, but possible
Chen et al,
VLDB 2012
Provision for peak workload
BIG DATA SOFTWARE
 Many open source projects
 Hadoop based on Google’s MR paper (Yahoo)
 Hive (SQL-on-Hadoop)
 Presto (Fast SQL)
 Production ready, running at scale at Yahoo, FB and many other
environments
ENTER HADOOP
 Open-source implementation of Map-reduce used by Google to
index trillions of web pages
 Allows programmers to write distributed programs using map and
reduce abstractions
 Ability to run these programs on large amounts of data
 Uses bunch of cheap hardware, can tolerate failures
HADOOP SCALES!
HIVE: SQL ON HADOOP
 Facebook had a Multi Petabyte Warehouse
 Had 80+ engineers writing Hadoop jobs
 Files are insufficient data abstractions
 Need tables, schemas, partitions, indices
 SQL is highly popular
 So, implement SQL on top of Hadoop
 Allowed non-programmers to process all the data
 FB open-sourced it
 Production ready
 Processes 25PB of data in FB
 Processes 20PB of data at Qubole
HIVE ALLOWS YOU TO DESCRIBE DATA
 Example
 My data lives in Amazon S3 in a specific location
 It is in delimited text format
 Please create a virtual table for me
 Number of data formats: JSON, Text, Binary, Avro, ProtoBuf, Thrift
 Analytics is often a downstream process
 Conversion of data is time consuming and not productive
create external table nation (N_NATIONKEY INT, N_NAME STRING,
N_REGIONKEY INT, N_COMMENT STRING)
ROW FORMAT DELIMITED
STORED AS TEXTFILE
LOCATION 's3n://public-qubole/datasets/tpch5G/nation';
HIVE EXTENSIBILITY
 Connect to external data sources like MongoDB
 Write code to understand new data formats - serdes
 Custom UDFs in Java
 Plug in custom code in python or any other language
SELECT
TRANSFORM (hosting_ids, user_id, d)
USING 'python combine_arrays.py' AS (hosting_ranks_array, user_id, d)
FROM s_table;
HIVE ALLOWS YOU TO QUERY THE
DATA
 SQL-Like
 Query is parallelized using Hadoop as execution engine
Select count(*) from nation;
Count(*)
Count(*)
Sum()
HIVE EXECUTION
 Split Hive query into multiple Hadoop/MR jobs
 Run Job 1, save intermediate output to HDFS
 Run Job 2..
 Return results
 Data parallel because every hadoop job runs on number of
machines
T11
100MB
T12
100MB
10 files
5 files
5 files
TASK PARALLELISM
T1 T2 T3
100MB 100MB 100MB10 files
EXECUTION MODEL 1
T1
100MB
T2
100MB
T3
100MB10 files
• Only 100MB of memory required
• Can stop and resume
• Allows for multiplexing multiple pipelines
• Can tolerate failures
• Spilling can be expensive
• Time to first result is high
EXECUTION MODEL 2
T1
100MB
T2
100MB
T3
100MB10 files
• Task parallelism
• Needs 3X memory
• No spilling, hence much faster
• Early first results
• Stop and resume is trickier
• Multiplexing is more difficult
• Cannot tolerate failures
ENTER PRESTO
 Hive was EM1 and had associated disadvantages
 Internal project at Facebook to implement EM2 (Presto)
 Use case was interactive queries over the same data
 Open sourced late 2013
 Promised much faster query performance
 In-memory processing, aggressive pipelining
 Supports all the data formats that Hive does
 Can’t plug in user code at this point, vanilla SQL
CONTRASTING HIVE AND PRESTO
Hive Presto
Uses Hadoop MR for
execution (EM1)
Pipelined execution model
(EM2)
Spills intermediate data to
FS
Intermediate data in
memory
Can tolerate failures Does not tolerate failures
Automatic join ordering User-specified join ordering
Can handle joins of two
large tables
One table needs to fit in
memory
Supports grouping sets Does not support GS
Plug in custom code Cannot plug in custom code
More data types Limited data types
Hive 0.11 vs Presto 0.60
PERFORMANCE COMPARISON
• Presto is 2.5-7x faster
• But, some queries just run out of
memory
• Contrasts the execution models
IN A NUTSHELL
SAMPLE SETUP
Cloud Storage
Sqoop
Application
Sync
Heavy duty queries Interactive queries
CRYSTAL BALL
 Hive is actively working on task parallelism as part of the Stinger
Initiative
 Presto is also making rapid progress in bridging some of its gaps
 There are other open source projects:
 Impala, Shark, Drill, Tajo
 Lots of goodies for users
CONCLUSION
 Big Data Analytics is becoming accessible and affordable
 Public clouds give flexibility and change economics
 Hive and Presto provide intuitive and powerful ways to interact
with your data

More Related Content

What's hot

Data Analytic Technology Platforms: Options and Tradeoffs
Data Analytic Technology Platforms: Options and TradeoffsData Analytic Technology Platforms: Options and Tradeoffs
Data Analytic Technology Platforms: Options and Tradeoffs
J Singh
 
Succinct Spark: Fast Interactive Queries on Compressed RDDs by Rachit Agarwal
Succinct Spark: Fast Interactive Queries on Compressed RDDs by Rachit AgarwalSuccinct Spark: Fast Interactive Queries on Compressed RDDs by Rachit Agarwal
Succinct Spark: Fast Interactive Queries on Compressed RDDs by Rachit Agarwal
Spark Summit
 

What's hot (20)

Cascalog
CascalogCascalog
Cascalog
 
Data Analytic Technology Platforms: Options and Tradeoffs
Data Analytic Technology Platforms: Options and TradeoffsData Analytic Technology Platforms: Options and Tradeoffs
Data Analytic Technology Platforms: Options and Tradeoffs
 
Data Engineering Basics
Data Engineering BasicsData Engineering Basics
Data Engineering Basics
 
Data Science at Scale by Sarah Guido
Data Science at Scale by Sarah GuidoData Science at Scale by Sarah Guido
Data Science at Scale by Sarah Guido
 
Building Better Analytics Workflows (Strata-Hadoop World 2013)
Building Better Analytics Workflows (Strata-Hadoop World 2013)Building Better Analytics Workflows (Strata-Hadoop World 2013)
Building Better Analytics Workflows (Strata-Hadoop World 2013)
 
Big data vahidamiri-tabriz-13960226-datastack.ir
Big data vahidamiri-tabriz-13960226-datastack.irBig data vahidamiri-tabriz-13960226-datastack.ir
Big data vahidamiri-tabriz-13960226-datastack.ir
 
Lighthouse - an open-source library to build data lakes - Kris Peeters
Lighthouse - an open-source library to build data lakes - Kris PeetersLighthouse - an open-source library to build data lakes - Kris Peeters
Lighthouse - an open-source library to build data lakes - Kris Peeters
 
From zero to hero with the actor model - Tamir Dresher - Odessa 2019
From zero to hero with the actor model  - Tamir Dresher - Odessa 2019From zero to hero with the actor model  - Tamir Dresher - Odessa 2019
From zero to hero with the actor model - Tamir Dresher - Odessa 2019
 
Building Scalable Big Data Pipelines
Building Scalable Big Data PipelinesBuilding Scalable Big Data Pipelines
Building Scalable Big Data Pipelines
 
H2O Rains with Databricks Cloud - NY 02.16.16
H2O Rains with Databricks Cloud - NY 02.16.16H2O Rains with Databricks Cloud - NY 02.16.16
H2O Rains with Databricks Cloud - NY 02.16.16
 
Fully Automated QA System For Large Scale Search And Recommendation Engines U...
Fully Automated QA System For Large Scale Search And Recommendation Engines U...Fully Automated QA System For Large Scale Search And Recommendation Engines U...
Fully Automated QA System For Large Scale Search And Recommendation Engines U...
 
Spark Summit EU talk by Pat Patterson
Spark Summit EU talk by Pat PattersonSpark Summit EU talk by Pat Patterson
Spark Summit EU talk by Pat Patterson
 
Watching Pigs Fly with the Netflix Hadoop Toolkit (Hadoop Summit 2013)
Watching Pigs Fly with the Netflix Hadoop Toolkit (Hadoop Summit 2013)Watching Pigs Fly with the Netflix Hadoop Toolkit (Hadoop Summit 2013)
Watching Pigs Fly with the Netflix Hadoop Toolkit (Hadoop Summit 2013)
 
Building a data pipeline to ingest data into Hadoop in minutes using Streamse...
Building a data pipeline to ingest data into Hadoop in minutes using Streamse...Building a data pipeline to ingest data into Hadoop in minutes using Streamse...
Building a data pipeline to ingest data into Hadoop in minutes using Streamse...
 
Succinct Spark: Fast Interactive Queries on Compressed RDDs by Rachit Agarwal
Succinct Spark: Fast Interactive Queries on Compressed RDDs by Rachit AgarwalSuccinct Spark: Fast Interactive Queries on Compressed RDDs by Rachit Agarwal
Succinct Spark: Fast Interactive Queries on Compressed RDDs by Rachit Agarwal
 
Big Data Day LA 2015 - Machine Learning on Largish Data by Szilard Pafka of E...
Big Data Day LA 2015 - Machine Learning on Largish Data by Szilard Pafka of E...Big Data Day LA 2015 - Machine Learning on Largish Data by Szilard Pafka of E...
Big Data Day LA 2015 - Machine Learning on Largish Data by Szilard Pafka of E...
 
Apache Hadoop
Apache HadoopApache Hadoop
Apache Hadoop
 
Bi on Big Data - Strata 2016 in London
Bi on Big Data - Strata 2016 in LondonBi on Big Data - Strata 2016 in London
Bi on Big Data - Strata 2016 in London
 
Another Intro To Hadoop
Another Intro To HadoopAnother Intro To Hadoop
Another Intro To Hadoop
 
Agile data warehousing
Agile data warehousingAgile data warehousing
Agile data warehousing
 

Similar to Big dataproposal

Hive @ Hadoop day seattle_2010
Hive @ Hadoop day seattle_2010Hive @ Hadoop day seattle_2010
Hive @ Hadoop day seattle_2010
nzhang
 
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Cloudera, Inc.
 
Hw09 Matchmaking In The Cloud
Hw09   Matchmaking In The CloudHw09   Matchmaking In The Cloud
Hw09 Matchmaking In The Cloud
Cloudera, Inc.
 

Similar to Big dataproposal (20)

Big Data and Hadoop
Big Data and HadoopBig Data and Hadoop
Big Data and Hadoop
 
Big dataanalyticsinthecloud
Big dataanalyticsinthecloudBig dataanalyticsinthecloud
Big dataanalyticsinthecloud
 
Big data and hadoop
Big data and hadoopBig data and hadoop
Big data and hadoop
 
Hive @ Hadoop day seattle_2010
Hive @ Hadoop day seattle_2010Hive @ Hadoop day seattle_2010
Hive @ Hadoop day seattle_2010
 
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
 
Hadoop Frameworks Panel__HadoopSummit2010
Hadoop Frameworks Panel__HadoopSummit2010Hadoop Frameworks Panel__HadoopSummit2010
Hadoop Frameworks Panel__HadoopSummit2010
 
عصر کلان داده، چرا و چگونه؟
عصر کلان داده، چرا و چگونه؟عصر کلان داده، چرا و چگونه؟
عصر کلان داده، چرا و چگونه؟
 
Big Data , Big Problem?
Big Data , Big Problem?Big Data , Big Problem?
Big Data , Big Problem?
 
Final deck
Final deckFinal deck
Final deck
 
Big Data with hadoop, Spark and BigQuery (Google cloud next Extended 2017 Kar...
Big Data with hadoop, Spark and BigQuery (Google cloud next Extended 2017 Kar...Big Data with hadoop, Spark and BigQuery (Google cloud next Extended 2017 Kar...
Big Data with hadoop, Spark and BigQuery (Google cloud next Extended 2017 Kar...
 
Building a Big Data platform with the Hadoop ecosystem
Building a Big Data platform with the Hadoop ecosystemBuilding a Big Data platform with the Hadoop ecosystem
Building a Big Data platform with the Hadoop ecosystem
 
Hadoop World 2011: Hadoop’s Life in Enterprise Systems - Y Masatani, NTTData
Hadoop World 2011: Hadoop’s Life in Enterprise Systems - Y Masatani, NTTDataHadoop World 2011: Hadoop’s Life in Enterprise Systems - Y Masatani, NTTData
Hadoop World 2011: Hadoop’s Life in Enterprise Systems - Y Masatani, NTTData
 
Big Data: An Overview
Big Data: An OverviewBig Data: An Overview
Big Data: An Overview
 
Prague data management meetup 2018-03-27
Prague data management meetup 2018-03-27Prague data management meetup 2018-03-27
Prague data management meetup 2018-03-27
 
Big data analytics: Technology's bleeding edge
Big data analytics: Technology's bleeding edgeBig data analytics: Technology's bleeding edge
Big data analytics: Technology's bleeding edge
 
Hw09 Matchmaking In The Cloud
Hw09   Matchmaking In The CloudHw09   Matchmaking In The Cloud
Hw09 Matchmaking In The Cloud
 
Hadoop Maharajathi,II-M.sc.,Computer Science,Bonsecours college for women
Hadoop Maharajathi,II-M.sc.,Computer Science,Bonsecours college for womenHadoop Maharajathi,II-M.sc.,Computer Science,Bonsecours college for women
Hadoop Maharajathi,II-M.sc.,Computer Science,Bonsecours college for women
 
How Hadoop Revolutionized Data Warehousing at Yahoo and Facebook
How Hadoop Revolutionized Data Warehousing at Yahoo and FacebookHow Hadoop Revolutionized Data Warehousing at Yahoo and Facebook
How Hadoop Revolutionized Data Warehousing at Yahoo and Facebook
 
Big data architectures and the data lake
Big data architectures and the data lakeBig data architectures and the data lake
Big data architectures and the data lake
 
Big data
Big dataBig data
Big data
 

More from Qubole

Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...
Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...
Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...
Qubole
 
Getting to 1.5M Ads/sec: How DataXu manages Big Data
Getting to 1.5M Ads/sec: How DataXu manages Big DataGetting to 1.5M Ads/sec: How DataXu manages Big Data
Getting to 1.5M Ads/sec: How DataXu manages Big Data
Qubole
 
Presto in the cloud
Presto in the cloudPresto in the cloud
Presto in the cloud
Qubole
 

More from Qubole (20)

Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...
Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...
Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...
 
7 Big Data Challenges and How to Overcome Them
7 Big Data Challenges and How to Overcome Them7 Big Data Challenges and How to Overcome Them
7 Big Data Challenges and How to Overcome Them
 
State of Big Data Adoption
State of Big Data AdoptionState of Big Data Adoption
State of Big Data Adoption
 
Big Data at Pinterest - Presented by Qubole
Big Data at Pinterest - Presented by QuboleBig Data at Pinterest - Presented by Qubole
Big Data at Pinterest - Presented by Qubole
 
5 Factors Impacting Your Big Data Project's Performance
5 Factors Impacting Your Big Data Project's Performance 5 Factors Impacting Your Big Data Project's Performance
5 Factors Impacting Your Big Data Project's Performance
 
Spark on Yarn
Spark on YarnSpark on Yarn
Spark on Yarn
 
Atlanta MLConf
Atlanta MLConfAtlanta MLConf
Atlanta MLConf
 
Running Spark on Cloud
Running Spark on CloudRunning Spark on Cloud
Running Spark on Cloud
 
Qubole State of the Big Data Industry
Qubole State of the Big Data IndustryQubole State of the Big Data Industry
Qubole State of the Big Data Industry
 
Big Data Platform at Pinterest
Big Data Platform at PinterestBig Data Platform at Pinterest
Big Data Platform at Pinterest
 
Atlanta Data Science Meetup | Qubole slides
Atlanta Data Science Meetup | Qubole slidesAtlanta Data Science Meetup | Qubole slides
Atlanta Data Science Meetup | Qubole slides
 
Qubole presentation for the Cleveland Big Data and Hadoop Meetup
Qubole presentation for the Cleveland Big Data and Hadoop Meetup   Qubole presentation for the Cleveland Big Data and Hadoop Meetup
Qubole presentation for the Cleveland Big Data and Hadoop Meetup
 
BIPD Tech Tuesday Presentation - Qubole
BIPD Tech Tuesday Presentation - QuboleBIPD Tech Tuesday Presentation - Qubole
BIPD Tech Tuesday Presentation - Qubole
 
Harnessing the Hadoop Ecosystem Optimizations in Apache Hive
Harnessing the Hadoop Ecosystem Optimizations in Apache HiveHarnessing the Hadoop Ecosystem Optimizations in Apache Hive
Harnessing the Hadoop Ecosystem Optimizations in Apache Hive
 
Optimizing Big Data to run in the Public Cloud
Optimizing Big Data to run in the Public CloudOptimizing Big Data to run in the Public Cloud
Optimizing Big Data to run in the Public Cloud
 
Getting to 1.5M Ads/sec: How DataXu manages Big Data
Getting to 1.5M Ads/sec: How DataXu manages Big DataGetting to 1.5M Ads/sec: How DataXu manages Big Data
Getting to 1.5M Ads/sec: How DataXu manages Big Data
 
Expert Big Data Tips
Expert Big Data TipsExpert Big Data Tips
Expert Big Data Tips
 
Presto in the cloud
Presto in the cloudPresto in the cloud
Presto in the cloud
 
Basic Sentiment Analysis using Hive
Basic Sentiment Analysis using HiveBasic Sentiment Analysis using Hive
Basic Sentiment Analysis using Hive
 
Effective Hive Queries
Effective Hive QueriesEffective Hive Queries
Effective Hive Queries
 

Recently uploaded

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 

Recently uploaded (20)

Modernizing Legacy Systems Using Ballerina
Modernizing Legacy Systems Using BallerinaModernizing Legacy Systems Using Ballerina
Modernizing Legacy Systems Using Ballerina
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptx
 
Choreo: Empowering the Future of Enterprise Software Engineering
Choreo: Empowering the Future of Enterprise Software EngineeringChoreo: Empowering the Future of Enterprise Software Engineering
Choreo: Empowering the Future of Enterprise Software Engineering
 
Quantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation ComputingQuantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation Computing
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
 
Decarbonising Commercial Real Estate: The Role of Operational Performance
Decarbonising Commercial Real Estate: The Role of Operational PerformanceDecarbonising Commercial Real Estate: The Role of Operational Performance
Decarbonising Commercial Real Estate: The Role of Operational Performance
 
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
Navigating Identity and Access Management in the Modern Enterprise
Navigating Identity and Access Management in the Modern EnterpriseNavigating Identity and Access Management in the Modern Enterprise
Navigating Identity and Access Management in the Modern Enterprise
 

Big dataproposal

  • 1. HIVE AND PRESTO FOR BIG DATA ANALYTICS IN THE CLOUD
  • 2. WHAT’S NEW ABOUT BIG DATA YOU SAY…  Traditionally, analytics on data internal to an organization  Customer data  ERP data  Some pre-digested external data like market research  Sophisticated analytics using new data sources  Social data  Website data Low density, fine grained and massive “Most EDWs are < 2TB”
  • 3. LOW DENSITY, HIGH VOLUME DATA Amul comment data: 18000 * 140 * 60 * 24 * 30 = 100 GB per month Category Unique visitors Retail – Luxury goods 20 million Retail – consumer goods 30 million Retail – tickets 26 million Social Media Website data Traditional technologies cannot handle this low-density, high volume data
  • 4. SKELETON OF A BIG DATA PROJECT Internal Data External Data TB - PBs Actionable report Analytics Workflow
  • 5. HOW DO THE BIG GUYS DO IT?  Build data centers  Buy or build custom big-data software  Hire ETL engineers who manage bringing data into the system  Hire admins to keep it all running  Hire data scientists to come up with interesting questions  Hire developers who can translate questions into programs
  • 6. Lots of upfront investment Long time to get started Lots of risks BIG DATA PROJECT ENTAILS
  • 7. LANDSCAPE IS CHANGING  Advent of public clouds  Cheap, reliable storage  Provision 10-1000s of machines in a couple of minutes  Pay as you go, grow as you please  Free / inexpensive big-data software  Hadoop, Hive, Presto
  • 8. CLOUD PRIMITIVES  Persistent object store e.g. AWS S3  Reliability is basically solved for you (*)  Ability to provision clusters with pre-built images in a couple of minutes  Pay by the hour (or by the minute)  Spot instances (AWS)  Relational DB as a Service  MySQL, PostgreSQL etc
  • 9. THE CLOUD CAN HANDLE YOUR DATA
  • 10. CLOUD’S COMPUTE FLEXIBILITY  Analytics workloads tend to be bursty  Most orgs struggle to predict usage 2-3 months down the line  Tend to overprovision compute  Result: < 30% utilization of their hardware  Cloud allows you to scale up and down  Trickier for a big data system, but possible Chen et al, VLDB 2012 Provision for peak workload
  • 11. BIG DATA SOFTWARE  Many open source projects  Hadoop based on Google’s MR paper (Yahoo)  Hive (SQL-on-Hadoop)  Presto (Fast SQL)  Production ready, running at scale at Yahoo, FB and many other environments
  • 12. ENTER HADOOP  Open-source implementation of Map-reduce used by Google to index trillions of web pages  Allows programmers to write distributed programs using map and reduce abstractions  Ability to run these programs on large amounts of data  Uses bunch of cheap hardware, can tolerate failures
  • 14. HIVE: SQL ON HADOOP  Facebook had a Multi Petabyte Warehouse  Had 80+ engineers writing Hadoop jobs  Files are insufficient data abstractions  Need tables, schemas, partitions, indices  SQL is highly popular  So, implement SQL on top of Hadoop  Allowed non-programmers to process all the data  FB open-sourced it  Production ready  Processes 25PB of data in FB  Processes 20PB of data at Qubole
  • 15. HIVE ALLOWS YOU TO DESCRIBE DATA  Example  My data lives in Amazon S3 in a specific location  It is in delimited text format  Please create a virtual table for me  Number of data formats: JSON, Text, Binary, Avro, ProtoBuf, Thrift  Analytics is often a downstream process  Conversion of data is time consuming and not productive create external table nation (N_NATIONKEY INT, N_NAME STRING, N_REGIONKEY INT, N_COMMENT STRING) ROW FORMAT DELIMITED STORED AS TEXTFILE LOCATION 's3n://public-qubole/datasets/tpch5G/nation';
  • 16. HIVE EXTENSIBILITY  Connect to external data sources like MongoDB  Write code to understand new data formats - serdes  Custom UDFs in Java  Plug in custom code in python or any other language SELECT TRANSFORM (hosting_ids, user_id, d) USING 'python combine_arrays.py' AS (hosting_ranks_array, user_id, d) FROM s_table;
  • 17. HIVE ALLOWS YOU TO QUERY THE DATA  SQL-Like  Query is parallelized using Hadoop as execution engine Select count(*) from nation; Count(*) Count(*) Sum()
  • 18. HIVE EXECUTION  Split Hive query into multiple Hadoop/MR jobs  Run Job 1, save intermediate output to HDFS  Run Job 2..  Return results  Data parallel because every hadoop job runs on number of machines T11 100MB T12 100MB 10 files 5 files 5 files
  • 19. TASK PARALLELISM T1 T2 T3 100MB 100MB 100MB10 files
  • 20. EXECUTION MODEL 1 T1 100MB T2 100MB T3 100MB10 files • Only 100MB of memory required • Can stop and resume • Allows for multiplexing multiple pipelines • Can tolerate failures • Spilling can be expensive • Time to first result is high
  • 21. EXECUTION MODEL 2 T1 100MB T2 100MB T3 100MB10 files • Task parallelism • Needs 3X memory • No spilling, hence much faster • Early first results • Stop and resume is trickier • Multiplexing is more difficult • Cannot tolerate failures
  • 22. ENTER PRESTO  Hive was EM1 and had associated disadvantages  Internal project at Facebook to implement EM2 (Presto)  Use case was interactive queries over the same data  Open sourced late 2013  Promised much faster query performance  In-memory processing, aggressive pipelining  Supports all the data formats that Hive does  Can’t plug in user code at this point, vanilla SQL
  • 23. CONTRASTING HIVE AND PRESTO Hive Presto Uses Hadoop MR for execution (EM1) Pipelined execution model (EM2) Spills intermediate data to FS Intermediate data in memory Can tolerate failures Does not tolerate failures Automatic join ordering User-specified join ordering Can handle joins of two large tables One table needs to fit in memory Supports grouping sets Does not support GS Plug in custom code Cannot plug in custom code More data types Limited data types Hive 0.11 vs Presto 0.60
  • 24. PERFORMANCE COMPARISON • Presto is 2.5-7x faster • But, some queries just run out of memory • Contrasts the execution models
  • 27. CRYSTAL BALL  Hive is actively working on task parallelism as part of the Stinger Initiative  Presto is also making rapid progress in bridging some of its gaps  There are other open source projects:  Impala, Shark, Drill, Tajo  Lots of goodies for users
  • 28. CONCLUSION  Big Data Analytics is becoming accessible and affordable  Public clouds give flexibility and change economics  Hive and Presto provide intuitive and powerful ways to interact with your data