SlideShare a Scribd company logo
1 of 45
Download to read offline
1ยฉ Cloudera, Inc. All rights reserved.
One Hadoop, Multiple Clouds
Andrei Savu | Tech Lead, Cloudera Director
2ยฉ Cloudera, Inc. All rights reserved.
About me
Tech Lead on Cloudera Director
Previously founder of axemblr.com
Contributed to Apache Whirr (PMC) & jclouds.
Twitter: https://twitter.com/andreisavu
LinkedIn: https://www.linkedin.com/in/sandrei
3ยฉ Cloudera, Inc. All rights reserved.
Cloudera Director
cloudera.com/director
Deploy and manage
enterprise-grade
Hadoop in the cloud
AWS & Google Cloud
Extensible via plugins
Journey to the Cloud
5ยฉ Cloudera, Inc. All rights reserved.
Do you use a public or
private cloud?
How do you run and
manage Hadoop?
6ยฉ Cloudera, Inc. All rights reserved.
What is this talk
about?
State of the World
Architectural Patterns
Imagine the Future
7ยฉ Cloudera, Inc. All rights reserved.
Gartner's 2015 Hype
Cycle for Emerging
Technologies (source)
Advanced Analytics
Hybrid Cloud
Internet of Things
8ยฉ Cloudera, Inc. All rights reserved.
Hybrid Clouds
Cloud Exchange
Application Portability
Private-Public
Public-Public
9ยฉ Cloudera, Inc. All rights reserved.
Cloud Wars
AWS
Microsoft Azure
Google Cloud
VMWare
Openstack
etc.
10ยฉ Cloudera, Inc. All rights reserved.
Data has Mass and
Gravity
11ยฉ Cloudera, Inc. All rights reserved.
Hadoop Environments
On-Premise versus Cloud
On-Premise Cloud
Storage Direct Attached Direct Attached or Object Store
Data Not shared across clusters Shared across multiple clusters
Sizing Fixed-size Dynamic based on load
Usage Model All users share cluster Clusters created as needed for apps/users
Resource Management (YARN)
HDFS
Process Discover Model Serve
Industry Standard Servers
(CPU, Memory, & Direct Attached Storage)
Resource Management (YARN)
HDFS
Process Discover Model Serve
Industry Standard Servers
(CPU & Memory)
Object
Storage
12ยฉ Cloudera, Inc. All rights reserved.
Cloud providers
shipping distributions
of Hadoop
Integration
Unlock Query Engines
Migration workloads
Is that a sustainable
advantage? Or just a
temporary stop gap?
13ยฉ Cloudera, Inc. All rights reserved.
Maturity level
On-prem vs. Cloud
Monitoring
Dev / Test / Prod
Availability
Durability
14ยฉ Cloudera, Inc. All rights reserved.
Common Architectural Patterns in the Cloud
Object Storage
Source Data Seed Data Backup/DR
ETL/MODELING
(Spark, MapReduce)
โ€ข Short-running clusters
โ€ข Elastic workload
โ€ข No local storage
necessary
|WASB |SWIFT |BLOB
โ€ข Long-running clusters
โ€ข Sized to demand
โ€ข Some local storage
BI/ANALYTICS
(Impala, Solr)
โ€ข Fixed clusters
โ€ข Periodic sync
โ€ข Default to local
storage
APP DELIVERY
(HBase, Kudu)
15ยฉ Cloudera, Inc. All rights reserved.
Cluster lifecycle
management
Create / Terminate
Discovery
Metadata
Monitoring
16ยฉ Cloudera, Inc. All rights reserved.
Work Queue
Workflows
Dispatch
Tracking
Decoupled
Fault Tolerant
17ยฉ Cloudera, Inc. All rights reserved.
Common Architectural Patterns in the Cloud
Object Storage
Source Data Seed Data Backup/DR
ETL/MODELING
(Spark, MapReduce)
โ€ข Short-running clusters
โ€ข Elastic workload
โ€ข No local storage
necessary
|WASB |SWIFT |BLOB
โ€ข Long-running clusters
โ€ข Sized to demand
โ€ข Some local storage
BI/ANALYTICS
(Impala, Solr)
โ€ข Fixed clusters
โ€ข Periodic sync
โ€ข Default to local
storage
APP DELIVERY
(HBase, Kudu)
18ยฉ Cloudera, Inc. All rights reserved.
Multi-user
Secure
Isolated
Friendly
19ยฉ Cloudera, Inc. All rights reserved.
Elastic
Grow or shrink
Business hours
Number of users
Storage vs. Compute
Cost efficient
20ยฉ Cloudera, Inc. All rights reserved.
Common Architectural Patterns in the Cloud
Object Storage
Source Data Seed Data Backup/DR
ETL/MODELING
(Spark, MapReduce)
โ€ข Short-running clusters
โ€ข Elastic workload
โ€ข No local storage
necessary
|WASB |SWIFT |BLOB
โ€ข Long-running clusters
โ€ข Sized to demand
โ€ข Some local storage
BI/ANALYTICS
(Impala, Solr)
โ€ข Fixed clusters
โ€ข Periodic sync
โ€ข Default to local
storage
APP DELIVERY
(HBase, Kudu)
21ยฉ Cloudera, Inc. All rights reserved.
Advanced Monitoring
Latency
Resource utilization
Consistent performance
22ยฉ Cloudera, Inc. All rights reserved.
High availability and
failure domains
Data durability
Repair within SLA
Host-to-instance
23ยฉ Cloudera, Inc. All rights reserved.
Backup and disaster
recovery
Object store centric
Active-Standby
24ยฉ Cloudera, Inc. All rights reserved.
Imagine the Future
Portable Experience
Self-service
Self-healing
Granular Security
Advanced Governance
Complete Management
Whatโ€™s your vision?
26ยฉ Cloudera, Inc. All rights reserved.
Thank you!
asavu@cloudera.com
27ยฉ Cloudera, Inc. All rights reserved.
Resources
Cloudera Director: http://www.cloudera.com/director
Interested in API level integration and scripting?
https://github.com/cloudera/director-sdk
https://github.com/cloudera/director-scripts
Interested in integration with another cloud platform?
https://github.com/cloudera/director-spi
https://github.com/cloudera/director-google-plugin
28ยฉ Cloudera, Inc. All rights reserved.
Whatโ€™s new in Cloudera Director 1.5?
http://blog.cloudera.com/blog/2015/08/whats-new-in-
cloudera-director-1-5/
Get Started
AWS Reference Guide
GCP Reference Guide
Try It Out
AWS Quickstart
Resources
Cloudera Director
Screenshots
ยฉ 2014 Cloudera, Inc. All rights reserved.
ยฉ 2014 Cloudera, Inc. All rights reserved.
ยฉ 2014 Cloudera, Inc. All rights reserved.
ยฉ 2014 Cloudera, Inc. All rights reserved.
ยฉ 2014 Cloudera, Inc. All rights reserved.
ยฉ 2014 Cloudera, Inc. All rights reserved.
ยฉ 2014 Cloudera, Inc. All rights reserved.
ยฉ 2014 Cloudera, Inc. All rights reserved.
ยฉ 2014 Cloudera, Inc. All rights reserved.
ยฉ 2014 Cloudera, Inc. All rights reserved.
ยฉ 2014 Cloudera, Inc. All rights reserved.
ยฉ 2014 Cloudera, Inc. All rights reserved.
ยฉ 2014 Cloudera, Inc. All rights reserved.
ยฉ 2014 Cloudera, Inc. All rights reserved.
ยฉ 2014 Cloudera, Inc. All rights reserved.
45ยฉ Cloudera, Inc. All rights reserved.
Thank you!
asavu@cloudera.com

More Related Content

What's hot

What's hot (20)

Azure realtime-interview questions - part 7
Azure realtime-interview questions - part 7Azure realtime-interview questions - part 7
Azure realtime-interview questions - part 7
ย 
When networks meets apps (open stack atlanta)
When networks meets apps (open stack atlanta)When networks meets apps (open stack atlanta)
When networks meets apps (open stack atlanta)
ย 
Google developer group 2021 - Introduction to cloud computing
Google developer group 2021 - Introduction to cloud computingGoogle developer group 2021 - Introduction to cloud computing
Google developer group 2021 - Introduction to cloud computing
ย 
Keeping Developers and Auditors Happy in the Cloud
Keeping Developers and Auditors Happy in the Cloud Keeping Developers and Auditors Happy in the Cloud
Keeping Developers and Auditors Happy in the Cloud
ย 
Must Know Azure Kubernetes Best Practices And Features For Better Resiliency ...
Must Know Azure Kubernetes Best Practices And Features For Better Resiliency ...Must Know Azure Kubernetes Best Practices And Features For Better Resiliency ...
Must Know Azure Kubernetes Best Practices And Features For Better Resiliency ...
ย 
Amazon Virtual Private Cloud - VPC 1
Amazon Virtual Private Cloud - VPC 1Amazon Virtual Private Cloud - VPC 1
Amazon Virtual Private Cloud - VPC 1
ย 
Session 2 - Exploring Cloud Computing with Amazon Web Services (AWS)
Session 2 - Exploring Cloud Computing with Amazon Web Services (AWS)Session 2 - Exploring Cloud Computing with Amazon Web Services (AWS)
Session 2 - Exploring Cloud Computing with Amazon Web Services (AWS)
ย 
Serverless computing
Serverless computingServerless computing
Serverless computing
ย 
AWS Study Group - Chapter 04 - Hybrid Cloud Architectures [Solution Architect...
AWS Study Group - Chapter 04 - Hybrid Cloud Architectures [Solution Architect...AWS Study Group - Chapter 04 - Hybrid Cloud Architectures [Solution Architect...
AWS Study Group - Chapter 04 - Hybrid Cloud Architectures [Solution Architect...
ย 
PaaS: An Introduction
PaaS: An IntroductionPaaS: An Introduction
PaaS: An Introduction
ย 
Harness the Power of Hybrid Cloud with AWS and Avere
Harness the Power of Hybrid Cloud with AWS and AvereHarness the Power of Hybrid Cloud with AWS and Avere
Harness the Power of Hybrid Cloud with AWS and Avere
ย 
Managing WordPress on Amazon Lightsail - July 2017 AWS Online Tech Talks
 Managing WordPress on Amazon Lightsail - July 2017 AWS Online Tech Talks Managing WordPress on Amazon Lightsail - July 2017 AWS Online Tech Talks
Managing WordPress on Amazon Lightsail - July 2017 AWS Online Tech Talks
ย 
AWS & Cloud competition from Azure, openstack
AWS & Cloud competition from Azure, openstack AWS & Cloud competition from Azure, openstack
AWS & Cloud competition from Azure, openstack
ย 
Building Hybrid Cloud Apps with Azure and Azure stack
Building Hybrid Cloud Apps with Azure and Azure stackBuilding Hybrid Cloud Apps with Azure and Azure stack
Building Hybrid Cloud Apps with Azure and Azure stack
ย 
Amazon relational database service (rds)
Amazon relational database service (rds)Amazon relational database service (rds)
Amazon relational database service (rds)
ย 
How to Sell Serverless to Your Colleagues
How to Sell Serverless to Your ColleaguesHow to Sell Serverless to Your Colleagues
How to Sell Serverless to Your Colleagues
ย 
Rein in Your Cloud Costs with Terraform and AWS Lambda
Rein in Your Cloud Costs with Terraform and AWS LambdaRein in Your Cloud Costs with Terraform and AWS Lambda
Rein in Your Cloud Costs with Terraform and AWS Lambda
ย 
Serverless Comparison: AWS vs Azure vs Google vs IBM
Serverless Comparison: AWS vs Azure vs Google vs IBMServerless Comparison: AWS vs Azure vs Google vs IBM
Serverless Comparison: AWS vs Azure vs Google vs IBM
ย 
Complex Analytics with NoSQL Data Store in Real Time
Complex Analytics with NoSQL Data Store in Real TimeComplex Analytics with NoSQL Data Store in Real Time
Complex Analytics with NoSQL Data Store in Real Time
ย 
Building a Hybrid Cloud with AWS and VMware vSphere
Building a Hybrid Cloud with AWS and VMware vSphereBuilding a Hybrid Cloud with AWS and VMware vSphere
Building a Hybrid Cloud with AWS and VMware vSphere
ย 

Viewers also liked

AnalyzingMovieData and Business Intelligence
AnalyzingMovieData and Business IntelligenceAnalyzingMovieData and Business Intelligence
AnalyzingMovieData and Business Intelligence
JUNWEI GUAN
ย 
Unit testing Agile OpenSpace
Unit testing Agile OpenSpaceUnit testing Agile OpenSpace
Unit testing Agile OpenSpace
Andrei Savu
ย 
Apache Accumulo and Cloudera
Apache Accumulo and ClouderaApache Accumulo and Cloudera
Apache Accumulo and Cloudera
Joey Echeverria
ย 
YARN High Availability
YARN High AvailabilityYARN High Availability
YARN High Availability
DataWorks Summit
ย 
HIVE: Data Warehousing & Analytics on Hadoop
HIVE: Data Warehousing & Analytics on HadoopHIVE: Data Warehousing & Analytics on Hadoop
HIVE: Data Warehousing & Analytics on Hadoop
Zheng Shao
ย 

Viewers also liked (19)

AnalyzingMovieData and Business Intelligence
AnalyzingMovieData and Business IntelligenceAnalyzingMovieData and Business Intelligence
AnalyzingMovieData and Business Intelligence
ย 
Single node hadoop cluster installation
Single node hadoop cluster installation Single node hadoop cluster installation
Single node hadoop cluster installation
ย 
Unit testing Agile OpenSpace
Unit testing Agile OpenSpaceUnit testing Agile OpenSpace
Unit testing Agile OpenSpace
ย 
Apache Accumulo and Cloudera
Apache Accumulo and ClouderaApache Accumulo and Cloudera
Apache Accumulo and Cloudera
ย 
CDH5ๆœ€ๆ–ฐๆƒ…ๅ ฑ #cwt2013
CDH5ๆœ€ๆ–ฐๆƒ…ๅ ฑ #cwt2013CDH5ๆœ€ๆ–ฐๆƒ…ๅ ฑ #cwt2013
CDH5ๆœ€ๆ–ฐๆƒ…ๅ ฑ #cwt2013
ย 
Recommendation Engine using Apache Mahout
Recommendation Engine using Apache MahoutRecommendation Engine using Apache Mahout
Recommendation Engine using Apache Mahout
ย 
Cloudera hadoop installation
Cloudera hadoop installationCloudera hadoop installation
Cloudera hadoop installation
ย 
YARN High Availability
YARN High AvailabilityYARN High Availability
YARN High Availability
ย 
Hadoop Operations for Production Systems (Strata NYC)
Hadoop Operations for Production Systems (Strata NYC)Hadoop Operations for Production Systems (Strata NYC)
Hadoop Operations for Production Systems (Strata NYC)
ย 
Extending and Automating Cloudera Manager via API
Extending and Automating Cloudera Manager via APIExtending and Automating Cloudera Manager via API
Extending and Automating Cloudera Manager via API
ย 
Cloudera Director: Unlock the Full Potential of Hadoop in the Cloud
Cloudera Director: Unlock the Full Potential of Hadoop in the CloudCloudera Director: Unlock the Full Potential of Hadoop in the Cloud
Cloudera Director: Unlock the Full Potential of Hadoop in the Cloud
ย 
Samsungโ€™s First 90-Days Building a Next-Generation Analytics Platform
Samsungโ€™s First 90-Days Building a Next-Generation Analytics PlatformSamsungโ€™s First 90-Days Building a Next-Generation Analytics Platform
Samsungโ€™s First 90-Days Building a Next-Generation Analytics Platform
ย 
Challenges for running Hadoop on AWS - AdvancedAWS Meetup
Challenges for running Hadoop on AWS - AdvancedAWS MeetupChallenges for running Hadoop on AWS - AdvancedAWS Meetup
Challenges for running Hadoop on AWS - AdvancedAWS Meetup
ย 
Cluster management and automation with cloudera manager
Cluster management and automation with cloudera managerCluster management and automation with cloudera manager
Cluster management and automation with cloudera manager
ย 
Cloudera Manager 5 (hadoop้‹็”จ) #cwt2013
Cloudera Manager 5 (hadoop้‹็”จ)  #cwt2013Cloudera Manager 5 (hadoop้‹็”จ)  #cwt2013
Cloudera Manager 5 (hadoop้‹็”จ) #cwt2013
ย 
Five Tips for Running Cloudera on AWS
Five Tips for Running Cloudera on AWSFive Tips for Running Cloudera on AWS
Five Tips for Running Cloudera on AWS
ย 
Multi-tenant, Multi-cluster and Multi-container Apache HBase Deployments
Multi-tenant, Multi-cluster and Multi-container Apache HBase DeploymentsMulti-tenant, Multi-cluster and Multi-container Apache HBase Deployments
Multi-tenant, Multi-cluster and Multi-container Apache HBase Deployments
ย 
HIVE: Data Warehousing & Analytics on Hadoop
HIVE: Data Warehousing & Analytics on HadoopHIVE: Data Warehousing & Analytics on Hadoop
HIVE: Data Warehousing & Analytics on Hadoop
ย 
Hive Quick Start Tutorial
Hive Quick Start TutorialHive Quick Start Tutorial
Hive Quick Start Tutorial
ย 

Similar to One Hadoop, Multiple Clouds - NYC Big Data Meetup

OOW-5185-Hybrid Cloud
OOW-5185-Hybrid CloudOOW-5185-Hybrid Cloud
OOW-5185-Hybrid Cloud
Ben Duan
ย 

Similar to One Hadoop, Multiple Clouds - NYC Big Data Meetup (20)

Hadoop on Cloud: Why and How?
Hadoop on Cloud: Why and How?Hadoop on Cloud: Why and How?
Hadoop on Cloud: Why and How?
ย 
Cloudera Analytics and Machine Learning Platform - Optimized for Cloud
Cloudera Analytics and Machine Learning Platform - Optimized for Cloud Cloudera Analytics and Machine Learning Platform - Optimized for Cloud
Cloudera Analytics and Machine Learning Platform - Optimized for Cloud
ย 
Cloud-Native Machine Learning: Emerging Trends and the Road Ahead
Cloud-Native Machine Learning: Emerging Trends and the Road AheadCloud-Native Machine Learning: Emerging Trends and the Road Ahead
Cloud-Native Machine Learning: Emerging Trends and the Road Ahead
ย 
Part 2: A Visual Dive into Machine Learning and Deep Learning โ€จ
Part 2: A Visual Dive into Machine Learning and Deep Learning โ€จPart 2: A Visual Dive into Machine Learning and Deep Learning โ€จ
Part 2: A Visual Dive into Machine Learning and Deep Learning โ€จ
ย 
Applications on Hadoop
Applications on HadoopApplications on Hadoop
Applications on Hadoop
ย 
Cloudera training: secure your Cloudera cluster
Cloudera training: secure your Cloudera clusterCloudera training: secure your Cloudera cluster
Cloudera training: secure your Cloudera cluster
ย 
Enterprise machine learning on k8s lessons learned and the road ahead
Enterprise machine learning on k8s   lessons learned and the road aheadEnterprise machine learning on k8s   lessons learned and the road ahead
Enterprise machine learning on k8s lessons learned and the road ahead
ย 
How Big Data Can Enable Analytics from the Cloud (Technical Workshop)
How Big Data Can Enable Analytics from the Cloud (Technical Workshop)How Big Data Can Enable Analytics from the Cloud (Technical Workshop)
How Big Data Can Enable Analytics from the Cloud (Technical Workshop)
ย 
Leveraging the cloud for analytics and machine learning 1.29.19
Leveraging the cloud for analytics and machine learning 1.29.19Leveraging the cloud for analytics and machine learning 1.29.19
Leveraging the cloud for analytics and machine learning 1.29.19
ย 
Big Data Fundamentals 6.6.18
Big Data Fundamentals 6.6.18Big Data Fundamentals 6.6.18
Big Data Fundamentals 6.6.18
ย 
Big Data Fundamentals
Big Data FundamentalsBig Data Fundamentals
Big Data Fundamentals
ย 
Cloudera GoDataFest Deploying Cloudera in the Cloud
Cloudera GoDataFest Deploying Cloudera in the CloudCloudera GoDataFest Deploying Cloudera in the Cloud
Cloudera GoDataFest Deploying Cloudera in the Cloud
ย 
Comment dรฉvelopper une stratรฉgie Big Data dans le cloud public avec l'offre P...
Comment dรฉvelopper une stratรฉgie Big Data dans le cloud public avec l'offre P...Comment dรฉvelopper une stratรฉgie Big Data dans le cloud public avec l'offre P...
Comment dรฉvelopper une stratรฉgie Big Data dans le cloud public avec l'offre P...
ย 
Hadoop security @ Philly Hadoop Meetup May 2015
Hadoop security @ Philly Hadoop Meetup May 2015Hadoop security @ Philly Hadoop Meetup May 2015
Hadoop security @ Philly Hadoop Meetup May 2015
ย 
Edge to ai analytics from edge to cloud with efficient movement of machine data
Edge to ai  analytics from edge to cloud with efficient movement of machine dataEdge to ai  analytics from edge to cloud with efficient movement of machine data
Edge to ai analytics from edge to cloud with efficient movement of machine data
ย 
A deep dive into running data analytic workloads in the cloud
A deep dive into running data analytic workloads in the cloudA deep dive into running data analytic workloads in the cloud
A deep dive into running data analytic workloads in the cloud
ย 
Cloudera ใฎใ‚ตใƒใƒผใƒˆใ‚จใƒณใ‚ธใƒ‹ใ‚ขใƒชใƒณใ‚ฐ #supennight
Cloudera ใฎใ‚ตใƒใƒผใƒˆใ‚จใƒณใ‚ธใƒ‹ใ‚ขใƒชใƒณใ‚ฐ #supennightCloudera ใฎใ‚ตใƒใƒผใƒˆใ‚จใƒณใ‚ธใƒ‹ใ‚ขใƒชใƒณใ‚ฐ #supennight
Cloudera ใฎใ‚ตใƒใƒผใƒˆใ‚จใƒณใ‚ธใƒ‹ใ‚ขใƒชใƒณใ‚ฐ #supennight
ย 
Kafka for DBAs
Kafka for DBAsKafka for DBAs
Kafka for DBAs
ย 
The Edge to AI Deep Dive Barcelona Meetup March 2019
The Edge to AI Deep Dive Barcelona Meetup March 2019The Edge to AI Deep Dive Barcelona Meetup March 2019
The Edge to AI Deep Dive Barcelona Meetup March 2019
ย 
OOW-5185-Hybrid Cloud
OOW-5185-Hybrid CloudOOW-5185-Hybrid Cloud
OOW-5185-Hybrid Cloud
ย 

More from Andrei Savu

Counters with Riak on Amazon EC2 at Hackover
Counters with Riak on Amazon EC2 at HackoverCounters with Riak on Amazon EC2 at Hackover
Counters with Riak on Amazon EC2 at Hackover
Andrei Savu
ย 
Polyglot Persistence & Big Data in the Cloud
Polyglot Persistence & Big Data in the CloudPolyglot Persistence & Big Data in the Cloud
Polyglot Persistence & Big Data in the Cloud
Andrei Savu
ย 
Apache Whirr
Apache WhirrApache Whirr
Apache Whirr
Andrei Savu
ย 
Apache ZooKeeper TechTuesday
Apache ZooKeeper TechTuesdayApache ZooKeeper TechTuesday
Apache ZooKeeper TechTuesday
Andrei Savu
ย 
HBase Feed Aggregator Wurbe 25
HBase Feed Aggregator Wurbe 25HBase Feed Aggregator Wurbe 25
HBase Feed Aggregator Wurbe 25
Andrei Savu
ย 

More from Andrei Savu (20)

The Evolving Landscape of Data Engineering
The Evolving Landscape of Data EngineeringThe Evolving Landscape of Data Engineering
The Evolving Landscape of Data Engineering
ย 
The Evolving Landscape of Data Engineering
The Evolving Landscape of Data EngineeringThe Evolving Landscape of Data Engineering
The Evolving Landscape of Data Engineering
ย 
Cloud as a Data Platform
Cloud as a Data PlatformCloud as a Data Platform
Cloud as a Data Platform
ย 
Apache Provisionr (incubating) - Bucharest JUG 10
Apache Provisionr (incubating) - Bucharest JUG 10Apache Provisionr (incubating) - Bucharest JUG 10
Apache Provisionr (incubating) - Bucharest JUG 10
ย 
Creating pools of Virtual Machines - ApacheCon NA 2013
Creating pools of Virtual Machines - ApacheCon NA 2013Creating pools of Virtual Machines - ApacheCon NA 2013
Creating pools of Virtual Machines - ApacheCon NA 2013
ย 
Data Scientist Toolbox
Data Scientist ToolboxData Scientist Toolbox
Data Scientist Toolbox
ย 
Axemblr Provisionr 0.3.x Overview
Axemblr Provisionr 0.3.x OverviewAxemblr Provisionr 0.3.x Overview
Axemblr Provisionr 0.3.x Overview
ย 
2012 in Review - Bucharest JUG
2012 in Review - Bucharest JUG2012 in Review - Bucharest JUG
2012 in Review - Bucharest JUG
ย 
Metrics for Web Applications - Netcamp 2012
Metrics for Web Applications - Netcamp 2012Metrics for Web Applications - Netcamp 2012
Metrics for Web Applications - Netcamp 2012
ย 
Counters with Riak on Amazon EC2 at Hackover
Counters with Riak on Amazon EC2 at HackoverCounters with Riak on Amazon EC2 at Hackover
Counters with Riak on Amazon EC2 at Hackover
ย 
Simple REST with Dropwizard
Simple REST with DropwizardSimple REST with Dropwizard
Simple REST with Dropwizard
ย 
Guava Overview Part 2 Bucharest JUG #2
Guava Overview Part 2 Bucharest JUG #2 Guava Overview Part 2 Bucharest JUG #2
Guava Overview Part 2 Bucharest JUG #2
ย 
Guava Overview. Part 1 @ Bucharest JUG #1
Guava Overview. Part 1 @ Bucharest JUG #1 Guava Overview. Part 1 @ Bucharest JUG #1
Guava Overview. Part 1 @ Bucharest JUG #1
ย 
Polyglot Persistence & Big Data in the Cloud
Polyglot Persistence & Big Data in the CloudPolyglot Persistence & Big Data in the Cloud
Polyglot Persistence & Big Data in the Cloud
ย 
Building a Great Team in Open Source - Open Agile 2011
Building a Great Team in Open Source - Open Agile 2011Building a Great Team in Open Source - Open Agile 2011
Building a Great Team in Open Source - Open Agile 2011
ย 
Apache Whirr
Apache WhirrApache Whirr
Apache Whirr
ย 
Automated Testing for Web Applications - Wurbe #36
Automated Testing for Web Applications - Wurbe #36Automated Testing for Web Applications - Wurbe #36
Automated Testing for Web Applications - Wurbe #36
ย 
Apache ZooKeeper TechTuesday
Apache ZooKeeper TechTuesdayApache ZooKeeper TechTuesday
Apache ZooKeeper TechTuesday
ย 
HBase Feed Aggregator Wurbe 25
HBase Feed Aggregator Wurbe 25HBase Feed Aggregator Wurbe 25
HBase Feed Aggregator Wurbe 25
ย 
Indekspot.com - Trouble free Apache Solr
Indekspot.com - Trouble free Apache SolrIndekspot.com - Trouble free Apache Solr
Indekspot.com - Trouble free Apache Solr
ย 

Recently uploaded

CALL ON โžฅ8923113531 ๐Ÿ”Call Girls Badshah Nagar Lucknow best Female service
CALL ON โžฅ8923113531 ๐Ÿ”Call Girls Badshah Nagar Lucknow best Female serviceCALL ON โžฅ8923113531 ๐Ÿ”Call Girls Badshah Nagar Lucknow best Female service
CALL ON โžฅ8923113531 ๐Ÿ”Call Girls Badshah Nagar Lucknow best Female service
anilsa9823
ย 
Hand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptxHand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptx
bodapatigopi8531
ย 
CHEAP Call Girls in Pushp Vihar (-DELHI )๐Ÿ” 9953056974๐Ÿ”(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )๐Ÿ” 9953056974๐Ÿ”(=)/CALL GIRLS SERVICECHEAP Call Girls in Pushp Vihar (-DELHI )๐Ÿ” 9953056974๐Ÿ”(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )๐Ÿ” 9953056974๐Ÿ”(=)/CALL GIRLS SERVICE
9953056974 Low Rate Call Girls In Saket, Delhi NCR
ย 
CALL ON โžฅ8923113531 ๐Ÿ”Call Girls Kakori Lucknow best sexual service Online โ˜‚๏ธ
CALL ON โžฅ8923113531 ๐Ÿ”Call Girls Kakori Lucknow best sexual service Online  โ˜‚๏ธCALL ON โžฅ8923113531 ๐Ÿ”Call Girls Kakori Lucknow best sexual service Online  โ˜‚๏ธ
CALL ON โžฅ8923113531 ๐Ÿ”Call Girls Kakori Lucknow best sexual service Online โ˜‚๏ธ
anilsa9823
ย 
TECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service providerTECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service provider
mohitmore19
ย 

Recently uploaded (20)

CALL ON โžฅ8923113531 ๐Ÿ”Call Girls Badshah Nagar Lucknow best Female service
CALL ON โžฅ8923113531 ๐Ÿ”Call Girls Badshah Nagar Lucknow best Female serviceCALL ON โžฅ8923113531 ๐Ÿ”Call Girls Badshah Nagar Lucknow best Female service
CALL ON โžฅ8923113531 ๐Ÿ”Call Girls Badshah Nagar Lucknow best Female service
ย 
Vip Call Girls Noida โžก๏ธ Delhi โžก๏ธ 9999965857 No Advance 24HRS Live
Vip Call Girls Noida โžก๏ธ Delhi โžก๏ธ 9999965857 No Advance 24HRS LiveVip Call Girls Noida โžก๏ธ Delhi โžก๏ธ 9999965857 No Advance 24HRS Live
Vip Call Girls Noida โžก๏ธ Delhi โžก๏ธ 9999965857 No Advance 24HRS Live
ย 
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdfThe Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
ย 
Reassessing the Bedrock of Clinical Function Models: An Examination of Large ...
Reassessing the Bedrock of Clinical Function Models: An Examination of Large ...Reassessing the Bedrock of Clinical Function Models: An Examination of Large ...
Reassessing the Bedrock of Clinical Function Models: An Examination of Large ...
ย 
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlanโ€™s ...
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlanโ€™s ...Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlanโ€™s ...
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlanโ€™s ...
ย 
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected WorkerHow To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
ย 
Software Quality Assurance Interview Questions
Software Quality Assurance Interview QuestionsSoftware Quality Assurance Interview Questions
Software Quality Assurance Interview Questions
ย 
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time ApplicationsUnveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
ย 
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
ย 
Right Money Management App For Your Financial Goals
Right Money Management App For Your Financial GoalsRight Money Management App For Your Financial Goals
Right Money Management App For Your Financial Goals
ย 
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
ย 
Hand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptxHand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptx
ย 
Microsoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdfMicrosoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdf
ย 
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AISyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
ย 
CHEAP Call Girls in Pushp Vihar (-DELHI )๐Ÿ” 9953056974๐Ÿ”(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )๐Ÿ” 9953056974๐Ÿ”(=)/CALL GIRLS SERVICECHEAP Call Girls in Pushp Vihar (-DELHI )๐Ÿ” 9953056974๐Ÿ”(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )๐Ÿ” 9953056974๐Ÿ”(=)/CALL GIRLS SERVICE
ย 
HR Software Buyers Guide in 2024 - HRSoftware.com
HR Software Buyers Guide in 2024 - HRSoftware.comHR Software Buyers Guide in 2024 - HRSoftware.com
HR Software Buyers Guide in 2024 - HRSoftware.com
ย 
Unlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language ModelsUnlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language Models
ย 
CALL ON โžฅ8923113531 ๐Ÿ”Call Girls Kakori Lucknow best sexual service Online โ˜‚๏ธ
CALL ON โžฅ8923113531 ๐Ÿ”Call Girls Kakori Lucknow best sexual service Online  โ˜‚๏ธCALL ON โžฅ8923113531 ๐Ÿ”Call Girls Kakori Lucknow best sexual service Online  โ˜‚๏ธ
CALL ON โžฅ8923113531 ๐Ÿ”Call Girls Kakori Lucknow best sexual service Online โ˜‚๏ธ
ย 
TECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service providerTECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service provider
ย 
Optimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTVOptimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTV
ย 

One Hadoop, Multiple Clouds - NYC Big Data Meetup

  • 1. 1ยฉ Cloudera, Inc. All rights reserved. One Hadoop, Multiple Clouds Andrei Savu | Tech Lead, Cloudera Director
  • 2. 2ยฉ Cloudera, Inc. All rights reserved. About me Tech Lead on Cloudera Director Previously founder of axemblr.com Contributed to Apache Whirr (PMC) & jclouds. Twitter: https://twitter.com/andreisavu LinkedIn: https://www.linkedin.com/in/sandrei
  • 3. 3ยฉ Cloudera, Inc. All rights reserved. Cloudera Director cloudera.com/director Deploy and manage enterprise-grade Hadoop in the cloud AWS & Google Cloud Extensible via plugins
  • 5. 5ยฉ Cloudera, Inc. All rights reserved. Do you use a public or private cloud? How do you run and manage Hadoop?
  • 6. 6ยฉ Cloudera, Inc. All rights reserved. What is this talk about? State of the World Architectural Patterns Imagine the Future
  • 7. 7ยฉ Cloudera, Inc. All rights reserved. Gartner's 2015 Hype Cycle for Emerging Technologies (source) Advanced Analytics Hybrid Cloud Internet of Things
  • 8. 8ยฉ Cloudera, Inc. All rights reserved. Hybrid Clouds Cloud Exchange Application Portability Private-Public Public-Public
  • 9. 9ยฉ Cloudera, Inc. All rights reserved. Cloud Wars AWS Microsoft Azure Google Cloud VMWare Openstack etc.
  • 10. 10ยฉ Cloudera, Inc. All rights reserved. Data has Mass and Gravity
  • 11. 11ยฉ Cloudera, Inc. All rights reserved. Hadoop Environments On-Premise versus Cloud On-Premise Cloud Storage Direct Attached Direct Attached or Object Store Data Not shared across clusters Shared across multiple clusters Sizing Fixed-size Dynamic based on load Usage Model All users share cluster Clusters created as needed for apps/users Resource Management (YARN) HDFS Process Discover Model Serve Industry Standard Servers (CPU, Memory, & Direct Attached Storage) Resource Management (YARN) HDFS Process Discover Model Serve Industry Standard Servers (CPU & Memory) Object Storage
  • 12. 12ยฉ Cloudera, Inc. All rights reserved. Cloud providers shipping distributions of Hadoop Integration Unlock Query Engines Migration workloads Is that a sustainable advantage? Or just a temporary stop gap?
  • 13. 13ยฉ Cloudera, Inc. All rights reserved. Maturity level On-prem vs. Cloud Monitoring Dev / Test / Prod Availability Durability
  • 14. 14ยฉ Cloudera, Inc. All rights reserved. Common Architectural Patterns in the Cloud Object Storage Source Data Seed Data Backup/DR ETL/MODELING (Spark, MapReduce) โ€ข Short-running clusters โ€ข Elastic workload โ€ข No local storage necessary |WASB |SWIFT |BLOB โ€ข Long-running clusters โ€ข Sized to demand โ€ข Some local storage BI/ANALYTICS (Impala, Solr) โ€ข Fixed clusters โ€ข Periodic sync โ€ข Default to local storage APP DELIVERY (HBase, Kudu)
  • 15. 15ยฉ Cloudera, Inc. All rights reserved. Cluster lifecycle management Create / Terminate Discovery Metadata Monitoring
  • 16. 16ยฉ Cloudera, Inc. All rights reserved. Work Queue Workflows Dispatch Tracking Decoupled Fault Tolerant
  • 17. 17ยฉ Cloudera, Inc. All rights reserved. Common Architectural Patterns in the Cloud Object Storage Source Data Seed Data Backup/DR ETL/MODELING (Spark, MapReduce) โ€ข Short-running clusters โ€ข Elastic workload โ€ข No local storage necessary |WASB |SWIFT |BLOB โ€ข Long-running clusters โ€ข Sized to demand โ€ข Some local storage BI/ANALYTICS (Impala, Solr) โ€ข Fixed clusters โ€ข Periodic sync โ€ข Default to local storage APP DELIVERY (HBase, Kudu)
  • 18. 18ยฉ Cloudera, Inc. All rights reserved. Multi-user Secure Isolated Friendly
  • 19. 19ยฉ Cloudera, Inc. All rights reserved. Elastic Grow or shrink Business hours Number of users Storage vs. Compute Cost efficient
  • 20. 20ยฉ Cloudera, Inc. All rights reserved. Common Architectural Patterns in the Cloud Object Storage Source Data Seed Data Backup/DR ETL/MODELING (Spark, MapReduce) โ€ข Short-running clusters โ€ข Elastic workload โ€ข No local storage necessary |WASB |SWIFT |BLOB โ€ข Long-running clusters โ€ข Sized to demand โ€ข Some local storage BI/ANALYTICS (Impala, Solr) โ€ข Fixed clusters โ€ข Periodic sync โ€ข Default to local storage APP DELIVERY (HBase, Kudu)
  • 21. 21ยฉ Cloudera, Inc. All rights reserved. Advanced Monitoring Latency Resource utilization Consistent performance
  • 22. 22ยฉ Cloudera, Inc. All rights reserved. High availability and failure domains Data durability Repair within SLA Host-to-instance
  • 23. 23ยฉ Cloudera, Inc. All rights reserved. Backup and disaster recovery Object store centric Active-Standby
  • 24. 24ยฉ Cloudera, Inc. All rights reserved. Imagine the Future Portable Experience Self-service Self-healing Granular Security Advanced Governance Complete Management Whatโ€™s your vision?
  • 25.
  • 26. 26ยฉ Cloudera, Inc. All rights reserved. Thank you! asavu@cloudera.com
  • 27. 27ยฉ Cloudera, Inc. All rights reserved. Resources Cloudera Director: http://www.cloudera.com/director Interested in API level integration and scripting? https://github.com/cloudera/director-sdk https://github.com/cloudera/director-scripts Interested in integration with another cloud platform? https://github.com/cloudera/director-spi https://github.com/cloudera/director-google-plugin
  • 28. 28ยฉ Cloudera, Inc. All rights reserved. Whatโ€™s new in Cloudera Director 1.5? http://blog.cloudera.com/blog/2015/08/whats-new-in- cloudera-director-1-5/ Get Started AWS Reference Guide GCP Reference Guide Try It Out AWS Quickstart Resources
  • 30. ยฉ 2014 Cloudera, Inc. All rights reserved.
  • 31. ยฉ 2014 Cloudera, Inc. All rights reserved.
  • 32. ยฉ 2014 Cloudera, Inc. All rights reserved.
  • 33. ยฉ 2014 Cloudera, Inc. All rights reserved.
  • 34. ยฉ 2014 Cloudera, Inc. All rights reserved.
  • 35. ยฉ 2014 Cloudera, Inc. All rights reserved.
  • 36. ยฉ 2014 Cloudera, Inc. All rights reserved.
  • 37. ยฉ 2014 Cloudera, Inc. All rights reserved.
  • 38. ยฉ 2014 Cloudera, Inc. All rights reserved.
  • 39. ยฉ 2014 Cloudera, Inc. All rights reserved.
  • 40. ยฉ 2014 Cloudera, Inc. All rights reserved.
  • 41. ยฉ 2014 Cloudera, Inc. All rights reserved.
  • 42. ยฉ 2014 Cloudera, Inc. All rights reserved.
  • 43. ยฉ 2014 Cloudera, Inc. All rights reserved.
  • 44. ยฉ 2014 Cloudera, Inc. All rights reserved.
  • 45. 45ยฉ Cloudera, Inc. All rights reserved. Thank you! asavu@cloudera.com