SlideShare a Scribd company logo
1 of 26
Hadoop on EC2 Configuring and running Hadoop clusters, using Cloudera distribution
My farm ,[object Object]
Start ,[object Object]
Confirm ,[object Object]
OK, it's running ,[object Object]
Set /etc/hosts ,[object Object]
Logging into an EC2 machine ,[object Object],[object Object],[object Object],[object Object]
Run command on the cluster ,[object Object],[object Object],[object Object],[object Object],[object Object]
Result of running ifconfig ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Edit local conf ,[object Object],[object Object]
masters ,[object Object]
core-site.xml ,[object Object]
mapred-site.xml ,[object Object]
slaves ,[object Object]
update-hadoop-cluster.sh ,[object Object]
Important gotchas ,[object Object],[object Object]
Important gotchas - 2 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Now start Hadoop services ,[object Object],[object Object],[object Object],[object Object]
Verify HDFS and MR ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
w3m http://localhost:50070
w3m next screen ,[object Object]
Start HBase ,[object Object],[object Object]
w3m http://localhost:60010
Stop HBase ,[object Object],[object Object],[object Object]
Amazon EMR ,[object Object]
Whirr ,[object Object]

More Related Content

What's hot

Running hadoop on ubuntu linux
Running hadoop on ubuntu linuxRunning hadoop on ubuntu linux
Running hadoop on ubuntu linux
TRCK
 
Build your own secure mail server on the cloud using Amazon Web Services
Build your own secure mail server on the cloud using Amazon Web ServicesBuild your own secure mail server on the cloud using Amazon Web Services
Build your own secure mail server on the cloud using Amazon Web Services
ponukumatla joel nishanth
 
Python Deployment with Fabric
Python Deployment with FabricPython Deployment with Fabric
Python Deployment with Fabric
andymccurdy
 

What's hot (20)

Ansible Network Automation session1
Ansible Network Automation session1Ansible Network Automation session1
Ansible Network Automation session1
 
#OktoCampus - Workshop : An introduction to Ansible
#OktoCampus - Workshop : An introduction to Ansible#OktoCampus - Workshop : An introduction to Ansible
#OktoCampus - Workshop : An introduction to Ansible
 
Hadoop single node setup
Hadoop single node setupHadoop single node setup
Hadoop single node setup
 
How to scheduled jobs in a cloudera cluster without oozie
How to scheduled jobs in a cloudera cluster without oozieHow to scheduled jobs in a cloudera cluster without oozie
How to scheduled jobs in a cloudera cluster without oozie
 
Vagrant, Ansible, and OpenStack on your laptop
Vagrant, Ansible, and OpenStack on your laptopVagrant, Ansible, and OpenStack on your laptop
Vagrant, Ansible, and OpenStack on your laptop
 
Running hadoop on ubuntu linux
Running hadoop on ubuntu linuxRunning hadoop on ubuntu linux
Running hadoop on ubuntu linux
 
Install and Configure Ubuntu for Hadoop Installation for beginners
Install and Configure Ubuntu for Hadoop Installation for beginners Install and Configure Ubuntu for Hadoop Installation for beginners
Install and Configure Ubuntu for Hadoop Installation for beginners
 
Learn basic ansible using docker
Learn basic ansible using dockerLearn basic ansible using docker
Learn basic ansible using docker
 
Ansible : what's ansible & use case by REX
Ansible :  what's ansible & use case by REXAnsible :  what's ansible & use case by REX
Ansible : what's ansible & use case by REX
 
How to configure a hive high availability connection with zeppelin
How to configure a hive high availability connection with zeppelinHow to configure a hive high availability connection with zeppelin
How to configure a hive high availability connection with zeppelin
 
Build your own secure mail server on the cloud using Amazon Web Services
Build your own secure mail server on the cloud using Amazon Web ServicesBuild your own secure mail server on the cloud using Amazon Web Services
Build your own secure mail server on the cloud using Amazon Web Services
 
My sql failover test using orchestrator
My sql failover test  using orchestratorMy sql failover test  using orchestrator
My sql failover test using orchestrator
 
Ansible not only for Dummies
Ansible not only for DummiesAnsible not only for Dummies
Ansible not only for Dummies
 
How to implement a gdpr solution in a cloudera architecture
How to implement a gdpr solution in a cloudera architectureHow to implement a gdpr solution in a cloudera architecture
How to implement a gdpr solution in a cloudera architecture
 
Introduction to Ansible
Introduction to AnsibleIntroduction to Ansible
Introduction to Ansible
 
DevOps(3) : Ansible - (MOSG)
DevOps(3) : Ansible - (MOSG)DevOps(3) : Ansible - (MOSG)
DevOps(3) : Ansible - (MOSG)
 
10 Million hits a day with WordPress using a $15 VPS
10 Million hits a day  with WordPress using a $15 VPS10 Million hits a day  with WordPress using a $15 VPS
10 Million hits a day with WordPress using a $15 VPS
 
Python Deployment with Fabric
Python Deployment with FabricPython Deployment with Fabric
Python Deployment with Fabric
 
Cloudera cluster setup and configuration
Cloudera cluster setup and configurationCloudera cluster setup and configuration
Cloudera cluster setup and configuration
 
Making Your Capistrano Recipe Book
Making Your Capistrano Recipe BookMaking Your Capistrano Recipe Book
Making Your Capistrano Recipe Book
 

Viewers also liked

Viewers also liked (11)

Configuring Your First Hadoop Cluster On EC2
Configuring Your First Hadoop Cluster On EC2Configuring Your First Hadoop Cluster On EC2
Configuring Your First Hadoop Cluster On EC2
 
Linux startup
Linux startupLinux startup
Linux startup
 
Toorcamp 2016
Toorcamp 2016Toorcamp 2016
Toorcamp 2016
 
Hadoop Hadoop & Spark meetup - Altiscale
Hadoop Hadoop & Spark meetup - AltiscaleHadoop Hadoop & Spark meetup - Altiscale
Hadoop Hadoop & Spark meetup - Altiscale
 
Cloudera search
Cloudera searchCloudera search
Cloudera search
 
Witsml data processing with kafka and spark streaming
Witsml data processing with kafka and spark streamingWitsml data processing with kafka and spark streaming
Witsml data processing with kafka and spark streaming
 
Hadoop as a service presented by Ajay Jha at Houston Hadoop Meetup
Hadoop as a service presented by Ajay Jha at Houston Hadoop MeetupHadoop as a service presented by Ajay Jha at Houston Hadoop Meetup
Hadoop as a service presented by Ajay Jha at Houston Hadoop Meetup
 
Oil and gas big data edition
Oil and gas  big data editionOil and gas  big data edition
Oil and gas big data edition
 
Building a distributed search system with Hadoop and Lucene
Building a distributed search system with Hadoop and LuceneBuilding a distributed search system with Hadoop and Lucene
Building a distributed search system with Hadoop and Lucene
 
Hadoop Overview & Architecture
Hadoop Overview & Architecture  Hadoop Overview & Architecture
Hadoop Overview & Architecture
 
Big Data Analytics with Hadoop
Big Data Analytics with HadoopBig Data Analytics with Hadoop
Big Data Analytics with Hadoop
 

Similar to Hadoop on ec2

allscripts.pdf-----schedule.sh------ #!binbash #ssh .docx
allscripts.pdf-----schedule.sh------ #!binbash #ssh .docxallscripts.pdf-----schedule.sh------ #!binbash #ssh .docx
allscripts.pdf-----schedule.sh------ #!binbash #ssh .docx
galerussel59292
 
Introduction To Programming in GNU-LINUX
Introduction To Programming in GNU-LINUXIntroduction To Programming in GNU-LINUX
Introduction To Programming in GNU-LINUX
Irfan Ahmed
 
Dockerfish-Tutorial
Dockerfish-TutorialDockerfish-Tutorial
Dockerfish-Tutorial
Brian Hood
 

Similar to Hadoop on ec2 (20)

Kickstart
KickstartKickstart
Kickstart
 
allscripts.pdf-----schedule.sh------ #!binbash #ssh .docx
allscripts.pdf-----schedule.sh------ #!binbash #ssh .docxallscripts.pdf-----schedule.sh------ #!binbash #ssh .docx
allscripts.pdf-----schedule.sh------ #!binbash #ssh .docx
 
Hadoop 2.4 installing on ubuntu 14.04
Hadoop 2.4 installing on ubuntu 14.04Hadoop 2.4 installing on ubuntu 14.04
Hadoop 2.4 installing on ubuntu 14.04
 
Cloud Automation with Opscode Chef
Cloud Automation with Opscode ChefCloud Automation with Opscode Chef
Cloud Automation with Opscode Chef
 
15 map reduce on azure
15 map reduce on azure15 map reduce on azure
15 map reduce on azure
 
15 map reduce on azure
15 map reduce on azure15 map reduce on azure
15 map reduce on azure
 
Sup intro
Sup introSup intro
Sup intro
 
More than Applications: (Ab)using Docker to Improve the Portability of Everyt...
More than Applications: (Ab)using Docker to Improve the Portability of Everyt...More than Applications: (Ab)using Docker to Improve the Portability of Everyt...
More than Applications: (Ab)using Docker to Improve the Portability of Everyt...
 
Deploying Docker (Provisioning /w Docker + Chef/Puppet) - DevopsDaysPGH
Deploying Docker (Provisioning /w Docker + Chef/Puppet) - DevopsDaysPGHDeploying Docker (Provisioning /w Docker + Chef/Puppet) - DevopsDaysPGH
Deploying Docker (Provisioning /w Docker + Chef/Puppet) - DevopsDaysPGH
 
Introduction To Programming in GNU-LINUX
Introduction To Programming in GNU-LINUXIntroduction To Programming in GNU-LINUX
Introduction To Programming in GNU-LINUX
 
Linuxserver harden
Linuxserver hardenLinuxserver harden
Linuxserver harden
 
Linux Containers From Scratch
Linux Containers From ScratchLinux Containers From Scratch
Linux Containers From Scratch
 
ERP System Implementation Kubernetes Cluster with Sticky Sessions
ERP System Implementation Kubernetes Cluster with Sticky Sessions ERP System Implementation Kubernetes Cluster with Sticky Sessions
ERP System Implementation Kubernetes Cluster with Sticky Sessions
 
Configure h base hadoop and hbase client
Configure h base hadoop and hbase clientConfigure h base hadoop and hbase client
Configure h base hadoop and hbase client
 
Jenkins 101: Getting Started
Jenkins 101: Getting StartedJenkins 101: Getting Started
Jenkins 101: Getting Started
 
Hadoop installation
Hadoop installationHadoop installation
Hadoop installation
 
Edup wifi for raspberry zero
Edup wifi  for raspberry zeroEdup wifi  for raspberry zero
Edup wifi for raspberry zero
 
Tame Your Build And Deployment Process With Hudson, PHPUnit, and SSH
Tame Your Build And Deployment Process With Hudson, PHPUnit, and SSHTame Your Build And Deployment Process With Hudson, PHPUnit, and SSH
Tame Your Build And Deployment Process With Hudson, PHPUnit, and SSH
 
Dev ops
Dev opsDev ops
Dev ops
 
Dockerfish-Tutorial
Dockerfish-TutorialDockerfish-Tutorial
Dockerfish-Tutorial
 

More from Mark Kerzner

FreeEed popcorn overview
FreeEed popcorn overviewFreeEed popcorn overview
FreeEed popcorn overview
Mark Kerzner
 
FreeEed presentation
FreeEed presentationFreeEed presentation
FreeEed presentation
Mark Kerzner
 
Night owl by Boyd Meyer of PROS
Night owl by Boyd Meyer of PROS Night owl by Boyd Meyer of PROS
Night owl by Boyd Meyer of PROS
Mark Kerzner
 
Porting your hadoop app to horton works hdp
Porting your hadoop app to horton works hdpPorting your hadoop app to horton works hdp
Porting your hadoop app to horton works hdp
Mark Kerzner
 
Google Office in Zurich, Switzerland
Google Office in Zurich, SwitzerlandGoogle Office in Zurich, Switzerland
Google Office in Zurich, Switzerland
Mark Kerzner
 
Fun art with fruit and vegetable
Fun art with fruit and vegetableFun art with fruit and vegetable
Fun art with fruit and vegetable
Mark Kerzner
 
Carnavale de Venice
Carnavale de VeniceCarnavale de Venice
Carnavale de Venice
Mark Kerzner
 

More from Mark Kerzner (20)

IBM Strategy for Spark
IBM Strategy for SparkIBM Strategy for Spark
IBM Strategy for Spark
 
Joe Witt presentation on Apache NiFi
Joe Witt presentation on Apache NiFiJoe Witt presentation on Apache NiFi
Joe Witt presentation on Apache NiFi
 
FreeEed popcorn overview
FreeEed popcorn overviewFreeEed popcorn overview
FreeEed popcorn overview
 
FreeEed presentation
FreeEed presentationFreeEed presentation
FreeEed presentation
 
Nutch + Hadoop scaled, for crawling protected web sites (hint: Selenium)
Nutch + Hadoop scaled, for crawling protected web sites (hint: Selenium)Nutch + Hadoop scaled, for crawling protected web sites (hint: Selenium)
Nutch + Hadoop scaled, for crawling protected web sites (hint: Selenium)
 
Night owl by Boyd Meyer of PROS
Night owl by Boyd Meyer of PROS Night owl by Boyd Meyer of PROS
Night owl by Boyd Meyer of PROS
 
SHMcloud vision
SHMcloud visionSHMcloud vision
SHMcloud vision
 
Porting your hadoop app to horton works hdp
Porting your hadoop app to horton works hdpPorting your hadoop app to horton works hdp
Porting your hadoop app to horton works hdp
 
Automated Hadoop Cluster Construction on EC2
Automated Hadoop Cluster Construction on EC2Automated Hadoop Cluster Construction on EC2
Automated Hadoop Cluster Construction on EC2
 
Open source e_discovery
Open source e_discoveryOpen source e_discovery
Open source e_discovery
 
FreEed - Open Source eDiscovery
FreEed - Open Source eDiscoveryFreEed - Open Source eDiscovery
FreEed - Open Source eDiscovery
 
Houston Hadoop Meetup Presentation by Vikram Oberoi of Cloudera
Houston Hadoop Meetup Presentation by Vikram Oberoi of ClouderaHouston Hadoop Meetup Presentation by Vikram Oberoi of Cloudera
Houston Hadoop Meetup Presentation by Vikram Oberoi of Cloudera
 
Google Office in Zurich, Switzerland
Google Office in Zurich, SwitzerlandGoogle Office in Zurich, Switzerland
Google Office in Zurich, Switzerland
 
Fun art with fruit and vegetable
Fun art with fruit and vegetableFun art with fruit and vegetable
Fun art with fruit and vegetable
 
Carnavale de Venice
Carnavale de VeniceCarnavale de Venice
Carnavale de Venice
 
Holocaust Memorial Tato
Holocaust Memorial TatoHolocaust Memorial Tato
Holocaust Memorial Tato
 
Yehuda Pen
Yehuda PenYehuda Pen
Yehuda Pen
 
Mark Chagall
Mark ChagallMark Chagall
Mark Chagall
 
Thailand Visite
Thailand VisiteThailand Visite
Thailand Visite
 
Venice views with music
Venice views with musicVenice views with music
Venice views with music
 

Recently uploaded

Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 

Recently uploaded (20)

Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your Business
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfTech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdf
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 

Hadoop on ec2