SlideShare a Scribd company logo

HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa

This session will discuss how you can represent your complete cluster with one config file and have it deployed to Cloud or Bare Metal. Infochmimps’ Ironfan builds on Opscode Chef to allow you to specify and orchestrate all flavors of your cluster’s deployment, monitoring and growth. Not just the core HBase/HDFS/MapReduce/Hive/Flume, etc. but all the elements including web / app servers, mysql, redis, rabbitmq and whatever other servers needed to implement your service. These same tools can manage variations for development, staging, R&D as well as the target “rendering” to various Clouds, Bare Metal or even Vagrant VMs.

1 of 24
Download to read offline
Orchestrating Clusters
with Ironfan and Chef
    Robert J. Berger - CTO Runa, Inc.
          rberger@runa.com
Hassles of Big Data Stack
      Deployments
• Lots of Moving Parts
 •   Hadoop/HBase just one sub-system
 •   Heterogeneous Tech
 •   Monitoring & Metrics Everywhere
 •   Details obscure the big picture
• Need repeatability & variations on themes
The Forest for the Trees
Forest
Dash        App
                     Monitor
board      Servers




Elastic              Hadoop
           AMQP
Search                M/R




Session
           HBase     MySQL
 Store
MySQL




                                                                                       Slave
                                                                 Slave               Slave
                                                               Slave             Master
                   GDash
                Statsd
               Sensu                                    Slave
          Logstash                                     Slave
         Ganglia                                   Slave
                        Monitoring




        Graphite                                  Slave
        Web                                     Slave                                                           Regionsrvr
                                               Slave                                                      Regionsrvr
                                           Slave
                                                          Hadoop
                                                           M/R




                                          Slave                                                  Regionsrvr
Trees




                                      Sec Master
                                                                                               Regionsrvr
                                      Master
                                                                                             Regionsrvr
                                                                                          Regionsrvr
               Clj App                                                                  Regionsrvr
              Clj App
                                                                                       Regionsrvr
          Clj App
                                                                                     Regionsrvr
                        App Servers




          Web
                                           Rabbit                                Regionsrvr
         Web
                                          Rabbit        AMQP                    Regionsrvr
        ELB
                                                                               HB Master




                                                                                                HBase
                                                                             HB Master
                                                                            ZooKeeper
                                                                          ZooKeeper
                                                                         ZooKeeper
                                           ES Server
                                          ES Server
          Rails App                     ES Server




                                                       Search
                                                       Elastic
                          Dashboard
         Rails App                     ES Server
         Web
                                                                                     Redis
                                                                                 Redis




                                                                                               Session
                                                                                                Store
                                                                                Redis
Leaves
Dashboard                        App Servers                                   Monitoring
 Nginx         MySQL Client       Elastic Load Balancer MySQL Client              Nginx             Ganglia Server
 Reverse Proxy Upstart Config      Nginx                 Upstart Config             Reverse Proxy     Ganglia Web
 Unicorn       Logstash Client    Reverse Proxy         Logstash Client           Java              Statsd Server
 Rails                            Swarmiji                                        Leiningen         Graphite Server
 Dashboard App                    Java                                            Jark              Grpahite Web
 Java                             Leiningen                                       Postfix            Python
 Postfix                           Jark                                            Cron jobs         Logstash
 Cron jobs                        Clojure Apps                                    Sensu Server      MySQL Client
 Sensu client                     HBase Client                                    Sensu Web         Upstart Config
 Sensu plugins                    Postfix                                          Sensu Client      Logstash Client
                                  Cron jobs                                       Sensu Plugins
                                  Sensu client
                                  Sensu plugins

Elastic Search                   AMQP                                     Hadoop M/R
 Elastic Search Server            RabbitMQ                                 Namenode
 Java                             RabbitMQ Plugins                         Secondary Namenode
 Cron jobs                        Cluster Config                            Tasktracker
 Sensu client                     Erlang                                   Jobtrackers
 Sensu plugins                    Cron jobs                                Bootstrap Namenmode
 Upstart Config                    Sensu client                             Java
                                  Sensu plugins                            JMX
                                  Upstart Config                            Cron jobs
                                                                           Sensu client
                                                                           Sensu plugins
                                                                           Ganglia Client
                                                                           Upstart Config

Session Store                    HBase                                             MySQL
Redis                             Namenode            Zookeeper                     MySQL Master
Cron jobs                         Secondary Namenode HBase Master                   MySQL Slaves
Sensu client                      Tasktracker         Regionserver                  Cluster Setup
Sensu plugins                     Jobtrackers                                       Cron jobs
Upstart Config                     Datanodes                                         Sensu client
                                  Bootstrap Namenmode                               Sensu plugins
                                  Java                                              Upstart Config
                                  JMX
                                  Cron jobs
                                  Sensu client
                                  Sensu plugins
                                  Ganglia Client
                                  Upstart Config

Recommended

Orchestrating HBase Cluster Deployment with Ironfan and Chef
Orchestrating HBase Cluster Deployment with Ironfan and ChefOrchestrating HBase Cluster Deployment with Ironfan and Chef
Orchestrating HBase Cluster Deployment with Ironfan and ChefRobert Berger
 
Rapid JCR applications development with Sling
Rapid JCR applications development with SlingRapid JCR applications development with Sling
Rapid JCR applications development with SlingBertrand Delacretaz
 
Scalable Architecture on Amazon AWS Cloud - Indicthreads cloud computing conf...
Scalable Architecture on Amazon AWS Cloud - Indicthreads cloud computing conf...Scalable Architecture on Amazon AWS Cloud - Indicthreads cloud computing conf...
Scalable Architecture on Amazon AWS Cloud - Indicthreads cloud computing conf...IndicThreads
 
Oracle+golden+gate+introduction
Oracle+golden+gate+introductionOracle+golden+gate+introduction
Oracle+golden+gate+introductionxiakaicd
 
Arquitectura dos Serviços da plataforma Windows Azure
Arquitectura dos Serviços da plataforma Windows AzureArquitectura dos Serviços da plataforma Windows Azure
Arquitectura dos Serviços da plataforma Windows AzureComunidade NetPonto
 
Cmu-2011-09.pptx
Cmu-2011-09.pptxCmu-2011-09.pptx
Cmu-2011-09.pptxTed Dunning
 
PhillyDB Hbase and MapR M7 - March 2013
PhillyDB Hbase and MapR M7 - March 2013PhillyDB Hbase and MapR M7 - March 2013
PhillyDB Hbase and MapR M7 - March 2013PhillyDB
 
Apache Con Us2007 Jcr In Action
Apache Con Us2007 Jcr In ActionApache Con Us2007 Jcr In Action
Apache Con Us2007 Jcr In Actionday
 

More Related Content

Viewers also liked

HBaseCon 2012 | HBase powered Merchant Lookup Service at Intuit
HBaseCon 2012 | HBase powered Merchant Lookup Service at IntuitHBaseCon 2012 | HBase powered Merchant Lookup Service at Intuit
HBaseCon 2012 | HBase powered Merchant Lookup Service at IntuitCloudera, Inc.
 
HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, Cloudera
HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, ClouderaHBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, Cloudera
HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, ClouderaCloudera, Inc.
 
Apache HBase Performance Tuning
Apache HBase Performance TuningApache HBase Performance Tuning
Apache HBase Performance TuningLars Hofhansl
 
HBaseCon 2015: Trafodion - Integrating Operational SQL into HBase
HBaseCon 2015: Trafodion - Integrating Operational SQL into HBaseHBaseCon 2015: Trafodion - Integrating Operational SQL into HBase
HBaseCon 2015: Trafodion - Integrating Operational SQL into HBaseHBaseCon
 
HBaseCon 2012 | HBase for the Worlds Libraries - OCLC
HBaseCon 2012 | HBase for the Worlds Libraries - OCLCHBaseCon 2012 | HBase for the Worlds Libraries - OCLC
HBaseCon 2012 | HBase for the Worlds Libraries - OCLCCloudera, Inc.
 
HBaseCon 2012 | Scaling GIS In Three Acts
HBaseCon 2012 | Scaling GIS In Three ActsHBaseCon 2012 | Scaling GIS In Three Acts
HBaseCon 2012 | Scaling GIS In Three ActsCloudera, Inc.
 
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics Cloudera, Inc.
 
Cross-Site BigTable using HBase
Cross-Site BigTable using HBaseCross-Site BigTable using HBase
Cross-Site BigTable using HBaseHBaseCon
 
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...Cloudera, Inc.
 
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...Cloudera, Inc.
 
HBaseCon 2012 | Unique Sets on HBase and Hadoop - Elliot Clark, StumbleUpon
HBaseCon 2012 | Unique Sets on HBase and Hadoop - Elliot Clark, StumbleUponHBaseCon 2012 | Unique Sets on HBase and Hadoop - Elliot Clark, StumbleUpon
HBaseCon 2012 | Unique Sets on HBase and Hadoop - Elliot Clark, StumbleUponCloudera, Inc.
 
HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARN
HBaseCon 2015: DeathStar - Easy, Dynamic,  Multi-tenant HBase via YARNHBaseCon 2015: DeathStar - Easy, Dynamic,  Multi-tenant HBase via YARN
HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARNHBaseCon
 
HBaseCon 2013: Rebuilding for Scale on Apache HBase
HBaseCon 2013: Rebuilding for Scale on Apache HBaseHBaseCon 2013: Rebuilding for Scale on Apache HBase
HBaseCon 2013: Rebuilding for Scale on Apache HBaseCloudera, Inc.
 
HBaseCon 2013: Evolving a First-Generation Apache HBase Deployment to Second...
HBaseCon 2013:  Evolving a First-Generation Apache HBase Deployment to Second...HBaseCon 2013:  Evolving a First-Generation Apache HBase Deployment to Second...
HBaseCon 2013: Evolving a First-Generation Apache HBase Deployment to Second...Cloudera, Inc.
 
HBaseCon 2013: Apache HBase on Flash
HBaseCon 2013: Apache HBase on FlashHBaseCon 2013: Apache HBase on Flash
HBaseCon 2013: Apache HBase on FlashCloudera, Inc.
 
HBaseCon 2013: Being Smarter Than the Smart Meter
HBaseCon 2013: Being Smarter Than the Smart MeterHBaseCon 2013: Being Smarter Than the Smart Meter
HBaseCon 2013: Being Smarter Than the Smart MeterCloudera, Inc.
 
HBaseCon 2013: 1500 JIRAs in 20 Minutes
HBaseCon 2013: 1500 JIRAs in 20 MinutesHBaseCon 2013: 1500 JIRAs in 20 Minutes
HBaseCon 2013: 1500 JIRAs in 20 MinutesCloudera, Inc.
 
HBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBase
HBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBaseHBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBase
HBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBaseCloudera, Inc.
 
HBaseCon 2012 | Building Mobile Infrastructure with HBase
HBaseCon 2012 | Building Mobile Infrastructure with HBaseHBaseCon 2012 | Building Mobile Infrastructure with HBase
HBaseCon 2012 | Building Mobile Infrastructure with HBaseCloudera, Inc.
 
HBaseCon 2013: Apache HBase, Meet Ops. Ops, Meet Apache HBase.
HBaseCon 2013: Apache HBase, Meet Ops. Ops, Meet Apache HBase.HBaseCon 2013: Apache HBase, Meet Ops. Ops, Meet Apache HBase.
HBaseCon 2013: Apache HBase, Meet Ops. Ops, Meet Apache HBase.Cloudera, Inc.
 

Viewers also liked (20)

HBaseCon 2012 | HBase powered Merchant Lookup Service at Intuit
HBaseCon 2012 | HBase powered Merchant Lookup Service at IntuitHBaseCon 2012 | HBase powered Merchant Lookup Service at Intuit
HBaseCon 2012 | HBase powered Merchant Lookup Service at Intuit
 
HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, Cloudera
HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, ClouderaHBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, Cloudera
HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, Cloudera
 
Apache HBase Performance Tuning
Apache HBase Performance TuningApache HBase Performance Tuning
Apache HBase Performance Tuning
 
HBaseCon 2015: Trafodion - Integrating Operational SQL into HBase
HBaseCon 2015: Trafodion - Integrating Operational SQL into HBaseHBaseCon 2015: Trafodion - Integrating Operational SQL into HBase
HBaseCon 2015: Trafodion - Integrating Operational SQL into HBase
 
HBaseCon 2012 | HBase for the Worlds Libraries - OCLC
HBaseCon 2012 | HBase for the Worlds Libraries - OCLCHBaseCon 2012 | HBase for the Worlds Libraries - OCLC
HBaseCon 2012 | HBase for the Worlds Libraries - OCLC
 
HBaseCon 2012 | Scaling GIS In Three Acts
HBaseCon 2012 | Scaling GIS In Three ActsHBaseCon 2012 | Scaling GIS In Three Acts
HBaseCon 2012 | Scaling GIS In Three Acts
 
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
 
Cross-Site BigTable using HBase
Cross-Site BigTable using HBaseCross-Site BigTable using HBase
Cross-Site BigTable using HBase
 
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
 
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
 
HBaseCon 2012 | Unique Sets on HBase and Hadoop - Elliot Clark, StumbleUpon
HBaseCon 2012 | Unique Sets on HBase and Hadoop - Elliot Clark, StumbleUponHBaseCon 2012 | Unique Sets on HBase and Hadoop - Elliot Clark, StumbleUpon
HBaseCon 2012 | Unique Sets on HBase and Hadoop - Elliot Clark, StumbleUpon
 
HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARN
HBaseCon 2015: DeathStar - Easy, Dynamic,  Multi-tenant HBase via YARNHBaseCon 2015: DeathStar - Easy, Dynamic,  Multi-tenant HBase via YARN
HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARN
 
HBaseCon 2013: Rebuilding for Scale on Apache HBase
HBaseCon 2013: Rebuilding for Scale on Apache HBaseHBaseCon 2013: Rebuilding for Scale on Apache HBase
HBaseCon 2013: Rebuilding for Scale on Apache HBase
 
HBaseCon 2013: Evolving a First-Generation Apache HBase Deployment to Second...
HBaseCon 2013:  Evolving a First-Generation Apache HBase Deployment to Second...HBaseCon 2013:  Evolving a First-Generation Apache HBase Deployment to Second...
HBaseCon 2013: Evolving a First-Generation Apache HBase Deployment to Second...
 
HBaseCon 2013: Apache HBase on Flash
HBaseCon 2013: Apache HBase on FlashHBaseCon 2013: Apache HBase on Flash
HBaseCon 2013: Apache HBase on Flash
 
HBaseCon 2013: Being Smarter Than the Smart Meter
HBaseCon 2013: Being Smarter Than the Smart MeterHBaseCon 2013: Being Smarter Than the Smart Meter
HBaseCon 2013: Being Smarter Than the Smart Meter
 
HBaseCon 2013: 1500 JIRAs in 20 Minutes
HBaseCon 2013: 1500 JIRAs in 20 MinutesHBaseCon 2013: 1500 JIRAs in 20 Minutes
HBaseCon 2013: 1500 JIRAs in 20 Minutes
 
HBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBase
HBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBaseHBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBase
HBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBase
 
HBaseCon 2012 | Building Mobile Infrastructure with HBase
HBaseCon 2012 | Building Mobile Infrastructure with HBaseHBaseCon 2012 | Building Mobile Infrastructure with HBase
HBaseCon 2012 | Building Mobile Infrastructure with HBase
 
HBaseCon 2013: Apache HBase, Meet Ops. Ops, Meet Apache HBase.
HBaseCon 2013: Apache HBase, Meet Ops. Ops, Meet Apache HBase.HBaseCon 2013: Apache HBase, Meet Ops. Ops, Meet Apache HBase.
HBaseCon 2013: Apache HBase, Meet Ops. Ops, Meet Apache HBase.
 

Similar to HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa

Ruby and Rails for womens
Ruby and Rails for womensRuby and Rails for womens
Ruby and Rails for womenss4nx
 
At Scale With Style (Erlang User Conference 2012)
At Scale With Style (Erlang User Conference 2012)At Scale With Style (Erlang User Conference 2012)
At Scale With Style (Erlang User Conference 2012)Wooga
 
Hadoop Conference Japan 2011 Fall
Hadoop Conference Japan 2011 FallHadoop Conference Japan 2011 Fall
Hadoop Conference Japan 2011 FallRyu Kobayashi
 
Rs on Rails 2011
Rs on Rails 2011Rs on Rails 2011
Rs on Rails 2011Fabio Akita
 
HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.
HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.
HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.Takatoshi Matsuo
 
ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012
ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012
ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012Amazon Web Services
 
Developer's Most Frequent Hadoop Headaches & How to Address Them__HadoopSumm...
Developer's Most Frequent Hadoop Headaches &  How to Address Them__HadoopSumm...Developer's Most Frequent Hadoop Headaches &  How to Address Them__HadoopSumm...
Developer's Most Frequent Hadoop Headaches & How to Address Them__HadoopSumm...Yahoo Developer Network
 
.NET Architects Day - DNAD 2011
.NET Architects Day - DNAD 2011.NET Architects Day - DNAD 2011
.NET Architects Day - DNAD 2011Fabio Akita
 
09 - Fábio Akita - Além do rails
09 - Fábio Akita - Além do rails09 - Fábio Akita - Além do rails
09 - Fábio Akita - Além do railsDNAD
 
Realtime Apache Hadoop at Facebook
Realtime Apache Hadoop at FacebookRealtime Apache Hadoop at Facebook
Realtime Apache Hadoop at Facebookparallellabs
 
Use Cases and Integration Scenarios with SAP Adaptive Computing Virtualization
Use Cases and Integration Scenarios with SAP Adaptive Computing VirtualizationUse Cases and Integration Scenarios with SAP Adaptive Computing Virtualization
Use Cases and Integration Scenarios with SAP Adaptive Computing VirtualizationGunther_01
 
Rails Metal, Rack, and Sinatra
Rails Metal, Rack, and SinatraRails Metal, Rack, and Sinatra
Rails Metal, Rack, and SinatraAdam Wiggins
 
Ram chinta hug-20120922-v1
Ram chinta hug-20120922-v1Ram chinta hug-20120922-v1
Ram chinta hug-20120922-v1Ram Chinta
 
UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Al...
UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Al...UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Al...
UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Al...Alex Gorbachev
 
Session 49 Practical Semantic Sticky Note
Session 49 Practical Semantic Sticky NoteSession 49 Practical Semantic Sticky Note
Session 49 Practical Semantic Sticky NoteISSGC Summer School
 
Hummingbird - Open Source for Small Satellites - GSAW 2012
Hummingbird - Open Source for Small Satellites - GSAW 2012Hummingbird - Open Source for Small Satellites - GSAW 2012
Hummingbird - Open Source for Small Satellites - GSAW 2012Logica_hummingbird
 

Similar to HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa (20)

Ruby and Rails for womens
Ruby and Rails for womensRuby and Rails for womens
Ruby and Rails for womens
 
At Scale With Style (Erlang User Conference 2012)
At Scale With Style (Erlang User Conference 2012)At Scale With Style (Erlang User Conference 2012)
At Scale With Style (Erlang User Conference 2012)
 
At Scale With Style
At Scale With StyleAt Scale With Style
At Scale With Style
 
Hadoop Conference Japan 2011 Fall
Hadoop Conference Japan 2011 FallHadoop Conference Japan 2011 Fall
Hadoop Conference Japan 2011 Fall
 
Rs on Rails 2011
Rs on Rails 2011Rs on Rails 2011
Rs on Rails 2011
 
HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.
HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.
HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.
 
Ruby on rails
Ruby on railsRuby on rails
Ruby on rails
 
ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012
ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012
ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012
 
Developer's Most Frequent Hadoop Headaches & How to Address Them__HadoopSumm...
Developer's Most Frequent Hadoop Headaches &  How to Address Them__HadoopSumm...Developer's Most Frequent Hadoop Headaches &  How to Address Them__HadoopSumm...
Developer's Most Frequent Hadoop Headaches & How to Address Them__HadoopSumm...
 
XS Japan 2008 Services English
XS Japan 2008 Services EnglishXS Japan 2008 Services English
XS Japan 2008 Services English
 
.NET Architects Day - DNAD 2011
.NET Architects Day - DNAD 2011.NET Architects Day - DNAD 2011
.NET Architects Day - DNAD 2011
 
09 - Fábio Akita - Além do rails
09 - Fábio Akita - Além do rails09 - Fábio Akita - Além do rails
09 - Fábio Akita - Além do rails
 
Realtime Apache Hadoop at Facebook
Realtime Apache Hadoop at FacebookRealtime Apache Hadoop at Facebook
Realtime Apache Hadoop at Facebook
 
Use Cases and Integration Scenarios with SAP Adaptive Computing Virtualization
Use Cases and Integration Scenarios with SAP Adaptive Computing VirtualizationUse Cases and Integration Scenarios with SAP Adaptive Computing Virtualization
Use Cases and Integration Scenarios with SAP Adaptive Computing Virtualization
 
Rails Metal, Rack, and Sinatra
Rails Metal, Rack, and SinatraRails Metal, Rack, and Sinatra
Rails Metal, Rack, and Sinatra
 
20070329 Tech Study
20070329 Tech Study 20070329 Tech Study
20070329 Tech Study
 
Ram chinta hug-20120922-v1
Ram chinta hug-20120922-v1Ram chinta hug-20120922-v1
Ram chinta hug-20120922-v1
 
UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Al...
UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Al...UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Al...
UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Al...
 
Session 49 Practical Semantic Sticky Note
Session 49 Practical Semantic Sticky NoteSession 49 Practical Semantic Sticky Note
Session 49 Practical Semantic Sticky Note
 
Hummingbird - Open Source for Small Satellites - GSAW 2012
Hummingbird - Open Source for Small Satellites - GSAW 2012Hummingbird - Open Source for Small Satellites - GSAW 2012
Hummingbird - Open Source for Small Satellites - GSAW 2012
 

More from Cloudera, Inc.

Partner Briefing_January 25 (FINAL).pptx
Partner Briefing_January 25 (FINAL).pptxPartner Briefing_January 25 (FINAL).pptx
Partner Briefing_January 25 (FINAL).pptxCloudera, Inc.
 
Cloudera Data Impact Awards 2021 - Finalists
Cloudera Data Impact Awards 2021 - Finalists Cloudera Data Impact Awards 2021 - Finalists
Cloudera Data Impact Awards 2021 - Finalists Cloudera, Inc.
 
2020 Cloudera Data Impact Awards Finalists
2020 Cloudera Data Impact Awards Finalists2020 Cloudera Data Impact Awards Finalists
2020 Cloudera Data Impact Awards FinalistsCloudera, Inc.
 
Edc event vienna presentation 1 oct 2019
Edc event vienna presentation 1 oct 2019Edc event vienna presentation 1 oct 2019
Edc event vienna presentation 1 oct 2019Cloudera, Inc.
 
Machine Learning with Limited Labeled Data 4/3/19
Machine Learning with Limited Labeled Data 4/3/19Machine Learning with Limited Labeled Data 4/3/19
Machine Learning with Limited Labeled Data 4/3/19Cloudera, Inc.
 
Data Driven With the Cloudera Modern Data Warehouse 3.19.19
Data Driven With the Cloudera Modern Data Warehouse 3.19.19Data Driven With the Cloudera Modern Data Warehouse 3.19.19
Data Driven With the Cloudera Modern Data Warehouse 3.19.19Cloudera, Inc.
 
Introducing Cloudera DataFlow (CDF) 2.13.19
Introducing Cloudera DataFlow (CDF) 2.13.19Introducing Cloudera DataFlow (CDF) 2.13.19
Introducing Cloudera DataFlow (CDF) 2.13.19Cloudera, Inc.
 
Introducing Cloudera Data Science Workbench for HDP 2.12.19
Introducing Cloudera Data Science Workbench for HDP 2.12.19Introducing Cloudera Data Science Workbench for HDP 2.12.19
Introducing Cloudera Data Science Workbench for HDP 2.12.19Cloudera, Inc.
 
Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19
Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19
Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19Cloudera, Inc.
 
Leveraging the cloud for analytics and machine learning 1.29.19
Leveraging the cloud for analytics and machine learning 1.29.19Leveraging the cloud for analytics and machine learning 1.29.19
Leveraging the cloud for analytics and machine learning 1.29.19Cloudera, Inc.
 
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19Cloudera, Inc.
 
Leveraging the Cloud for Big Data Analytics 12.11.18
Leveraging the Cloud for Big Data Analytics 12.11.18Leveraging the Cloud for Big Data Analytics 12.11.18
Leveraging the Cloud for Big Data Analytics 12.11.18Cloudera, Inc.
 
Modern Data Warehouse Fundamentals Part 3
Modern Data Warehouse Fundamentals Part 3Modern Data Warehouse Fundamentals Part 3
Modern Data Warehouse Fundamentals Part 3Cloudera, Inc.
 
Modern Data Warehouse Fundamentals Part 2
Modern Data Warehouse Fundamentals Part 2Modern Data Warehouse Fundamentals Part 2
Modern Data Warehouse Fundamentals Part 2Cloudera, Inc.
 
Modern Data Warehouse Fundamentals Part 1
Modern Data Warehouse Fundamentals Part 1Modern Data Warehouse Fundamentals Part 1
Modern Data Warehouse Fundamentals Part 1Cloudera, Inc.
 
Extending Cloudera SDX beyond the Platform
Extending Cloudera SDX beyond the PlatformExtending Cloudera SDX beyond the Platform
Extending Cloudera SDX beyond the PlatformCloudera, Inc.
 
Federated Learning: ML with Privacy on the Edge 11.15.18
Federated Learning: ML with Privacy on the Edge 11.15.18Federated Learning: ML with Privacy on the Edge 11.15.18
Federated Learning: ML with Privacy on the Edge 11.15.18Cloudera, Inc.
 
Analyst Webinar: Doing a 180 on Customer 360
Analyst Webinar: Doing a 180 on Customer 360Analyst Webinar: Doing a 180 on Customer 360
Analyst Webinar: Doing a 180 on Customer 360Cloudera, Inc.
 
Build a modern platform for anti-money laundering 9.19.18
Build a modern platform for anti-money laundering 9.19.18Build a modern platform for anti-money laundering 9.19.18
Build a modern platform for anti-money laundering 9.19.18Cloudera, Inc.
 
Introducing the data science sandbox as a service 8.30.18
Introducing the data science sandbox as a service 8.30.18Introducing the data science sandbox as a service 8.30.18
Introducing the data science sandbox as a service 8.30.18Cloudera, Inc.
 

More from Cloudera, Inc. (20)

Partner Briefing_January 25 (FINAL).pptx
Partner Briefing_January 25 (FINAL).pptxPartner Briefing_January 25 (FINAL).pptx
Partner Briefing_January 25 (FINAL).pptx
 
Cloudera Data Impact Awards 2021 - Finalists
Cloudera Data Impact Awards 2021 - Finalists Cloudera Data Impact Awards 2021 - Finalists
Cloudera Data Impact Awards 2021 - Finalists
 
2020 Cloudera Data Impact Awards Finalists
2020 Cloudera Data Impact Awards Finalists2020 Cloudera Data Impact Awards Finalists
2020 Cloudera Data Impact Awards Finalists
 
Edc event vienna presentation 1 oct 2019
Edc event vienna presentation 1 oct 2019Edc event vienna presentation 1 oct 2019
Edc event vienna presentation 1 oct 2019
 
Machine Learning with Limited Labeled Data 4/3/19
Machine Learning with Limited Labeled Data 4/3/19Machine Learning with Limited Labeled Data 4/3/19
Machine Learning with Limited Labeled Data 4/3/19
 
Data Driven With the Cloudera Modern Data Warehouse 3.19.19
Data Driven With the Cloudera Modern Data Warehouse 3.19.19Data Driven With the Cloudera Modern Data Warehouse 3.19.19
Data Driven With the Cloudera Modern Data Warehouse 3.19.19
 
Introducing Cloudera DataFlow (CDF) 2.13.19
Introducing Cloudera DataFlow (CDF) 2.13.19Introducing Cloudera DataFlow (CDF) 2.13.19
Introducing Cloudera DataFlow (CDF) 2.13.19
 
Introducing Cloudera Data Science Workbench for HDP 2.12.19
Introducing Cloudera Data Science Workbench for HDP 2.12.19Introducing Cloudera Data Science Workbench for HDP 2.12.19
Introducing Cloudera Data Science Workbench for HDP 2.12.19
 
Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19
Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19
Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19
 
Leveraging the cloud for analytics and machine learning 1.29.19
Leveraging the cloud for analytics and machine learning 1.29.19Leveraging the cloud for analytics and machine learning 1.29.19
Leveraging the cloud for analytics and machine learning 1.29.19
 
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19
 
Leveraging the Cloud for Big Data Analytics 12.11.18
Leveraging the Cloud for Big Data Analytics 12.11.18Leveraging the Cloud for Big Data Analytics 12.11.18
Leveraging the Cloud for Big Data Analytics 12.11.18
 
Modern Data Warehouse Fundamentals Part 3
Modern Data Warehouse Fundamentals Part 3Modern Data Warehouse Fundamentals Part 3
Modern Data Warehouse Fundamentals Part 3
 
Modern Data Warehouse Fundamentals Part 2
Modern Data Warehouse Fundamentals Part 2Modern Data Warehouse Fundamentals Part 2
Modern Data Warehouse Fundamentals Part 2
 
Modern Data Warehouse Fundamentals Part 1
Modern Data Warehouse Fundamentals Part 1Modern Data Warehouse Fundamentals Part 1
Modern Data Warehouse Fundamentals Part 1
 
Extending Cloudera SDX beyond the Platform
Extending Cloudera SDX beyond the PlatformExtending Cloudera SDX beyond the Platform
Extending Cloudera SDX beyond the Platform
 
Federated Learning: ML with Privacy on the Edge 11.15.18
Federated Learning: ML with Privacy on the Edge 11.15.18Federated Learning: ML with Privacy on the Edge 11.15.18
Federated Learning: ML with Privacy on the Edge 11.15.18
 
Analyst Webinar: Doing a 180 on Customer 360
Analyst Webinar: Doing a 180 on Customer 360Analyst Webinar: Doing a 180 on Customer 360
Analyst Webinar: Doing a 180 on Customer 360
 
Build a modern platform for anti-money laundering 9.19.18
Build a modern platform for anti-money laundering 9.19.18Build a modern platform for anti-money laundering 9.19.18
Build a modern platform for anti-money laundering 9.19.18
 
Introducing the data science sandbox as a service 8.30.18
Introducing the data science sandbox as a service 8.30.18Introducing the data science sandbox as a service 8.30.18
Introducing the data science sandbox as a service 8.30.18
 

Recently uploaded

ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...
ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...
ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...SearchNorwich
 
Act Like an Owner, Challenge Like a VC by former CPO, Tripadvisor
Act Like an Owner,  Challenge Like a VC by former CPO, TripadvisorAct Like an Owner,  Challenge Like a VC by former CPO, Tripadvisor
Act Like an Owner, Challenge Like a VC by former CPO, TripadvisorProduct School
 
How to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanHow to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanDatabarracks
 
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlueVM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlueShapeBlue
 
iOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostingeriOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostingerssuser9354ce
 
Launching New Products In Companies Where It Matters Most by Product Director...
Launching New Products In Companies Where It Matters Most by Product Director...Launching New Products In Companies Where It Matters Most by Product Director...
Launching New Products In Companies Where It Matters Most by Product Director...Product School
 
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxThe Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxNeo4j
 
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptxGraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptxNeo4j
 
National Institute of Standards and Technology (NIST) Cybersecurity Framework...
National Institute of Standards and Technology (NIST) Cybersecurity Framework...National Institute of Standards and Technology (NIST) Cybersecurity Framework...
National Institute of Standards and Technology (NIST) Cybersecurity Framework...MichaelBenis1
 
New ThousandEyes Product Features and Release Highlights: February 2024
New ThousandEyes Product Features and Release Highlights: February 2024New ThousandEyes Product Features and Release Highlights: February 2024
New ThousandEyes Product Features and Release Highlights: February 2024ThousandEyes
 
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueCloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueShapeBlue
 
Large Language Models and Applications in Healthcare
Large Language Models and Applications in HealthcareLarge Language Models and Applications in Healthcare
Large Language Models and Applications in HealthcareAsma Ben Abacha
 
Transcript: Trending now: Book subjects on the move in the Canadian market - ...
Transcript: Trending now: Book subjects on the move in the Canadian market - ...Transcript: Trending now: Book subjects on the move in the Canadian market - ...
Transcript: Trending now: Book subjects on the move in the Canadian market - ...BookNet Canada
 
Artificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeArtificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeJosh Gellers
 
TrustArc Webinar - TrustArc's Latest AI Innovations
TrustArc Webinar - TrustArc's Latest AI InnovationsTrustArc Webinar - TrustArc's Latest AI Innovations
TrustArc Webinar - TrustArc's Latest AI InnovationsTrustArc
 
PrismCRM-RealEstate-SalesCRM_byCode5Company
PrismCRM-RealEstate-SalesCRM_byCode5CompanyPrismCRM-RealEstate-SalesCRM_byCode5Company
PrismCRM-RealEstate-SalesCRM_byCode5CompanyMustafa Kuğu
 
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)Jay Zhao
 
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoRevolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoProduct School
 
My Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceMy Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceVijayananda Mohire
 

Recently uploaded (20)

ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...
ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...
ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...
 
Act Like an Owner, Challenge Like a VC by former CPO, Tripadvisor
Act Like an Owner,  Challenge Like a VC by former CPO, TripadvisorAct Like an Owner,  Challenge Like a VC by former CPO, Tripadvisor
Act Like an Owner, Challenge Like a VC by former CPO, Tripadvisor
 
How to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanHow to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response Plan
 
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlueVM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
 
iOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostingeriOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostinger
 
In sharing we trust. Taking advantage of a diverse consortium to build a tran...
In sharing we trust. Taking advantage of a diverse consortium to build a tran...In sharing we trust. Taking advantage of a diverse consortium to build a tran...
In sharing we trust. Taking advantage of a diverse consortium to build a tran...
 
Launching New Products In Companies Where It Matters Most by Product Director...
Launching New Products In Companies Where It Matters Most by Product Director...Launching New Products In Companies Where It Matters Most by Product Director...
Launching New Products In Companies Where It Matters Most by Product Director...
 
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxThe Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
 
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptxGraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
 
National Institute of Standards and Technology (NIST) Cybersecurity Framework...
National Institute of Standards and Technology (NIST) Cybersecurity Framework...National Institute of Standards and Technology (NIST) Cybersecurity Framework...
National Institute of Standards and Technology (NIST) Cybersecurity Framework...
 
New ThousandEyes Product Features and Release Highlights: February 2024
New ThousandEyes Product Features and Release Highlights: February 2024New ThousandEyes Product Features and Release Highlights: February 2024
New ThousandEyes Product Features and Release Highlights: February 2024
 
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueCloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
 
Large Language Models and Applications in Healthcare
Large Language Models and Applications in HealthcareLarge Language Models and Applications in Healthcare
Large Language Models and Applications in Healthcare
 
Transcript: Trending now: Book subjects on the move in the Canadian market - ...
Transcript: Trending now: Book subjects on the move in the Canadian market - ...Transcript: Trending now: Book subjects on the move in the Canadian market - ...
Transcript: Trending now: Book subjects on the move in the Canadian market - ...
 
Artificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeArtificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human Justice
 
TrustArc Webinar - TrustArc's Latest AI Innovations
TrustArc Webinar - TrustArc's Latest AI InnovationsTrustArc Webinar - TrustArc's Latest AI Innovations
TrustArc Webinar - TrustArc's Latest AI Innovations
 
PrismCRM-RealEstate-SalesCRM_byCode5Company
PrismCRM-RealEstate-SalesCRM_byCode5CompanyPrismCRM-RealEstate-SalesCRM_byCode5Company
PrismCRM-RealEstate-SalesCRM_byCode5Company
 
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
 
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoRevolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
 
My Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceMy Journey towards Artificial Intelligence
My Journey towards Artificial Intelligence
 

HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa

  • 1. Orchestrating Clusters with Ironfan and Chef Robert J. Berger - CTO Runa, Inc. rberger@runa.com
  • 2. Hassles of Big Data Stack Deployments • Lots of Moving Parts • Hadoop/HBase just one sub-system • Heterogeneous Tech • Monitoring & Metrics Everywhere • Details obscure the big picture • Need repeatability & variations on themes
  • 3. The Forest for the Trees
  • 4. Forest Dash App Monitor board Servers Elastic Hadoop AMQP Search M/R Session HBase MySQL Store
  • 5. MySQL Slave Slave Slave Slave Master GDash Statsd Sensu Slave Logstash Slave Ganglia Slave Monitoring Graphite Slave Web Slave Regionsrvr Slave Regionsrvr Slave Hadoop M/R Slave Regionsrvr Trees Sec Master Regionsrvr Master Regionsrvr Regionsrvr Clj App Regionsrvr Clj App Regionsrvr Clj App Regionsrvr App Servers Web Rabbit Regionsrvr Web Rabbit AMQP Regionsrvr ELB HB Master HBase HB Master ZooKeeper ZooKeeper ZooKeeper ES Server ES Server Rails App ES Server Search Elastic Dashboard Rails App ES Server Web Redis Redis Session Store Redis
  • 6. Leaves Dashboard App Servers Monitoring Nginx MySQL Client Elastic Load Balancer MySQL Client Nginx Ganglia Server Reverse Proxy Upstart Config Nginx Upstart Config Reverse Proxy Ganglia Web Unicorn Logstash Client Reverse Proxy Logstash Client Java Statsd Server Rails Swarmiji Leiningen Graphite Server Dashboard App Java Jark Grpahite Web Java Leiningen Postfix Python Postfix Jark Cron jobs Logstash Cron jobs Clojure Apps Sensu Server MySQL Client Sensu client HBase Client Sensu Web Upstart Config Sensu plugins Postfix Sensu Client Logstash Client Cron jobs Sensu Plugins Sensu client Sensu plugins Elastic Search AMQP Hadoop M/R Elastic Search Server RabbitMQ Namenode Java RabbitMQ Plugins Secondary Namenode Cron jobs Cluster Config Tasktracker Sensu client Erlang Jobtrackers Sensu plugins Cron jobs Bootstrap Namenmode Upstart Config Sensu client Java Sensu plugins JMX Upstart Config Cron jobs Sensu client Sensu plugins Ganglia Client Upstart Config Session Store HBase MySQL Redis Namenode Zookeeper MySQL Master Cron jobs Secondary Namenode HBase Master MySQL Slaves Sensu client Tasktracker Regionserver Cluster Setup Sensu plugins Jobtrackers Cron jobs Upstart Config Datanodes Sensu client Bootstrap Namenmode Sensu plugins Java Upstart Config JMX Cron jobs Sensu client Sensu plugins Ganglia Client Upstart Config
  • 7. <configuration> Molecules <property> <name>hbase.rootdir</name> <value>hdfs://ip-10-17-57-58.ec2.internal:8020/hadoop/hbase</value> <description>The directory shared by region servers. Should be fully-qualified to include the filesystem to use. E.g: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR </description> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> <description>The mode the cluster will be in. Possible values are false: standalone and pseudo-distributed setups with managed Zookeeper true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh) </description> </property> <property> <name>hbase.zookeeper.quorum</name> <value>master0-cluster0.runa.com,regionserver0- cluster0.runa.com,regionserver1-cluster0.runa.com</value> <description>Comma separated list of servers in the ZooKeeper Quorum. For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com". By default this is set to localhost for local and pseudo-distributed modes of operation. For a fully-distributed setup, this should be set to a full list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in hbase-env.sh this is the list of servers which we will start/stop ZooKeeper on. </description> </property>
  • 8. Config Management: Leaves & Molecules • Chef, Puppet, Cfengine • Much better than shell scripts or cli jocks • Infrastructure as code • Still No Forest Perspective
  • 9. Ironfan: Forest, Trees, Leaves and Molecules • Builds on top of Chef • Cluster Description in Single File • Your System Diagram Come to Life • Components announce capabilities • Service Discovery automates interconnects • Knife (CLI) extension controls cluster and component life-cycles
  • 10. Basic Chef Community + Own Chef Server Cookbooks Search Auth & ACLs Nodes Roles / Cookbooks Data Bags Attributes N N N Chef Knife: Upload Cookboks to Chef Server N Launch Instances / Bootstrap Servers VMs and/or Servers Running Chef-Client Chef Development Host
  • 11. Ironfan Pantry + Chef + Ironfan Community + Own Chef Server Cookbooks Search / Search Auth & Discovery ACLs Nodes Nodes Attributes Roles / Cookbooks Data Bags Attributes Discovery N N N Chef Knife + Ironfan Gem: Launch / Bootstrap / Manage N Whole Clusters, All Facets or Specific Instances VMs and/or Servers Running Chef-Client Chef Development Host
  • 12. Ironfan Components • Ironfan Gem: • Knife Plugins to orchestrate clusters • Logic to sync Chef Server & Cloud[s] • Silverware: Coordinate Discovery of Services • Ironfan-Homebase: Ironfan tuned Chef-Repo • Ironfan-Pantry: Cookbooks tuned for Clusters • Ironfan-CI: Testing of Ironfan clusters and Cookbooks
  • 13. Cluster Config: Forest View ClusterChef.cluster 'base0-cluster0' do Global setup_role_implications cloud :ec2 do Cloud & Cluster region 'us-east-1' Configs availability_zones ['us-east-1b'] backing 'ebs' image_name 'natty' security_group(cluster_root) do authorize_port_range(22) end end role "base_role" role "chef_client" role "base0-cluster0" role "production" role "runastack" role "ebs_volumes_attach" Roles & Recipes role "ebs_volumes_mount" are the “Leaves” Facets are facet 'master' do the “Trees” instances 1 cloud.image_id 'ami-93c31afa' cloud.flavor "cc1.4xlarge" role "big_package" role "hadoop_master" role "hbase_master" recipe "cluster_chef::cluster_webfront" recipe "hbase::utils" recipe "route53::runa" role "monitored_client" end
  • 14. Global Cloud & Recipe Confgs ClusterChef.cluster 'base0-cluster0' do setup_role_implications Cluster Name cloud :ec2 do Cloud region 'us-east-1' Configs availability_zones ['us-east-1b'] backing 'ebs' Configure image_name 'natty' Security security_group(cluster_root) do Group authorize_port_range(22) end end Shared role "base_role" Roles role "chef_client" role "base0-cluster0" role "production" role "runastack" role "ebs_volumes_attach" role "ebs_volumes_mount"
  • 15. Facet Name Facets add Specifics facet 'master' do Number of Cloud instances 1 Copies Overrides cloud.image_id 'ami-93c31afa' cloud.flavor "cc1.4xlarge" role "hadoop_master" Facet role "hbase_master" Roles & recipe "cluster_chef::cluster_webfront" recipe "hbase::utils" Recipes recipe "route53::runa" role "monitored_client" Facet Name end Number of Cloud facet 'regionserver' do instances 7 Copies Overrides cloud.image_id 'ami-93c31afa' cloud.flavor "cc1.4xlarge" role "hadoop_slave" role "hbase_regionserver" Facet Make one recipe "hbase::utils" Roles & instance recipe "route53::runa" role "monitored_client" Recipes special server 0 do role "zookeeper_server" end end
  • 16. Facets Composed of Components • Components are Services • Nginx, MySQL server, Zookeeper, HBMaster, Namenode, etc. • Chef Cookbooks manage components • Ironfan Silverware for service discovery • Auto-Connects components together
  • 17. Silverware Service Discovery • Recipe that creates a service , announces it announce(:hadoop, :namenode) • Recipe that requires a service , discovers it hbase_config = Mash.new({ :namenode_fqdn => discover(:hadoop, :namenode ).private_hostname), :jobtracker_addr => discover(:hadoop, :jobtracker).private_ip), :zookeeper_addrs => discover_all(:zookeeper, :server).map(&:private_ip).sort, :ganglia => discover(:ganglia, :server), :ganglia_addr => discover(:ganglia, :server).private_hostname), :private_ip => private_ip_of(node) })
  • 18. Aspects enable Zeroconf Amenities • A log aspect would enable the following amenities • logrotated to manage its logs • flume to archive logs to a location • A port aspect would enable • Configuration of firewall • Monitoring of port uptime & latency • Remote checks that firewalled ports do NOT respond
  • 19. Knife Cluster: Lifecycle Management • A Plugin for Opscode Chef Knife • Deployment & Lifecycle Operations: • launch, bootstrap, kill, start, stop • Access and Chef Operations • ssh, kick, proxy • Utilities • show, sync
  • 20. Knife Launch Cluster, Facet or Instance[s] • Launch cluster launchcluster knife all the nodes in a base0-master0 • Launchcluster launch base0-master0 facet 0 knife just a single instance of a single master • Launchclusterinstances base0-master0 regionserver knife all the launch of a facet
  • 21. Stop/Start Cluster, Facet or Instance[s] • Stop whole cluster base0-master0 knife cluster stop • Stop a cluster stop base0-master0 master 0 knife single instance of a single facet • Stop allcluster stop abase0-master0 regionserver knife instances of facet
  • 22. Same Knife Command to Launch Vagrant[s] • Can use the same cluster configurations and knife command to launch Vagrants knife cluster vagrant up base0-master0 • Still Experimental
  • 23. Ironfan-CI • Jenkins based Continuous Integration of Clusters • Still Experimental • Uses Discovery to automate baseline test creation • Leverages Vagrant to create clean test environments
  • 24. References • Basic Chef Stuff: http://wiki.opscode.com/display/chef/Home • Ironfan Screencast: http://vimeo.com/37279372 • Ironfan Wiki for the most complete info : https://github.com/infochimps-labs/ironfan/wiki • The Forest for the Trees Photo - Ame Otoko http://www.flickr.com/photos/ameotoko/5383225925/