Orchestrating Clusterswith Ironfan and Chef    Robert J. Berger - CTO Runa, Inc.          rberger@runa.com
Hassles of Big Data Stack      Deployments• Lots of Moving Parts •   Hadoop/HBase just one sub-system •   Heterogeneous Te...
The Forest for the Trees
ForestDash        App                     Monitorboard      ServersElastic              Hadoop           AMQPSearch       ...
MySQL                                                                                       Slave                         ...
LeavesDashboard                        App Servers                                   Monitoring Nginx         MySQL Client...
Molecules<configuration>  <property>    <name>hbase.rootdir</name>    <value>hdfs://ip-10-19-65-58.ec2.internal:8020/hadoo...
Config Management:   Leaves & Molecules• Chef, Puppet, Cfengine• Much better than shell scripts or cli jocks• Infrastructur...
Ironfan: Forest, Trees,    Leaves and Molecules•   Open Source Project from Infochimps•   Builds on top of Chef•   Cluster...
Basic ChefCommunity + Own                   Chef Server  Cookbooks           Search                     Auth &            ...
Chef + IronfanCommunity + Own                   Chef Server  Cookbooks           Search                     Auth &        ...
Ironfan Pantry +                    Chef + IronfanCommunity + Own                    Chef Server    Cookbooks          Sea...
Ironfan Components• Ironfan Gem:  •   Knife Plugins to orchestrate clusters  •   Logic to sync Chef Server & Cloud[s]• Sil...
Cluster Config: Forest View                  ClusterChef.cluster base0-cluster0 do    Global          setup_role_implicatio...
Global Cloud & Recipe Confgs         ClusterChef.cluster base0-cluster0 do           setup_role_implications              ...
Facet Name             Facets add Specifics             facet master do                                                    ...
Facets Composed of     Components• Components are Services  • Nginx, MySQL server, Zookeeper, HBMaster,    Namenode, etc.•...
Silverware:        Service Discovery•   Announcements enable Service Discovery•   Implemented using Chef Node Attributes  ...
Silverware Service Discovery• Recipe that creates a service , announces itannounce(:hadoop, :namenode)• Recipe that requir...
Silverware: Announce Capabilities• Goes beyond basic component discovery• Capabilities • Serve webpages, execute script, s...
Aspects enable Zeroconf       Amenities•   A log aspect would enable the following amenities    •   logrotated    to manag...
Knife Cluster: Lifecycle Management• A Plugin for Opscode Chef Knife• Deployment & Lifecycle Operations:  • launch, bootst...
Knife Launch Cluster,       Facet or Instance[s]• Launch cluster launchcluster  knife         all the nodes in a          ...
Stop/Start Cluster, Facet        or Instance[s]• Stop whole cluster base0-master0  knife cluster stop• Stop a cluster stop...
Same Knife Command to   Launch Vagrant[s]• Can use the same cluster configurations  and knife command to launch Vagrants  k...
Ironfan-CI• Jenkins based Continuous Integration of  Clusters• Still Experimental• Uses Discovery to automate baseline tes...
References•   Basic Chef Stuff:    http://wiki.opscode.com/display/chef/Home•   Ironfan Screencast:    http://vimeo.com/37...
Upcoming SlideShare
Loading in...5
×

Orchestrating HBase Cluster Deployment with Ironfan and Chef

3,220

Published on

HBaseCon 2012 Lightning Talk on how you can represent your complete cluster with one config file and have it deployed to various target environments.

Infochimps’ Ironfan builds on Opscode Chef to allow you to specify and orchestrate all flavors of your cluster’s deployment, monitoring and growth.

Not just the core HBase/HDFS/MapReduce/Hive/Flume, etc. but all the elements including web / app servers, MySQL, Redis, RabbitMQ and whatever other servers needed to implement your service.

These same tools can manage variations for development, staging, R&D as well as the target “rendering” to various Clouds, Bare Metal or even Vagrant VMs.

Published in: Technology
2 Comments
10 Likes
Statistics
Notes
No Downloads
Views
Total Views
3,220
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
40
Comments
2
Likes
10
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Orchestrating HBase Cluster Deployment with Ironfan and Chef

    1. 1. Orchestrating Clusterswith Ironfan and Chef Robert J. Berger - CTO Runa, Inc. rberger@runa.com
    2. 2. Hassles of Big Data Stack Deployments• Lots of Moving Parts • Hadoop/HBase just one sub-system • Heterogeneous Tech • Monitoring & Metrics Everywhere • Details obscure the big picture• Need repeatability & variations on themes
    3. 3. The Forest for the Trees
    4. 4. ForestDash App Monitorboard ServersElastic Hadoop AMQPSearch M/RSession HBase MySQL Store
    5. 5. MySQL Slave Slave Slave Slave Master GDash Statsd Sensu Slave Logstash Slave Ganglia Slave Monitoring Graphite Slave Web Slave Regionsrvr Slave Regionsrvr Slave Hadoop M/R Slave RegionsrvrTrees Sec Master Regionsrvr Master Regionsrvr Regionsrvr Clj App Regionsrvr Clj App Regionsrvr Clj App Regionsrvr App Servers Web Rabbit Regionsrvr Web Rabbit AMQP Regionsrvr ELB HB Master HBase HB Master ZooKeeper ZooKeeper ZooKeeper ES Server ES Server Rails App ES Server Search Elastic Dashboard Rails App ES Server Web Redis Redis Session Store Redis
    6. 6. LeavesDashboard App Servers Monitoring Nginx MySQL Client Elastic Load Balancer MySQL Client Nginx Ganglia Server Reverse Proxy Upstart Config Nginx Upstart Config Reverse Proxy Ganglia Web Unicorn Logstash Client Reverse Proxy Logstash Client Java Statsd Server Rails Swarmiji Leiningen Graphite Server Dashboard App Java Jark Grpahite Web Java Leiningen Postfix Python Postfix Jark Cron jobs Logstash Cron jobs Clojure Apps Sensu Server MySQL Client Sensu client HBase Client Sensu Web Upstart Config Sensu plugins Postfix Sensu Client Logstash Client Cron jobs Sensu Plugins Sensu client Sensu pluginsElastic Search AMQP Hadoop M/R Elastic Search Server RabbitMQ Namenode Java RabbitMQ Plugins Secondary Namenode Cron jobs Cluster Config Tasktracker Sensu client Erlang Jobtrackers Sensu plugins Cron jobs Bootstrap Namenmode Upstart Config Sensu client Java Sensu plugins JMX Upstart Config Cron jobs Sensu client Sensu plugins Ganglia Client Upstart ConfigSession Store HBase MySQLRedis Namenode Zookeeper MySQL MasterCron jobs Secondary Namenode HBase Master MySQL SlavesSensu client Tasktracker Regionserver Cluster SetupSensu plugins Jobtrackers Cron jobsUpstart Config Datanodes Sensu client Bootstrap Namenmode Sensu plugins Java Upstart Config JMX Cron jobs Sensu client Sensu plugins Ganglia Client Upstart Config
    7. 7. Molecules<configuration> <property> <name>hbase.rootdir</name> <value>hdfs://ip-10-19-65-58.ec2.internal:8020/hadoop/hbase</value> <description>The directory shared by region servers. Should be fully-qualified to include the filesystem to use. E.g: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR </description> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> <description>The mode the cluster will be in. Possible values are false: standalone and pseudo-distributed setups with managed Zookeeper true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh) </description> </property> <property> <name>hbase.zookeeper.quorum</name> <value>master.example.com,region0.example.com,region1.example.com</value> <description>Comma separated list of servers in the ZooKeeper Quorum. For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com". By default this is set to localhost for local and pseudo-distributed modes of operation. For a fully-distributed setup, this should be set to a full list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in hbase-env.sh this is the list of servers which we will start/stop ZooKeeper on. </description> </property>
    8. 8. Config Management: Leaves & Molecules• Chef, Puppet, Cfengine• Much better than shell scripts or cli jocks• Infrastructure as code• Still No Forest Perspective
    9. 9. Ironfan: Forest, Trees, Leaves and Molecules• Open Source Project from Infochimps• Builds on top of Chef• Cluster Description in Single File • Your System Diagram Come to Life• Components announce capabilities • Service Discovery automates interconnects• Knife (CLI) extension controls cluster and component life-cycles
    10. 10. Basic ChefCommunity + Own Chef Server Cookbooks Search Auth & ACLs Nodes Roles / Cookbooks Data Bags Attributes N N N Chef Knife: Upload Cookboks to Chef Server N Launch Instances / Bootstrap Servers VMs and/or Servers Running Chef-Client Chef Development Host
    11. 11. Chef + IronfanCommunity + Own Chef Server Cookbooks Search Auth & ACLs Nodes Roles / Cookbooks Data Bags Attributes N N N Chef Knife + Ironfan Gem: Launch / Bootstrap / Manage N Whole Clusters, All Facets or Specific Instances VMs and/or Servers Running Chef-Client Chef Development Host
    12. 12. Ironfan Pantry + Chef + IronfanCommunity + Own Chef Server Cookbooks Search / Search Auth & Discovery ACLs Nodes Nodes Attributes Roles / Cookbooks Data Bags Attributes Discovery N N N Chef Knife + Ironfan Gem: Launch / Bootstrap / Manage N Whole Clusters, All Facets or Specific Instances VMs and/or Servers Running Chef-Client Chef Development Host
    13. 13. Ironfan Components• Ironfan Gem: • Knife Plugins to orchestrate clusters • Logic to sync Chef Server & Cloud[s]• Silverware: Coordinate Discovery of Services• Ironfan-Homebase: Ironfan tuned Chef-Repo• Ironfan-Pantry: Cookbooks tuned for Clusters• Ironfan-CI: Testing of Ironfan clusters and Cookbooks
    14. 14. Cluster Config: Forest View ClusterChef.cluster base0-cluster0 do Global setup_role_implications cloud :ec2 doCloud & Cluster region us-east-1 Configs availability_zones [us-east-1b] backing ebs image_name natty security_group(cluster_root) do authorize_port_range(22) end end role "base_role" role "chef_client" role "base0-cluster0" role "production" role "runastack" role "ebs_volumes_attach" Roles & Recipes role "ebs_volumes_mount" are the “Leaves” Facets are facet master do the “Trees” instances 1 cloud.image_id ami-93c31afa cloud.flavor "cc1.4xlarge" role "big_package" role "hadoop_master" role "hbase_master" recipe "cluster_chef::cluster_webfront" recipe "hbase::utils" recipe "route53::runa" role "monitored_client" end
    15. 15. Global Cloud & Recipe Confgs ClusterChef.cluster base0-cluster0 do setup_role_implications Cluster Name cloud :ec2 doCloud region us-east-1Configs availability_zones [us-east-1b] backing ebs Configure image_name natty Security security_group(cluster_root) do Group authorize_port_range(22) end endShared role "base_role" Roles role "chef_client" role "base0-cluster0" role "production" role "runastack" role "ebs_volumes_attach" role "ebs_volumes_mount"
    16. 16. Facet Name Facets add Specifics facet master do Number of Cloud instances 1 CopiesOverrides cloud.image_id ami-93c31afa cloud.flavor "cc1.4xlarge" role "hadoop_master" Facet role "hbase_master" Roles & recipe "cluster_chef::cluster_webfront" recipe "hbase::utils" Recipes recipe "route53::runa" role "monitored_client"Facet Name end Number of Cloud facet regionserver do instances 7 CopiesOverrides cloud.image_id ami-93c31afa cloud.flavor "cc1.4xlarge" role "hadoop_slave" role "hbase_regionserver" FacetMake one recipe "hbase::utils" Roles &instance recipe "route53::runa" role "monitored_client" Recipes special server 0 do role "zookeeper_server" end end
    17. 17. Facets Composed of Components• Components are Services • Nginx, MySQL server, Zookeeper, HBMaster, Namenode, etc.• Chef Cookbooks manage components• Ironfan Silverware for service discovery • Auto-Connects components together
    18. 18. Silverware: Service Discovery• Announcements enable Service Discovery• Implemented using Chef Node Attributes • DSL for Chef Attributes• Components announce Services & Amenities • Enables zero-conf for things like Logging, Ports, Dashboard sources, etc.• Announcements effectively define a components Contract or API
    19. 19. Silverware Service Discovery• Recipe that creates a service , announces itannounce(:hadoop, :namenode)• Recipe that requires a service , discovers ithbase_config = Mash.new({ :namenode_fqdn => discover(:hadoop, :namenode ).private_hostname), :jobtracker_addr => discover(:hadoop, :jobtracker).private_ip), :zookeeper_addrs =>discover_all(:zookeeper, :server).map(&:private_ip).sort, :ganglia => discover(:ganglia, :server), :ganglia_addr => discover(:ganglia, :server).private_hostname), :private_ip => private_ip_of(node) })
    20. 20. Silverware: Announce Capabilities• Goes beyond basic component discovery• Capabilities • Serve webpages, execute script, send metrics, answer queries• Aspects • Ports, daemons, logs, files, dashboards
    21. 21. Aspects enable Zeroconf Amenities• A log aspect would enable the following amenities • logrotated to manage its logs • flume to archive logs to a location• A port aspect would enable • Configuration of firewall • Monitoring of port uptime & latency • Remote checks that firewalled ports do NOT respond
    22. 22. Knife Cluster: Lifecycle Management• A Plugin for Opscode Chef Knife• Deployment & Lifecycle Operations: • launch, bootstrap, kill, start, stop• Access and Chef Operations • ssh, kick, proxy• Utilities • show, sync
    23. 23. Knife Launch Cluster, Facet or Instance[s]• Launch cluster launchcluster knife all the nodes in a base0-master0• Launchcluster launch base0-master0 facet 0 knife just a single instance of a single master• Launchclusterinstances base0-master0 regionserver knife all the launch of a facet
    24. 24. Stop/Start Cluster, Facet or Instance[s]• Stop whole cluster base0-master0 knife cluster stop• Stop a cluster stop base0-master0 master 0 knife single instance of a single facet• Stop allcluster stop abase0-master0 regionserver knife instances of facet
    25. 25. Same Knife Command to Launch Vagrant[s]• Can use the same cluster configurations and knife command to launch Vagrants knife cluster vagrant up base0-master0• Still Experimental
    26. 26. Ironfan-CI• Jenkins based Continuous Integration of Clusters• Still Experimental• Uses Discovery to automate baseline test creation• Leverages Vagrant to create clean test environments
    27. 27. References• Basic Chef Stuff: http://wiki.opscode.com/display/chef/Home• Ironfan Screencast: http://vimeo.com/37279372• Ironfan Wiki for the most complete info : https://github.com/infochimps-labs/ironfan/wiki• The Forest for the Trees Photo - Ame Otoko http://www.flickr.com/photos/ameotoko/5383225925/
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×