What Big Data Folks Need to Know About DevOps

Developer, SysAdmin, Hacker,Community ManagerMany biz & dev environmentsOpscode: Training, Services &Evangelism Copyright © 2011 Opscode, Inc - All Rights Reserved http://www.ﬂickr.com/photos/anotherphotograph/2100904507/sizes/o/ 2

option :json_attribs, :short => "-j JSON_ATTRIBS", :long => "--json-attributes JSON_ATTRIBS", :description => "Load attributes from a JSON ﬁle orURL", :proc => nil option :node_name, :short => "-N NODE_NAME", :long => "--node-name NODE_NAME", :description => "The node name for this client", Defaults are sane, but :proc => nil easily changed Copyright © 2011 Opscode, Inc - All Rights Reserved 16

Nodes have Attributes{ "kernel": { Kernel info! "machine": "x86_64", "name": "Darwin", "os": "Darwin", "version": "Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386", "release": "10.4.0" }, "platform_version": "10.6.4", "platform": "mac_os_x", "platform_build": "10F569", "domain": "local", Platform info! "os": "darwin", "current_user": "jtimberman", "ohai_time": 1278602661.60043, "os_version": "10.4.0", "uptime": "18 days 17 hours 49 minutes 18 seconds", "ipaddress": "10.13.37.116", "hostname": "cider", "fqdn": "cider.local", "uptime_seconds": 1619358 Hostname and IP!} Copyright © 2011 Opscode, Inc - All Rights Reserved 26

Resources...Declare a description of the state a part of the node should be in http://www.ﬂickr.com/photos/xiaming/382205902/sizes/l/

Resources...Declare a description of the state a part of the node should be in package "apache2" do version "2.2.11-2ubuntu2.6" action :install end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end http://www.ﬂickr.com/photos/xiaming/382205902/sizes/l/

Resources... Declare a description of the state a part of the node should be in‣ Have a type package "apache2" do version "2.2.11-2ubuntu2.6" action :install end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end http://www.ﬂickr.com/photos/xiaming/382205902/sizes/l/

Resources... Declare a description of the state a part of the node should be in‣ Have a type package "apache2" do version "2.2.11-2ubuntu2.6" action :install‣ Have a name end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end http://www.ﬂickr.com/photos/xiaming/382205902/sizes/l/

Resources... Declare a description of the state a part of the node should be in‣ Have a type package "apache2" do version "2.2.11-2ubuntu2.6" action :install‣ Have a name end template "/etc/apache2/apache2.conf" do‣ Have parameters source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end http://www.ﬂickr.com/photos/xiaming/382205902/sizes/l/

Resources... Declare a description of the state a part of the node should be in‣ Have a type package "apache2" do version "2.2.11-2ubuntu2.6" action :install‣ Have a name end template "/etc/apache2/apache2.conf" do‣ Have parameters source "apache2.conf.erb" owner "root"‣ Take action to put the group "root" mode 0644 resource in the action :create declared state end http://www.ﬂickr.com/photos/xiaming/382205902/sizes/l/

Providers...Know how to actually perform the actions speciﬁed by a resource. http://www.ﬂickr.com/photos/affableslinky/562950216/

Providers...Know how to actually perform the actions speciﬁed by a resource. Apt, Yum, Rubygems,Multiple providers Portage, Macports,per resource type. FreeBSD Ports, etc. http://www.ﬂickr.com/photos/affableslinky/562950216/

http://www.ﬂickr.com/photos/acurbelo/2628837104/sizes/o/

Resources http://www.ﬂickr.com/photos/acurbelo/2628837104/sizes/o/

ResourcesPlatform http://www.ﬂickr.com/photos/acurbelo/2628837104/sizes/o/

ResourcesPlatformProvider http://www.ﬂickr.com/photos/acurbelo/2628837104/sizes/o/

Recipes...Apply resources in the order they are speciﬁed http://www.ﬂickr.com/photos/roadsidepictures/2478953342/sizes/o/

Recipes...Apply resources in the order they are speciﬁed package "apache2" do version "2.2.11-2ubuntu2.6" action :install end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end http://www.ﬂickr.com/photos/roadsidepictures/2478953342/sizes/o/

Recipes... Apply resources in the order they are speciﬁed package "apache2" do version "2.2.11-2ubuntu2.6" action :install 1‣ Evaluates resources in end the order they appear template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create 2 end http://www.ﬂickr.com/photos/roadsidepictures/2478953342/sizes/o/

Recipes... Apply resources in the order they are speciﬁed‣ Evaluates resources in [ the order they appear "package[apache2]", "template[/etc/apache2/apache2.conf]"‣ Adds each resource to ] the Resource Collection http://www.ﬂickr.com/photos/roadsidepictures/2478953342/sizes/o/

A user data bag item...% knife data bag show users jtimberman{ "comment": "Joshua Timberman", "groups": "sysadmin", "ssh_keys": "ssh-rsa SUPERSEKRATS jtimberman@cider", "ﬁles": { ".zshrc": { "mode": "0644", "source": "dot-zshrc" }, ".vimrc": { "mode": "0644", "source": "dot-vimrc" } }, "id": "jtimberman", "uid": 7004, "shell": "/usr/bin/zsh", "openid": "http://jtimberman.myopenid.com/"} Copyright © 2011 Opscode, Inc - All Rights Reserved

sysadmins = search(:users, groups:sysadmin)sysadminss.each do |u| user u[id] do uid u[id] shell u[shell] comment u[comment] supports :manage_home => true home "/home/#{u[id]}" end directory "/home/#{u[id]}/.ssh" do owner u[id] group u[id] mode 0700 end template "/home/#{u[id]}/.ssh/authorized_keys" do source "authorized_keys.erb" owner u[id] group u[id] mode 0600 variables :ssh_keys => u[ssh_keys] endend Copyright © 2011 Opscode, Inc - All Rights Reserved 43

Cluster Chef’s FacetsRoleshadoop_master hadoop_namenode hadoop_secondarynamenod e hadoop_jobtrackerhadoop_worker hadoop_datanode hadoop_tasktracker Copyright © 2011 Opscode, Inc - All Rights Reserved 53

Is this thing on?$ knife clusterAvailable cluster subcommands: (for details, knife SUB-COMMAND --help)** CLUSTER COMMANDS **knife cluster launch CLUSTER_NAME FACET_NAME (options)knife cluster show CLUSTER_NAME FACET_NAME (options)knife cluster bootstrap CLUSTER_NAME FACET_NAME SERVER_FQDN (options) Copyright © 2011 Opscode, Inc - All Rights Reserved 55

knife ec2 server createCreates EC2 instance via APIRetrieves local configurationSSH to instance ‣ Cluster Chef extends this security groups picks the AMI builds the number of speciﬁed nodes ‣ Writes chef configuration and authentication ‣ Installs Ruby and Chef ‣ Runs Chef with specified run list Copyright © 2011 Opscode, Inc - All Rights Reserved 57

Still a bit of tweaking$ knife ssh "role:demohadoop_master" "sudo service hadoop-0.20-datanodestop; sudo service hadoop-0.20-namenode stop; sudo service hadoop-0.20-tasktracker stop; sudo service hadoop-0.20-jobtracker stop; sudo servicehadoop-0.20-secondarynamenode stop; sudo -u hdfs hadoop fs -chown -Rhbase:hbase /hadoop/hbase; sudo chef-client" -x ubuntu -aec2.public_hostname -i ~/.chef/keypairs/demohadoop.pem Copyright © 2011 Opscode, Inc - All Rights Reserved 58

Is it really on?‣ Configure your network settings to use a SOCKS proxy‣ http://ec2-public-ipaddress.compute-1.amazonaws.com‣ copy & paste the SSH command‣ Profit! Copyright © 2011 Opscode, Inc - All Rights Reserved 60

Resources/Questionswww.opscode.com/chefIRC and Mailing lists‣ irc.freenode.net #chef‣ lists.opscode.comTwitter:‣ @opscode, #opschef‣ @mattrayQuestions? Copyright © 2011 Opscode, Inc - All Rights Reserved 62

http://www.opscode.com	344
http://twitter.com	4
http://www.onlydoo.com	3
http://paper.li	2
https://twitter.com	1
http://translate.googleusercontent.com	1
http://www.linkedin.com	1

What Big Data Folks Need to Know About DevOps

by Matt Ray on Jan 31, 2011

Accessibility

Categories

Upload Details

Usage Rights

7 Embeds 356

Statistics

What Big Data Folks Need to Know About DevOps Presentation Transcript