Puppet Dashboard at HEP, Cambridge


Published on

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Puppet Dashboard at HEP, Cambridge

  1. 1. Santanu Das!santanu@hep.phy.cam.ac.uk!Introduction to Puppet Dashboard!HEPSYSMAN, RAL!10th – 11th May, 2012Dept. of Physics!Cavendish Laboratory!
  2. 2. 10/05/2012 2What is Puppet?Puppet is the local SysAdmin  According  to  Puppet  Lab:  “Put  simply,  Puppet  is  a  system  for  automa1ng  system  administra1on  tasks.”    It’s  a  declarave  language  for  expressing  system  configura=on,  a  client  and  server  for  distribu=ng  it,  and  a  library  for  realizing  the  configura=on.    Basically,  it’s  a  configura=on  management  tool,  performs  post-­‐[OS]-­‐installa=on  administra=ve  tasks.    Takes  a  generic  instruc=ons  (e.g.  ensure  MySQL  is  installed)  and  performs  the  task  the  “right  way”.    Uses  Facter  -­‐  a  system  assay  tool,  which  makes  system  informa=on  available  to  Puppet  as  variables  -­‐  to  get  the  lay  of  the  land.    santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory
  3. 3. [epel-puppet]name=epel puppetbaseurl=http://tmz.fedorapeople.org/repo/puppet/epel/5/$basearch/enabled=0gpgcheck=010/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory3Fairly simple: yum install < puppet-server | puppet >  Use  tmz  epel-­‐puppet  for  the  latest  version:        epel  is  s=ll  required  for  other  dependencies.    puppetlab  and  ruby  are  also  required  for  puppet-­‐dashboard:  Installation: Various repo[puppetlabs]name=Puppet Labs Packagesbaseurl=http://yum.puppetlabs.com/base/enabled=0gpgcheck=0[ruby]name=rubybaseurl=http://repo.premiumhelp.eu/ruby/gpgcheck=0enabled=0
  4. 4. 10/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory4Basic DesignTypically a Client/Server [Master/Agent] formation  Two-­‐part  installa=on:  puppet-­‐server  on  the  server  and  puppet[-­‐agent]  on  the  client  machine.    puppetmasterd  runs  on  server;  puppetd  runs  on  the  agents.    Each  agent  contacts  the  master  periodically  for  updates  (default:  30  mins)  
  5. 5. 10/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory5# /etc/puppet/modules/motd/manifests/init.pp# /etc/puppet/modules/motd/templates/motd.erb:Sample 1: motdclass motd {$ascii = generate(/bin/sh, -c, "/usr/bin/figlet -c -w 60 ${hostname}")file { motd:name => /etc/motd, mode => 0664,owner => root, group => root,content => template(motd/motd.erb’}}<%= ascii %>==================================================Welcome to <%= fqdn %>Access to and use of this server is restricted to thoseactivities expressly permitted by the system administrationstaff. If youre not sure if its allowed, then DONT DO IT. manifests/modules.pp:import ‘motd’manifests/nodes.pp:include motd
  6. 6. 10/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory6Puppet: DashboardBasically a facility management tool  A  Ruby-­‐on-­‐Rail  web  app  -­‐  graphical  representa=on  of  Puppet  Network.    Provides  a  quick  visual  snapshot  of  important  system  informa=on  and  delivers  valuable  reports.    Compara=vely  a  new  project  but  overall  stable    Requirements:    Ruby  -­‐  v1.8.7  is  required    (v1.9.2  is  not  fully  supported)    RubyGems    Rubygem-­‐rake    MySQL    Ruby-­‐MySQL    LS6/RHEL6  is  recomended;  Installing  on  SL5/RHEL5  somewhat  impossible  because  of  the  version  Ruby  used  
  7. 7. 10/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory7Dashboard : Installation  Three  main  installa=on  op=ons  –  rpm,  source  from  Git  and  tarball    Official  yum  repo  (and  deb)  provided  by  Puppet  Labs        epel-­‐repo  is  also  required  for  addi=onal  packages    “yum  install  puppet-­‐dashboard”  should  take  care  of  all  the  dependencies    Default  installa=on  loca=on:  /usr/share/puppet-­‐dashboard    Files/directories  should  be    owned  by  puppet-­‐dashboard  user    Values  are  specified  in:  /etc/sysconfig/puppet-­‐dashboard  [puppetlabs]name=Puppet Labs Packagesbaseurl=http://yum.puppetlabs.com/el/6/products/$basearchEnabled=1gpgcheck=1gpgkey=http://yum.puppetlabs.com/RPM-GPG-KEY-puppetlabs
  8. 8. 10/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory8Dashboard : Configuration1Setting up database  Dashboard  needs  a  user/password  and  at  least  one  database    Can  be  install  on    remote  machine  as  well      Tuning  Needs  to  configure  the  Maximum  Packet  Size  to  accommodate  the  larger  rows  in  the  database  (in  /etc/my.cnf)  mysql> CREATE DATABASE pup_dash CHARACTER SET utf8;Query OK, 1 row affected (0.00 sec)Mysql> GRANT ALL PRIVILEGES ON pup_dash.* TO puppet@’localhost IDENTIFIED BY’pup_db_password;Query OK, 0 row affected (0.00 sec)## Up to [occasional] 17MB row is possiblemax_allowed_packet = 24M
  9. 9. 10/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory9Dashboard : Configuration2Configuration files  Main  configura=on  files  –  config/database.yml  &  config/sekngs.yml    Need  to  add  the  database  parameter  in  database.yml        produc1on  gives  the  best  performance, development  yields  beler  logging    The  same  database  can  be  used  for  produc=on  and  development    Supplied  sekngs.yml.example  can  simply  be  copied  to  seGngs.yml  and  doesn’t  required  any  changes  for  normal  opera=ons.  production:host: puppet.hep.phy.cam.ac.ukdatabase: pup_dashusername: puppetpassword: pup_db_userencoding: utf8adapter: mysqldevelopment:host: puppet.hep.phy.cam.ac.ukdatabase: pup_dashusername: puppetpassword: pup_db_userencoding: utf8adapter: mysql
  10. 10. 10/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory10Dashboard : Configuration3Preparing the Schema  Rail  doesn’t  consider  produc1on  as  the  default  environment;  must  be  specified  manually    Database  table  crea=on  is  done  manually  using  db:migrate  rake  task  and  can  be  run  mul=ple  =me  without  any  problem.    Dashboard  now  can  be  started  using  Ruby’s  built-­‐in  WEBrick  server    Dashdoard  instance  on  default  port  3000  using  “produc=on”  environment.    The  UI  is  available  at  hIp://localhost:3000  rake RAILS_ENV=production db:migratecd /usr/share/puppet-dashboardsudo -u puppet-dashboard ./script/server -e production
  11. 11. 10/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory11Dashboard : Configuration4Using dashboard as Reporter  Two  requirements  for  dashboard  to  receive  reports:    Agents  have  to  be  configured  to  submit  reports  to  the  master    Master  needs  to  be  configured  to  send  the  reports  to  Dashboard        Tes=ng  from  the  agent:  puppet  agent  –test  [  or  puppetd  –t  ]    Conforma=on  on  the  web  GUI:  1  pending  task      delayed_job  workers  must  be  ac=vated  to  asynchronously  process  the  tasks  ## puppet.conf (on the master)[master]reports = store, http, logreportdir = /var/lib/puppet/reportsreporturl = http://localhost:3000/reports/upload## puppet.conf (on agents)[agent]report = trueenv RAILS_ENV=production script/delayed_job -p dashboard -n 4 -m start
  12. 12. 10/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory12Dashboard : Web GUI
  13. 13. 10/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory13Dashboard : Web GUI
  14. 14. 10/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory14Dashboard : Web GUI
  15. 15. 10/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory15Dashboard : Production ServerDashboard with Apache/Passenger  Default  WEBrick  server  suffers  from  performance  issue      More  reliable  using  Apache  (2.2)  with  Phusion  Passenger  (mod_rail)    2  [most  easiest]  ways  to  install  Passenger:            Sample  vhost  file  is  provided  as  ext/dashboard-­‐vhost.conf  Passenger  runs  Rails  apps  in  the  produc=on  environment  by  default  gem install passengerpassenger-install-apache2-moduleyum install http://passenger.stealthymonkeys.com/rhel/6/passenger-release.noarch.rpmpassenger-install-apache2-moduleyum install mod_passenger
  16. 16. 10/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory16Dashboard : Production ServerDashboard with Apache/Passenger  PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2PassengerRuby /usr/bin/rubyPassengerHighPerformance onPassengerMaxPoolSize 12PassengerPoolIdleTime 1500PassengerStatThrottleRate 120RailsAutoDetect On
  17. 17. 10/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory17Dashboard : Production ServerDashboard with Apache/Passenger  <VirtualHost *:80>ServerName puppet.hep.phy.cam.ac.ukDocumentRoot /usr/share/puppet-dashboard/public/<Directory /usr/share/puppet-dashboard/public/>Options NoneOrder allow,denyallow from all</Directory>ErrorLog /var/log/httpd/puppet-dashboard_error.logLogLevel warnCustomLog /var/log/httpd/puppet-dashboard_access.log combinedServerSignature On
  18. 18. 10/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory18Dashboard : Error Messages  Could  not  retrieve  catalog:  cant  convert  nil  into  String  This  error  message  happens  when  a  template  file  is  missing.      undefined  method  `closed?  for  nil:NilClass  Some  error  in  a  template  -­‐  for  example  missing  a  closing  quote.    Could  not  retrieve  catalog  from  remote  server  This  can  occur  if  /var/lib/puppet/client_yaml/  is  missing  on  the  client    Could  not  intern  from  pson:  Could  not  convert  from  pson:    For  several  reason:  Version  of  puppet  agent  is  higher  then  puppet  master,  WEBrick  fails  to  handle  requests  etc.    Cannot  override  local  resource  on  node  There  is  a  duplicate  defini1on.  Search  the  database  like  this:  "select  hosts.name  from  hosts,resources  where  restype=Opsviewmonitored  and  1tle=foo  and  hosts.id  =  resources.host_id;”  
  19. 19. 10/05/2012 santanu@hep.phy.cam.ac.uk, HEPCavendish Laboratory19Any Questions??Thank you!!