Bpug mcollective 20140624


Published on

Slides of the Belgian Puppet User Group Meetup "something about MCollective" we held on the 24 of June 2014.
The source of these slides can be found at https://github.com/witjoh/BPUG_MCollective

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Bpug mcollective 20140624

  1. 1. Belgian Puppet Users Group Something About MCollective 24th of June 2014 Hosted by Telenet/Hostbasket Lochristi - Belgium
  2. 2. Agenda
  3. 3. Agenda Orchestration & MCollective Hands-on - Setting up MCollective The 'mco' command MCollective Agents And what about Tomorrow ?
  4. 4. Orchestration
  5. 5. Orchestration on Wikipedia Orchestration describes the automated arrangement, coordination, and management of complex computer systems, middleware, and services.
  6. 6. Orchestration in human Language ? Parallel job execution System
  7. 7. MCollective Marionette Collective: Framework Uses Publish Subscribe Middleware Very scalable (form small to huge clusters) Broadcast paradigm (Network is the only source of truth) no central database - no complex naming convention Simple command line tools Extremely pluggable Community extentions available
  8. 8. MCollective build upon existing middleware uses existing authentication/authorisation models uses existing clustering techniques uses existing routing/network isolation methods
  9. 9. Marionette Collective Pluggable core middleware (STOMP compliant) Authorisation serialisation Data Sources (Chef & Puppet supported + Facter[Community]) Mcollective as transport (eg, central service inventory system)
  10. 10. MCollective - Components
  11. 11. Overview - Components
  12. 12. The MCollective Server - mcollectived
  13. 13. The MCollective Client
  14. 14. MCollective Middleware Overview
  15. 15. Inside The MCollective Middleware
  16. 16. Middleware Choices ActiveMQ - preferred Best tested Performance is great Powerfull and flexible security features Scaleable by clustering Pain in the #$@% to configure detailed docs on docs.puppetlabs.com Connector is shipped with MCollective RabbitMQ Not that good tested as ActiveMQ Not documented @ docs.puppetlabs Connector is shipped with MCollective Generic Stomp Connector (Deprecated) Custom Connector Plugins
  17. 17. Getting dirty hands
  18. 18. Vagrant boxes
  19. 19. What we need Centos vagrant boxes images puppetlabs vagrant boxes Centos 6.5 64bit nocm Centos 6.5 32bit nocm Minimal centos6.5 vagrant box centos minimal 64 bit version centos minimal 32 bit My vagrantfile with bridged networking (with puppetlabs centos 6.5i nocm box) Vagrantfile (showoff download link)
  20. 20. Vagrant setup Based on the Vagrantfile from previous slide. Only one ActiveMQ server (running on my laptop) Only the ':johan' image is needed. mkdir -p bpug_vagrant/puppet ; cd bpug_vagrant (puppet = shared folder) download Vagrantfile Used domainname = koewacht.net change johan to 'yourname' (should be unique) adjust box_url ( eg file://'downloaded box file' ) adjust memory settings (currently 1GB) starting the vagrant box : vagrant up 'yourname' Having trouble -- shout !! logging into your box vagrant ssh 'yourname' sudo -i
  21. 21. Info we need
  22. 22. The setup One central ActiveMQ server (already up and running) Many MCollective nodes Your Virtual Boxes ... Server role Client role Bridged mode, so we can see each others node Installation done by hand
  23. 23. Info we need before hand The ipaddress ActiveMQ server (dhcp based) The passwords for configuration files : client: 29l6wD2mIzbLpbp4GMnUzchHp2XWpKk8N8dcxXCnDRU= server: 04BpZofasX1dDexFsqZcgfM1tkC4VCGI6hoziWMu7zw= Pre-shared key: Gw8nclOGn1YiIMvEAxgeZ7jrL1ErCdZZXm2e7JX2S4o= ( keys are generated with : $ openssl rand -base64 32 )
  24. 24. Requirements We are using packages from Puppetlabs repos Mcollective clients/servers Working NTP Ruby 1.8.7/1.9.3 2.0.0 not supported yet 1.9.0/1.9.2 will fail Ruby stomp +1.2.2 Mcollective + 2.5.0 5MB disk 256 MB ram
  25. 25. Requirements Continued Middleware Broker 500 MB ram Messaging middleware : ActiveMQ 5.8 with stomp connector RabbitMQ 2.8 with stomp connector Disk Space for Middleware server : 15MB Some CPU & Network capacity (+2 connections per server) platforms puppetlabs repo RHEL 5|6|7 Fedora 19 - 20 Debian Lucid|Precise|Saucy|Sid|Squeeze|Trusty|Wheezy
  26. 26. Installing the packages
  27. 27. Installing Puppetlabs repos osfamily == RedHat $ sudo yum install http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm $ sudo yum install http://yum.puppetlabs.com/puppetlabs-release-fedora-20.noarch.rpm osfamily == Debian $ wget http://apt.puppetlabs.com/puppetlabs-release-sid.deb $ sudo dpkg -i puppetlabs-release-sid.deb $ sudo apt-get update (replace sid with your version)
  28. 28. Installing ActiveMQ - How I Did It
  29. 29. Installing the package On osfamily == RedHat $ sudo yum install activemq $ sudo chkconfig activemq on On osfamily == Debian $ sudo apt-get install activemq $ sudo sysv-rc-conf activemq on
  30. 30. ActiveMQ Configuration The /etc/activemq/activemq.xml Line number correspond to download-able activemq.xml file Enable Purging the Broker 35 <broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" useJmx="true" schedulePeriodForDestinationPurge="60000"> Disable producerFlowControl & memory cleanup 50 <destinationPolicy> 51 <policyMap> 52 <policyEntries> 53 <!-- MCollective generally expects producer flow control to be turned off. --> 54 <policyEntry topic=">" producerFlowControl="false" memoryLimit="1mb"/> 55 <!-- MCollective will generate many single-use reply queues, 56 which should be garbage-collected after five minutes to conserve memory. --> 57 <policyEntry queue="*.reply.>" gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000"/>
  31. 31. ActiveMQ Configuration - continued The /etc/activemq/activemq.xml define logins for clients and servers in simpleAuthenticationPlugins 104 <simpleAuthenticationPlugin> 105 <users> 106 <authenticationUser username="client" password="29l6wD2mIzbLpbp4GMnUzchHp2XWpKk8N8dcxXCnDRU=" groups="servers,clients,everyone"/> 107 <authenticationUser username="server" password="04BpZofasX1dDexFsqZcgfM1tkC4VCGI6hoziWMu7zw=" groups="servers,everyone"/> 108 </users> 109 </simpleAuthenticationPlugin>
  32. 32. ActiveMQ Configuration - continued The /etc/activemq/activemq.xml Define permissions for clients and servers in authorizationPlugins 110 <authorizationPlugin> 111 <map> 112 <authorizationMap> 113 <authorizationEntries> 114 <authorizationEntry queue=">" write="admins" read="admins" admin="admins" /> 115 <authorizationEntry topic=">" write="admins" read="admins" admin="admins" /> 116 <authorizationEntry queue="mcollective.>" write="clients" read="clients" admin="clients" /> 117 <authorizationEntry topic="mcollective.>" write="clients" read="clients" admin="clients" /> 118 <authorizationEntry queue="mcollective.nodes" read="servers" admin="servers" /> 119 <authorizationEntry queue="mcollective.reply.>" write="servers" admin="servers" /> 120 <authorizationEntry topic="mcollective.*.agent" read="servers" admin="servers" /> 121 <authorizationEntry topic="mcollective.registration.agent" write="servers" read="servers" admin="servers" /> 122 <authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/> 123 </authorizationEntries> 124 </authorizationMap> 125 </map> 126 </authorizationPlugin>
  33. 33. ActiveMQ Configuration - continued The /etc/activemq/activemq.xml Transports - Only one transport should be enabled 156 <transportConnectors> 157 <transportConnector name="stomp+nio" uri="stomp+nio://"/> 158 </transportConnectors> Disable web console (commented out) 170 <!-- disabled for security reasons 171 <import resource="jetty.xml"/> 172 -->
  34. 34. Fire it up - and check $ service activemq start $ netstat -an | grep 61613 $ tail -200f /var/log/activemq/activemq.log | less In the real world Adjust firewall (port 61613) Selinux and equivalents
  35. 35. Installing MCollective Servers
  36. 36. Installing the package osfamily == RedHat $ sudo yum install mcollective $ sudo chkconfig mcollective on osfamily == Debian $ sudo apt-get install ruby-stomp mcollective $ sudo sysv-rc-conf mcollective on
  37. 37. MCollective Server Configuration /etc/mcollective/server.cfg (based on the downloadable server.cfg) user and password are also defined in activemq.xml on messaging server 6 plugin.activemq.pool.size = 1 7 plugin.activemq.pool.1.host = activemq.koewacht.net 8 plugin.activemq.pool.1.port = 61613 9 plugin.activemq.pool.1.user = server 10 plugin.activemq.pool.1.password = 04BpZofasX1dDexFsqZcgfM1tkC4VCGI6hoziWMu7zw= pre-shared-key form earlier slides 17 # Security provider 18 securityprovider = psk 19 plugin.psk = Gw8nclOGn1YiIMvEAxgeZ7jrL1ErCdZZXm2e7JX2S4o= Check the libdir directory 22 libdir = /usr/libexec/mcollective
  38. 38. Fire it up - and verify $ service mcollective start $ netstat -an | grep 61613 tcp 0 0 ESTABLISHED
  39. 39. The MCollective Client
  40. 40. Installing the MCollective Client Package osfamily == RedHat $ sudo yum install mcollective-client oSfamily == Debian $ sudo apt-get install mcollective-client
  41. 41. Configuring the MCollective Client (based on the downloadable client.cfg) user and password are also defined in activemq.xml on messaging server 3 connector = activemq 4 plugin.activemq.pool.size = 1 5 plugin.activemq.pool.1.host = activemq.koewacht.net 6 plugin.activemq.pool.1.port = 61613 7 plugin.activemq.pool.1.user = client 8 plugin.activemq.pool.1.password = 29l6wD2mIzbLpbp4GMnUzchHp2XWpKk8N8dcxXCnDRU= 9 plugin.activemq.heartbeat_interval = 30 pre-shared-key form earlier slides 17 # Security provider 18 securityprovider = psk 19 plugin.psk = Gw8nclOGn1YiIMvEAxgeZ7jrL1ErCdZZXm2e7JX2S4o= Check the libdir directory 22 libdir = /usr/libexec/mcollective
  42. 42. Testing the Setup so Far
  43. 43. Testing the Basic Setup The MCollective Ping Test low level query [vagrant@johan ~]$ mco ping activeMQ.koewacht.net time=176.15 ms johan.koewacht.net time=185.95 ms Troubleshooting Are the passwords & user/groups correct middleware server : activemq.xml mcollective server.cfg mcollective client.cfg Networking check for port 61613
  44. 44. MCollective Command Line Client
  45. 45. Introduction mco command-line client Connector Clients uses 2 plugins connector plugin (connection to middleware) ActiveMQ security plugin (sign & optionally encript data) PSK (pre-shared key) same connectors on all MCollective components (clients/servers/middleware)
  46. 46. Introduction mco command-line client Inventory builtin plugin gathers info about MCollective server server configuration server stats available plugins Configuration Classes Facts (aka facter)
  47. 47. Introduction mco command-line client Inventory - example run $ mco inventory heliotrope Inventory for heliotrope: Inventory for heliotrope: Server Statistics: Version: 2.5.0 Start Time: Mon Apr 14 03:11:12 -0700 2014 Config File: /etc/mcollective/server.cfg Collectives: mcollective Main Collective: mcollective Process ID: 1334 Total Messages: 16 Messages Passed Filters: 13 Messages Filtered: 3 Expired Messages: 0 Replies Sent: 12 Total Processor Time: 38.56 seconds System Time: 128.22 seconds Agents: discovery rpcutil Data Plugins: agent fstat Configuration Management Classes: No classes applied Facts: No facts known
  48. 48. Inventory continued custom output format ruby script use it as script argument inventory do format "%20s %8s %10s %-20s" fields {[ identity, facts["architecture"],facts["operatingsystem"], facts["operatingsystemrelease"]]} end $ mco inventory --script inventory.mc geode x86_64 CentOS 6.4 sunstone amd64 Ubuntu 13.10 heliotrope x86_64 CentOS 6.5
  49. 49. Discovery mc plugin built in defined in client.cfg (mc plugin) 13 # Use auto-discovery 14 default_discovery_method = mc sends broadcast queries mco plugin doc mc flatfile plugin list of hostnames from file mco plugin doc flatfile
  50. 50. Discovery flatfile plugin $ cat /path/to/hostlist fireagate heliotrope $ mco rpc rpcutil ping --disc-method flatfile --disc-option /path/to/hostlist Discovering hosts using the flatfile method .... 2 * [ ============================================================>] 2 / 2 heliotrope Timestamp: 1385012042 fireagate Timestamp: 1385012044 Finished processing 2 / 2 hosts in 146.13 ms mco rpc rpcutil is how to invoke a direct call to the API without using the client application.
  51. 51. MCollective's filters Can be used on all MCollective commands $ mco help <command> Host Filters -W, --with FILTER Combined classes and facts filter -S, --select FILTER Compound filter combining facts and classes -F, --wf, --with-fact fact=val Match hosts with a certain fact -C, --wc, --with-class CLASS Match hosts with a certain config management class -A, --wa, --with-agent AGENT Match hosts with a certain agent -I, --wi, --with-identity IDENT Match hosts with a certain configured identity
  52. 52. MCollective filters - examples $ mco find -with-identity /i/ $ mco find -with-identity /^webd/ $ mco find -with-class webserver $ mco find -with-fact operatingsystem=CentOS $ mco find -with-agent package Filters requires the mc Discovery Plugin. Flatfile discovery only supports identity filter
  53. 53. MCollective combined filters Types of combined filters Puppet Classes & Facter facts $ mco ping --with "/^webd/ operatingsystem=CentOS" Select filter combination of Factes and Classes Boolean Logic ( AND - OR - NOT|! ) $ mco ping --select "operatingsystem=CentOS and /nameserver/" $ mco ping --select "operatingsystem=CentOS and !environment=dev" $ mco ping --select "( /httpd/ or /nginx/ ) and is_virtual=true" CentOS hosts named web followed by a number. Ping only CentOS hosts which have the nameserver class applied to them. Ping every CentOS host which isn’t in the dev environment. match virtualized hosts with either the httpd or nginx Puppet class applied to them.
  54. 54. Add limitations to MCollective command Limit option Control how many servers get the request --one --limit --limit matching server $ mco ping --limit 15 $ mco ping --one --with-fact operatingsystem=CentOS $ mco ping --limit 5 --with-class webserver $ mco ping --limit 33% --with-class webserver Fifteen servers of any type Only one CentOS server Five servers which have the webserver Puppet class applied to them One third of the servers which have the webserver Puppet class applied to them
  55. 55. Add limitations to MCollective command batch option Controls how many servers receive the request in batch Controls time between batches $ mco ping --batch 5 --batch-sleep 30 --with-fact country=de $ mco package upgrade sudo --batch 10 --batch-sleep 20 Ping batches of five German servers every 30 seconds Fast upgrade sudo in batches of ten servers spaced twenty seconds apart
  56. 56. Controlling mco command output --json output in json format --no-progress supress status bar --verbose timing discovery full RPC statistics
  57. 57. Facts key/value pairs inventory server Facter generate facts Installing facter osfamily == RedHat $ sudo yum install facter osfamily == Debian $ sudo apt-get install facter
  58. 58. Facter facts & MCollective configure /etc/mcollecive/server.cfg 30 # facter 31 factsource=yaml 32 plugin.yaml=/etc/mcollective/facts.yaml Generate a facts.yaml file $ facter -y > /etc/mcollective/facts.yaml optionally add a crontab $ cat /etc/cron.d/facts.sh */30 * * * * facter -y >/etc/mcollective/facts.yaml restart mcollective $ mco inventory nodename
  59. 59. MCollective & Puppet Classes Only works with puppet Puppet agents : writes classes.txt $statedir (/var/lib/puppet/state) agent node runs MCollective server puppet agent --configprint classfile must match classesfile /etc/mcollective/server.cfg We can simulate puppet classes by faking a classes.txt in /etc/mcollective/classes.txt
  60. 60. MCollective Agents Extending MCollective
  61. 61. MCollective Agents Connector Agents uses 2 plugins connector plugin (connection to middleware) ActiveMQ security plugin (sign & optionally encript data) PSK (pre-shared key) same connectors on all MCollective components (clients/servers/middleware) Agent Parts Agent part (servers) DDL (servers & clients) Client part (clients) Common part (servers & clients)
  62. 62. MCollective Agent - Installing from Packages From the PuppetLabs Repositories Install on every MCollective Server Many community MCollective Agents (eg. github) osfamily == RedHat $ sudo yum install mcollective-filemgr-agent $ sudo yum install mcollective-nettest-agent $ sudo yum install mcollective-package-agent $ sudo yum install mcollective-service-agent osfamily == Debian $ sudo apt-get install mcollective-filemgr-agent $ sudo apt-get install mcollective-nettest-agent $ sudo apt-get install mcollective-package-agent $ sudo apt-get install mcollective-service-agent
  63. 63. MCollective Agent - Inside the Package [vagrant@johan ~]$ rpm -ql mcollective-package-common-4.3.0-1.el6.noarch /usr/libexec/mcollective/mcollective/agent/package.ddl /usr/libexec/mcollective/mcollective/util/package /usr/libexec/mcollective/mcollective/util/package/base.rb /usr/libexec/mcollective/mcollective/util/package/packagehelpers.rb /usr/libexec/mcollective/mcollective/util/package/puppetpackage.rb [vagrant@johan ~]$ rpm -ql mcollective-package-agent-4.3.0-1.el6.noarch /usr/libexec/mcollective/mcollective/agent/package.rb [vagrant@johan ~]$ rpm -ql mcollective-package-client-4.3.0-1.el6.noarch /usr/libexec/mcollective/mcollective/application/package.rb
  64. 64. MCollective Agent - The Components The DLL file DDL = Data Description Language Definition remote methods Description input format Description generated output metadata author version license ... Used for Validating Input If you stick to code convention
  65. 65. MColelctive Agent - The Components MCollective Agent DLL Example [vagrant@johan ~]$ cat /usr/libexec/mcollective/mcollective/agent/package.ddl metadata :name => "package", :description => "Install and uninstall software packages", :author => "R.I.Pienaar", :license => "ASL 2.0", :version => "4.3.0", :url => "https://github.com/puppetlabs/mcollective-package-agent", :timeout => 180 requires :mcollective => "2.2.1" ["install", "update", "uninstall", "purge"].each do |act| action act, :description => "#{act.capitalize} a package" do input :package, :prompt => "Package Name", :description => "Package to #{act}", :type => :string, :validation => :shellsafe, :optional => false, :maxlength => 90 output :output, :description => "Output from the package manager", :display_as => "Output" output :epoch, :description => "Package epoch number", :display_as => "Epoch" ................. ...........
  66. 66. MCollective Agent - The Components The Agent Plugin Installed on all MCollective servers Uses DLL for Meta Data & initialization Defines Agent Actions Action : individual tasks the agent can do
  67. 67. MCollective Agent - The Components The Client Installed only on MCollective clients Provides access to agents and actions Uses also DDL eg. input validation .... Clients - Agents - DLL are strongly coupled
  68. 68. MCollective Client Help [vagrant@johan ~]$ mco help plugin package MCollective Plugin Application Usage: mco plugin package [options] <directory> mco plugin info <directory> mco plugin doc <plugin> mco plugin doc <type/plugin> mco plugin generate agent <pluginname> [actions=val,val] mco plugin generate data <pluginname> [outputs=val,val] info : Display plugin information including package details. package : Create all available plugin packages. doc : Display documentation for a specific plugin. Application Options -n, --name NAME Plugin name --postinstall POSTINSTALL Post install script --preinstall PREINSTALL Pre install script --revision REVISION Revision number .... -h, --help Display this screen The Marionette Collective 2.5.2 [vagrant@johan ~]$
  69. 69. And What About Tomorrow
  70. 70. This is not the end, Just the beginning Delve much deeper into MCollective Read, Read and Read even more Experiment as much as you can Secure your MCollective Infrastructure Authentication connector Tuning your ActiveMQ Puppetlabs Docs on ActiveMQ & MCollective Manage yout MCollective infrastructure with puppet Puppetlabs MCollective Module on the Forge Learning MCollective pupept module Great for getting more insight in managing MCollective with puppet
  71. 71. references wikipedia - Orchastration(computing PuppetLabs MCollective online docs Introduction to orchestration using MCollective - Pieter Loubser Inroduction to Mcollective - R.I. Pienaar MCollective Installed. And now ? - Thomas Gelf Learning MCollective - Jo Rhet (O'Reilly) This Presentation on Github
  72. 72. ??? Questions ???
  73. 73. Thank You For Attending - - - Thanks go to our Host for Tonight Telenet/Hostbasket Do not forget the coolest T-Shirt