The document summarizes an agenda for a Belgian Puppet Users Group meeting about MCollective. The agenda includes an overview of orchestration and MCollective, hands-on setup of MCollective, the MCollective command line tool, MCollective agents, and future plans. It also provides details about installing and configuring MCollective clients, servers, and the ActiveMQ middleware.
5. Orchestration on Wikipedia
Orchestration describes the automated
arrangement, coordination, and management of
complex computer systems, middleware, and
services.
7. MCollective
Marionette Collective:
Framework
Uses Publish Subscribe Middleware
Very scalable (form small to huge clusters)
Broadcast paradigm (Network is the only source of truth)
no central database - no complex naming convention
Simple command line tools
Extremely pluggable
Community extentions available
16. Middleware Choices
ActiveMQ - preferred
Best tested
Performance is great
Powerfull and flexible security features
Scaleable by clustering
Pain in the #$@% to configure
detailed docs on docs.puppetlabs.com
Connector is shipped with MCollective
RabbitMQ
Not that good tested as ActiveMQ
Not documented @ docs.puppetlabs
Connector is shipped with MCollective
Generic Stomp Connector (Deprecated)
Custom Connector Plugins
19. What we need
Centos vagrant boxes images
puppetlabs vagrant boxes
Centos 6.5 64bit nocm
Centos 6.5 32bit nocm
Minimal centos6.5 vagrant box
centos minimal 64 bit version
centos minimal 32 bit
My vagrantfile with bridged networking (with puppetlabs centos 6.5i nocm box)
Vagrantfile (showoff download link)
20. Vagrant setup
Based on the Vagrantfile from previous slide.
Only one ActiveMQ server (running on my laptop)
Only the ':johan' image is needed.
mkdir -p bpug_vagrant/puppet ; cd bpug_vagrant (puppet = shared folder)
download Vagrantfile
Used domainname = koewacht.net
change johan to 'yourname' (should be unique)
adjust box_url ( eg file://'downloaded box file' )
adjust memory settings (currently 1GB)
starting the vagrant box :
vagrant up 'yourname'
Having trouble -- shout !!
logging into your box
vagrant ssh 'yourname'
sudo -i
22. The setup
One central ActiveMQ server (already up and running)
Many MCollective nodes
Your Virtual Boxes ...
Server role
Client role
Bridged mode, so we can see each others node
Installation done by hand
23. Info we need before hand
The ipaddress ActiveMQ server (dhcp based)
The passwords for configuration files :
client: 29l6wD2mIzbLpbp4GMnUzchHp2XWpKk8N8dcxXCnDRU=
server: 04BpZofasX1dDexFsqZcgfM1tkC4VCGI6hoziWMu7zw=
Pre-shared key: Gw8nclOGn1YiIMvEAxgeZ7jrL1ErCdZZXm2e7JX2S4o=
( keys are generated with : $ openssl rand -base64 32 )
24. Requirements
We are using packages from Puppetlabs repos
Mcollective clients/servers
Working NTP
Ruby
1.8.7/1.9.3
2.0.0 not supported yet
1.9.0/1.9.2 will fail
Ruby stomp +1.2.2
Mcollective + 2.5.0
5MB disk
256 MB ram
25. Requirements Continued
Middleware Broker
500 MB ram
Messaging middleware :
ActiveMQ 5.8 with stomp connector
RabbitMQ 2.8 with stomp connector
Disk Space for Middleware server : 15MB
Some CPU & Network capacity (+2 connections per server)
platforms puppetlabs repo
RHEL 5|6|7
Fedora 19 - 20
Debian Lucid|Precise|Saucy|Sid|Squeeze|Trusty|Wheezy
29. Installing the package
On osfamily == RedHat
$ sudo yum install activemq
$ sudo chkconfig activemq on
On osfamily == Debian
$ sudo apt-get install activemq
$ sudo sysv-rc-conf activemq on
30. ActiveMQ Configuration
The /etc/activemq/activemq.xml
Line number correspond to download-able activemq.xml file
Enable Purging the Broker
35 <broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" useJmx="true"
schedulePeriodForDestinationPurge="60000">
Disable producerFlowControl & memory cleanup
50 <destinationPolicy>
51 <policyMap>
52 <policyEntries>
53 <!-- MCollective generally expects producer flow control to be turned off. -->
54 <policyEntry topic=">" producerFlowControl="false" memoryLimit="1mb"/>
55 <!-- MCollective will generate many single-use reply queues,
56 which should be garbage-collected after five minutes to conserve memory. -->
57 <policyEntry queue="*.reply.>" gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000"/>
31. ActiveMQ Configuration - continued
The /etc/activemq/activemq.xml
define logins for clients and servers in simpleAuthenticationPlugins
104 <simpleAuthenticationPlugin>
105 <users>
106 <authenticationUser username="client" password="29l6wD2mIzbLpbp4GMnUzchHp2XWpKk8N8dcxXCnDRU="
groups="servers,clients,everyone"/>
107 <authenticationUser username="server" password="04BpZofasX1dDexFsqZcgfM1tkC4VCGI6hoziWMu7zw="
groups="servers,everyone"/>
108 </users>
109 </simpleAuthenticationPlugin>
33. ActiveMQ Configuration - continued
The /etc/activemq/activemq.xml
Transports - Only one transport should be enabled
156 <transportConnectors>
157 <transportConnector name="stomp+nio" uri="stomp+nio://0.0.0.0:61613"/>
158 </transportConnectors>
Disable web console (commented out)
170 <!-- disabled for security reasons
171 <import resource="jetty.xml"/>
172 -->
34. Fire it up - and check
$ service activemq start
$ netstat -an | grep 61613
$ tail -200f /var/log/activemq/activemq.log | less
In the real world
Adjust firewall (port 61613)
Selinux and equivalents
37. MCollective Server Configuration
/etc/mcollective/server.cfg
(based on the downloadable server.cfg)
user and password are also defined in activemq.xml on messaging server
6 plugin.activemq.pool.size = 1
7 plugin.activemq.pool.1.host = activemq.koewacht.net
8 plugin.activemq.pool.1.port = 61613
9 plugin.activemq.pool.1.user = server
10 plugin.activemq.pool.1.password = 04BpZofasX1dDexFsqZcgfM1tkC4VCGI6hoziWMu7zw=
pre-shared-key form earlier slides
17 # Security provider
18 securityprovider = psk
19 plugin.psk = Gw8nclOGn1YiIMvEAxgeZ7jrL1ErCdZZXm2e7JX2S4o=
Check the libdir directory
22 libdir = /usr/libexec/mcollective
38. Fire it up - and verify
$ service mcollective start
$ netstat -an | grep 61613
tcp 0 0 192.168.10.223:50737 192.168.10.231:61613 ESTABLISHED
43. Testing the Basic Setup
The MCollective Ping Test
low level query
[vagrant@johan ~]$ mco ping
activeMQ.koewacht.net time=176.15 ms
johan.koewacht.net time=185.95 ms
Troubleshooting
Are the passwords & user/groups correct
middleware server : activemq.xml
mcollective server.cfg
mcollective client.cfg
Networking
check for port 61613
45. Introduction mco command-line client
Connector
Clients uses 2 plugins
connector plugin (connection to middleware)
ActiveMQ
security plugin (sign & optionally encript data)
PSK (pre-shared key)
same connectors on all MCollective components
(clients/servers/middleware)
46. Introduction mco command-line client
Inventory
builtin plugin
gathers info about MCollective server
server configuration
server stats
available plugins
Configuration Classes
Facts (aka facter)
47. Introduction mco command-line client
Inventory - example run
$ mco inventory heliotrope
Inventory for heliotrope:
Inventory for heliotrope:
Server Statistics:
Version: 2.5.0
Start Time: Mon Apr 14 03:11:12 -0700 2014
Config File: /etc/mcollective/server.cfg
Collectives: mcollective
Main Collective: mcollective
Process ID: 1334
Total Messages: 16
Messages Passed Filters: 13
Messages Filtered: 3
Expired Messages: 0
Replies Sent: 12
Total Processor Time: 38.56 seconds
System Time: 128.22 seconds
Agents:
discovery rpcutil
Data Plugins:
agent fstat
Configuration Management Classes:
No classes applied
Facts:
No facts known
48. Inventory continued
custom output format
ruby script
use it as script argument
inventory do
format "%20s %8s %10s %-20s"
fields {[ identity, facts["architecture"],facts["operatingsystem"], facts["operatingsystemrelease"]]}
end
$ mco inventory --script inventory.mc
geode x86_64 CentOS 6.4
sunstone amd64 Ubuntu 13.10
heliotrope x86_64 CentOS 6.5
49. Discovery
mc plugin
built in
defined in client.cfg (mc plugin)
13 # Use auto-discovery
14 default_discovery_method = mc
sends broadcast queries
mco plugin doc mc
flatfile plugin
list of hostnames from file
mco plugin doc flatfile
50. Discovery
flatfile plugin
$ cat /path/to/hostlist
fireagate
heliotrope
$ mco rpc rpcutil ping --disc-method flatfile --disc-option /path/to/hostlist
Discovering hosts using the flatfile method .... 2
* [ ============================================================>] 2 / 2
heliotrope
Timestamp: 1385012042
fireagate
Timestamp: 1385012044
Finished processing 2 / 2 hosts in 146.13 ms
mco rpc rpcutil is how to invoke a direct call to the API without using the client application.
51. MCollective's filters
Can be used on all MCollective commands
$ mco help <command>
Host Filters
-W, --with FILTER Combined classes and facts filter
-S, --select FILTER Compound filter combining facts and classes
-F, --wf, --with-fact fact=val Match hosts with a certain fact
-C, --wc, --with-class CLASS Match hosts with a certain config management class
-A, --wa, --with-agent AGENT Match hosts with a certain agent
-I, --wi, --with-identity IDENT Match hosts with a certain configured identity
53. MCollective combined filters
Types of combined filters
Puppet Classes & Facter facts
$ mco ping --with "/^webd/ operatingsystem=CentOS"
Select filter
combination of
Factes and Classes
Boolean Logic ( AND - OR - NOT|! )
$ mco ping --select "operatingsystem=CentOS and /nameserver/"
$ mco ping --select "operatingsystem=CentOS and !environment=dev"
$ mco ping --select "( /httpd/ or /nginx/ ) and is_virtual=true"
CentOS hosts named web followed by a number.
Ping only CentOS hosts which have the nameserver class applied to them.
Ping every CentOS host which isn’t in the dev environment.
match virtualized hosts with either the httpd or nginx Puppet class applied to them.
54. Add limitations to MCollective command
Limit option
Control how many servers get the request
--one
--limit
--limit matching server
$ mco ping --limit 15
$ mco ping --one --with-fact operatingsystem=CentOS
$ mco ping --limit 5 --with-class webserver
$ mco ping --limit 33% --with-class webserver
Fifteen servers of any type
Only one CentOS server
Five servers which have the webserver Puppet class applied to them
One third of the servers which have the webserver Puppet class applied to them
55. Add limitations to MCollective command
batch option
Controls how many servers receive the request in batch
Controls time between batches
$ mco ping --batch 5 --batch-sleep 30 --with-fact country=de
$ mco package upgrade sudo --batch 10 --batch-sleep 20
Ping batches of five German servers every 30 seconds
Fast upgrade sudo in batches of ten servers spaced twenty seconds apart
56. Controlling mco command output
--json
output in json format
--no-progress
supress status bar
--verbose
timing discovery
full RPC statistics
59. MCollective & Puppet Classes
Only works with puppet
Puppet agents :
writes classes.txt
$statedir (/var/lib/puppet/state)
agent node runs MCollective server
puppet agent --configprint classfile
must match classesfile /etc/mcollective/server.cfg
We can simulate puppet classes by faking a classes.txt in /etc/mcollective/classes.txt
64. MCollective Agent - The Components
The DLL file
DDL = Data Description Language
Definition remote methods
Description input format
Description generated output
metadata
author
version
license
...
Used for Validating Input
If you stick to code convention
66. MCollective Agent - The Components
The Agent Plugin
Installed on all MCollective servers
Uses DLL for Meta Data & initialization
Defines Agent Actions
Action : individual tasks the agent can do
67. MCollective Agent - The Components
The Client
Installed only on MCollective clients
Provides access to agents and actions
Uses also DDL
eg. input validation
....
Clients - Agents - DLL are strongly coupled
68. MCollective Client Help
[vagrant@johan ~]$ mco help plugin package
MCollective Plugin Application
Usage: mco plugin package [options] <directory>
mco plugin info <directory>
mco plugin doc <plugin>
mco plugin doc <type/plugin>
mco plugin generate agent <pluginname> [actions=val,val]
mco plugin generate data <pluginname> [outputs=val,val]
info : Display plugin information including package details.
package : Create all available plugin packages.
doc : Display documentation for a specific plugin.
Application Options
-n, --name NAME Plugin name
--postinstall POSTINSTALL Post install script
--preinstall PREINSTALL Pre install script
--revision REVISION Revision number
....
-h, --help Display this screen
The Marionette Collective 2.5.2
[vagrant@johan ~]$
70. This is not the end,
Just the beginning
Delve much deeper into MCollective
Read, Read and Read even more
Experiment as much as you can
Secure your MCollective Infrastructure
Authentication connector
Tuning your ActiveMQ
Puppetlabs Docs on ActiveMQ & MCollective
Manage yout MCollective infrastructure with puppet
Puppetlabs MCollective Module on the Forge
Learning MCollective pupept module
Great for getting more insight in managing MCollective with
puppet
71. references
wikipedia - Orchastration(computing
PuppetLabs MCollective online docs
Introduction to orchestration using MCollective - Pieter Loubser
Inroduction to Mcollective - R.I. Pienaar
MCollective Installed. And now ? - Thomas Gelf
Learning MCollective - Jo Rhet (O'Reilly)
This Presentation on Github