One important part of the DevOps movement is infrastructure automation, especially if you are running your application on top of services like Amazon EC2. Everybody's dream is to be able to bootstrap and deploy hundreds or even thousands of machines with a few simple commands. This talk will tell you how you can do this using Open Source tools like Chef and mcollective. Chef manages your servers configuration using a nice Ruby DSL while mcollective orchestrates and commands all your nodes.

1. Build your own clouds
with Chef & MCollective
Jonathan Weiss

2. Who am I?
Working for Peritor in Berlin, Germany
Written, maintain, or involved in
 Webistrano
 Capistrano
 SimplyStored
 Happening
 The great fire of London
http://github.com/jweiss
@jweiss
2

3. Scalarium
Amazon EC2 Cluster Management
 Auto-Config
 Self-Healing
 Auto-Scaling
 One-click-deployment
www.scalarium.com
3

4. Cloud Computing
Cloud Computing
4

5. 5

6. 6

7. What you get
Open
Solaris Oracle Suse
Windows
Linux
Ubuntu
CentOS Red
IBM
Hat
Infrastructure as a Service

8. What you really want
Load
balancer
App server
Database
Fully configured stack of servers, dependencies, firewalls, …
+
your own application deployed

9. Automation is
needed for
• Configuration
• Healing
• Scaling
• Deployment
flickr.com/photos/torek/3788181603

10. Automation Solutions

11. Reasons to build your own
 In-house capacity
 Governance: Security and regulations
 Flexibility (or lack of…)
11

12. Ingredients
12

13. Components
Communication
Host
configuration
Command &
Control

14. Command & Control
Responsibilities:
 Communication with IaaS API
 User Interface (Web, CLI, API):
Query, deploy, start, stop
 Repository of projects &configuration
 Controls which hosts do what&when
14

15. Command & Control
Responsibilities: Implementation:
 Communication with IaaS API  Fog & other API implemenations
 User Interface (Web, CLI, API):  Rails
Query, deploy, start, stop
 Repository of projects &configuration  CouchDB
 Controls which hosts do what&when  Async wokers & agents
15

16. Communication
Responsibilities:
 Distribute commands and responses
in a scalable & fault tolerant way
16

17. Communication
Responsibilities: Implementation:
 Distribute commands and responses  MCollective / ActiveMQ
in a scalable & fault tolerant way
17

18. Host Configuration
Responsibilities:
 Actually executes commands
 Responds to cluster changes
 Ensures local host is configured
according to commands:
Install software, start/stop services, ...
18

19. Host Configuration
Responsibilities: Implementation:
 Actually executes commands  Chef
 Responds to cluster changes  MCollective Agents
 Ensures local host is configured
according to commands:
Install software, start/stop services, ...
19

20. Host Configuration
Bootstrapping new servers
20

21. Chef
Open Source Provisioning Tool
 Configures your servers
 Cookbooks & recipes
 Ruby DSL
21

22. Chef
22

23. Chef Server
23

24. Chef Solo
24

25. Chef-Solo vs. Chef-Server
Chef-Server
 Complex
 No concept of different environments, e.g. staging vs. production
 Didn‘t want to run one Chef server per project (and env)
 No lifecycle events, e.g. deployment vs. bootstrap
 Pull vs. push
Chef-Solo
 MCollective agents to push cookbooks and Chef runs
25

26. Cookbooks
Cookbooks
apache2
memcached
mysql
rails
26

27. Memcached Cookbook
Cookbooks
memcachd
attributes
recipes
templates
27

28. Memcached Cookbook
memcached
attributes
default.rb
recipes
service.rb
install.rb
templates
default
memcached.conf.erb
28

29. Attributes
cookbooks/memcached/attributes/default.rb
Set default values for variables to use in recipes & templates
29

30. Recipes
cookbooks/memcached/recipes/service.rb
Actual instructions to execute
30

31. Recipes
cookbooks/memcached/recipes/install.rb
31

32. Templates
cookbooks/memcached/templates/default/memcached.conf.erb
32

33. Chef Solo Run
Configuration JSON: /etc/chef/solo.json
Run Chef Solo
33

34. Available Resources
Cookbook File Ifconfig SCM
Cron Link Script
Deploy Log Service
Directory Mdadm Subversion
Erlang Call Mount Template
Execute Package User
File Remote Directory
Git Remote File
Group Route
HTTP Request Ruby Block
34

35. Deployment with Chef
Deploy resource
 Compatible with Capistrano layout
 Supports Capistrano hooks via deploy/hook_name.rb
 Handles source checkout & callbacks
Call custom script
 Arbitrary options, e.g:
Download WAR file
Compile Source
…
 Automatically wrapped in Chef error handling & notifications
35

36. Deploy Resource
36

37. Communication
Relaying messages to servers
37

38. Marionette Collective
„a framework to build server orchestration
or parallel job execution systems“
38

39. MCollective Architecture
39

40. RPC Calls
Service: chef-run
Service: update-cookbooks
Client
Discover
agents
Service: chef-run
Service: update-cookbooks
Message Routing
Agents

41. RPC Calls
Service: chef-run
Service: update-cookbooks
Client
Service: chef-run
Service: update-cookbooks
Call
agents
Message Routing
Agents

42. MCollective Agent
42

43. Calling an Agent
43

44. Calling an Agent
44

45. Filters Facts
Limit responding agents by filters of facts/classes:
Classes
 Arbitary used defined settings  role
 Can be set by userdata
Facts
 Information about local machine
 Plugins for Chef/Ohai, Puppet/Facter
 Examples: Linux version, installed packages, ....
45

46. Filters on the Command Line
46

47. Filters in Code
47

48. MCollective Chef
Call Chef action on remote agents/machines
48

49. MCollective Chef
Agent running Chef
49

50. Security
Clients
 Stomp / ActiveMQ user
 AES+RSA plugins: encrypt sign every message
 SSL plugin: sign every message
 TLS for connection encryption
Middleware
 Topic permissions and subclusters
Agents
 All client security measures available
 RPC authorization and auditing
50

51. Scalability Redundancy
ActiveMQ offers different means
 Network of brokers
 Broker clusters
 Master/Slave
51

52. Command Control
Controlling the cloud
52

53. Command Software
General capabilities
 Stores cluster model
 Interacts with IaaS API (EC2, VMWare friends)
 Handles recovery presence
 Integrates communication solution
 Generates host configuration / Chef events
Custom requirements
 User management authorization
 Business rules processes
 Deployment recipes scripts
53

54. Server Roles
54

55. Command Software
55

56. Experiences
Chef
 Flexible and powerful
 Easy to learn first steps - easy to write spaghetti recipes
 Some annoyances: pseudo idempotent and 2 phases
 “Write once, test everywhere”
MCollective
 Simple and robust
 Missing presences and events
(Nanite has them but has different problems)
 Integrated with Puppet in the future?
56

57. QA
Peritor GmbH
Blücherstr. 22, Hof III Aufgang 6
10961 Berlin
Tel.: +49 (0)30 69 20 09 84 0
Fax: +49 (0)30 69 20 09 84 9
Internet: www.peritor.com
E-Mail: info@peritor.com
© Peritor GmbH - Alle Rechte vorbehalten

58. Idempotent
Bug or feature?
58

59. Will never happen, even if
/data/logs/redhat will be created!
59

60. Two Phases of Chef
1) Compile
Load all attributesrecipes and build list of actions/dependencies
Compute attribute tree
2) Run Random Ruby Code runs here
Actually execute the resources
60

61. Two Phases of Chef
 Use only_if and not_if
 Tell Chef to execute your ressource during compile phase
61