• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Build your own clouds with Chef and MCollective
 

Build your own clouds with Chef and MCollective

on

  • 6,167 views

One important part of the DevOps movement is infrastructure automation, especially if you are running your application on top of services like Amazon EC2....

One important part of the DevOps movement is infrastructure automation, especially if you are running your application on top of services like Amazon EC2.

Everybody's dream is to be able to bootstrap and deploy hundreds or even thousands of machines with a few simple commands. This talk will tell you how you can do this using Open Source tools like Chef and mcollective. Chef manages your servers configuration using a nice Ruby DSL while mcollective orchestrates and commands all your nodes.

Statistics

Views

Total Views
6,167
Views on SlideShare
6,167
Embed Views
0

Actions

Likes
13
Downloads
118
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Build your own clouds with Chef and MCollective Build your own clouds with Chef and MCollective Presentation Transcript

    • Build your own cloudswith Chef & MCollectiveJonathan Weiss
    • Who am I?Working for Peritor in Berlin, GermanyWritten, maintain, or involved in   Webistrano   Capistrano   SimplyStored   Happening   The great fire of Londonhttp://github.com/jweiss@jweiss 2
    • ScalariumAmazon EC2 Cluster Management   Auto-Config   Self-Healing   Auto-Scaling   One-click-deploymentwww.scalarium.com 3
    • Cloud Computing Cloud Computing 4
    • 5
    • 6
    • What you get Open Solaris Oracle Suse Windows Linux Ubuntu CentOS Red IBM Hat Infrastructure as a Service
    • What you really want Load balancer App server Database Fully configured stack of servers, dependencies, firewalls, … + your own application deployed
    • Automation is needed for • Configuration • Healing • Scaling • Deploymentflickr.com/photos/torek/3788181603
    • Automation Solutions
    • Reasons to build your own  In-house capacity  Governance: Security and regulations  Flexibility (or lack of…) 11
    • Ingredients 12
    • Components Communication Hostconfiguration Command & Control
    • Command & ControlResponsibilities:   Communication with IaaS API   User Interface (Web, CLI, API): Query, deploy, start, stop   Repository of projects &configuration   Controls which hosts do what&when 14
    • Command & ControlResponsibilities: Implementation:   Communication with IaaS API   Fog & other API implemenations   User Interface (Web, CLI, API):   Rails Query, deploy, start, stop   Repository of projects &configuration   CouchDB   Controls which hosts do what&when   Async wokers & agents 15
    • CommunicationResponsibilities:   Distribute commands and responses in a scalable & fault tolerant way 16
    • CommunicationResponsibilities: Implementation:   Distribute commands and responses   MCollective / ActiveMQ in a scalable & fault tolerant way 17
    • Host ConfigurationResponsibilities:   Actually executes commands   Responds to cluster changes   Ensures local host is configured according to commands: Install software, start/stop services, ... 18
    • Host ConfigurationResponsibilities: Implementation:   Actually executes commands   Chef   Responds to cluster changes   MCollective Agents   Ensures local host is configured according to commands: Install software, start/stop services, ... 19
    • Host ConfigurationBootstrapping new servers 20
    • ChefOpen Source Provisioning Tool   Configures your servers   Cookbooks & recipes   Ruby DSL 21
    • Chef 22
    • Chef Server 23
    • Chef Solo 24
    • Chef-Solo vs. Chef-ServerChef-Server   Complex   No concept of different environments, e.g. staging vs. production   Didn‘t want to run one Chef server per project (and env)   No lifecycle events, e.g. deployment vs. bootstrap   Pull vs. pushChef-Solo   MCollective agents to push cookbooks and Chef runs 25
    • Cookbooks Cookbooks apache2 memcached mysql rails 26
    • Memcached Cookbook Cookbooks memcachd attributes recipes templates 27
    • Memcached Cookbook memcached attributes default.rb recipes service.rb install.rb templates default memcached.conf.erb 28
    • Attributescookbooks/memcached/attributes/default.rbSet default values for variables to use in recipes & templates 29
    • Recipescookbooks/memcached/recipes/service.rbActual instructions to execute 30
    • Recipescookbooks/memcached/recipes/install.rb 31
    • Templatescookbooks/memcached/templates/default/memcached.conf.erb 32
    • Chef Solo Run Configuration JSON: /etc/chef/solo.json Run Chef Solo 33
    • Available Resources Cookbook File Ifconfig SCM Cron Link Script Deploy Log Service Directory Mdadm Subversion Erlang Call Mount Template Execute Package User File Remote Directory Git Remote File Group Route HTTP Request Ruby Block 34
    • Deployment with ChefDeploy resource   Compatible with Capistrano layout   Supports Capistrano hooks via deploy/hook_name.rb   Handles source checkout & callbacksCall custom script   Arbitrary options, e.g: Download WAR file Compile Source …   Automatically wrapped in Chef error handling & notifications 35
    • Deploy Resource 36
    • CommunicationRelaying messages to servers 37
    • Marionette Collective„a framework to build server orchestration or parallel job execution systems“ 38
    • MCollective Architecture 39
    • RPC Calls Service: chef-run Service: update-cookbooks Client Discover  agents Service: chef-run Service: update-cookbooks Message Routing Agents
    • RPC Calls Service: chef-run Service: update-cookbooks Client Service: chef-run Service: update-cookbooks Call agents Message Routing Agents
    • MCollective Agent 42
    • Calling an Agent 43
    • Calling an Agent 44
    • Filters & FactsLimit responding agents by filters of facts/classes:Classes   Arbitary used defined settings  role   Can be set by userdataFacts   Information about local machine   Plugins for Chef/Ohai, Puppet/Facter   Examples: Linux version, installed packages, .... 45
    • Filters on the Command Line 46
    • Filters in Code 47
    • MCollective & ChefCall Chef action on remote agents/machines 48
    • MCollective & ChefAgent running Chef 49
    • SecurityClients   Stomp / ActiveMQ user   AES+RSA plugins: encrypt & sign every message   SSL plugin: sign every message   TLS for connection encryptionMiddleware   Topic permissions and subclustersAgents   All client security measures available   RPC authorization and auditing 50
    • Scalability & RedundancyActiveMQ offers different means   Network of brokers   Broker clusters   Master/Slave 51
    • Command & ControlControlling the cloud 52
    • Command SoftwareGeneral capabilities   Stores cluster model   Interacts with IaaS API (EC2, VMWare & friends)   Handles recovery & presence   Integrates communication solution   Generates host configuration / Chef eventsCustom requirements   User management & authorization   Business rules & processes   Deployment recipes & scripts 53
    • Server Roles 54
    • Command Software 55
    • ExperiencesChef   Flexible and powerful   Easy to learn first steps - easy to write spaghetti recipes   Some annoyances: pseudo idempotent and 2 phases   “Write once, test everywhere”MCollective   Simple and robust   Missing presences and events (Nanite has them but has different problems)   Integrated with Puppet in the future? 56
    • Q&APeritor GmbHBlücherstr. 22, Hof III Aufgang 610961 BerlinTel.: +49 (0)30 69 20 09 84 0Fax: +49 (0)30 69 20 09 84 9Internet: www.peritor.comE-Mail: info@peritor.com© Peritor GmbH - Alle Rechte vorbehalten
    • IdempotentBug or feature? 58
    • Will never happen, even if/data/logs/redhat will be created! 59
    • Two Phases of Chef1) Compile Load all attributes&recipes and build list of actions/dependencies Compute attribute tree2) Run Random Ruby Code runs here Actually execute the resources 60
    • Two Phases of Chef Use only_if and not_if Tell Chef to execute your ressource during compile phase 61