• Save
PuppetCamp - How Puppet helped us to standardize, communicate and work together
Upcoming SlideShare
Loading in...5
×
 

PuppetCamp - How Puppet helped us to standardize, communicate and work together

on

  • 5,272 views

Where we were coming from, which problems puppet solved and which new problems puppet introduced and how we solved them. (we have at least 15 different data centers, with thousands of servers)

Where we were coming from, which problems puppet solved and which new problems puppet introduced and how we solved them. (we have at least 15 different data centers, with thousands of servers)

How to manage a large repo with many puppet admins
How important is it to automate your puppetmasters as well
How to coordinate when changes are actually pushed to the clients

Why we built Gini, and how those ideas became Foreman.

Statistics

Views

Total Views
5,272
Slideshare-icon Views on SlideShare
5,257
Embed Views
15

Actions

Likes
10
Downloads
0
Comments
0

1 Embed 15

http://www.slideshare.net 15

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    PuppetCamp - How Puppet helped us to standardize, communicate and work together PuppetCamp - How Puppet helped us to standardize, communicate and work together Presentation Transcript

    • Puppet As A Framework [email_address] 10/01/2009
    • Infineon at a Glance The Company
      • Infineon provides semiconductor and system solutions, focusing on three central needs of our modern society: energy efficiency, communications and security
      • Revenue in FY 2008: 4.321 billion EUR
      • Some 26,000 employees worldwide (as of June 2009)
      • Strong technology portfolio with about 22,900 patents and patent applications
      • More than 30 major R&D locations
      • Germany‘s largest semiconductor company
    • Focus Areas and Target Markets Industrial & Multimarket Automotive Wireline Communications Wireless Solutions Chip Card & Security Energy Efficiency Communications Security 9-months FY09 revenue split Sale of WLC signed July 7
    • Where we are coming from
      • Many Locations and Data Centres
      • Many administrators
      • Different tools
      • Short maintenance window on our servers
        • Never on the same time across the different sites
        • Each host type might have a different maintenance window.
      • You don't understand our requirements
      • You don't understand our solutions
      • Our solutions wont fix your problems
      • My setup is different then yours
      • We are in different timezones
      • We don't want to change
    • Project Goals
      • Harmonization - Same look and feel everywhere (enables consolidation / global operation )
      • Simplify Servers management
      • Implement and maintain Global standards
      • Accurate Configuration Database
      • Rapid recovery and deployment of services globally
      • Eliminates duplicate / unnecessary effort
      • Audits / Compliance
      • Go home early
    • Typical System Lifecycle Pre Installation CMDB DNS,DHCP Network … Initial Configuration Kickstart Jumpstart Custom Scripts Fixes Updates Audits
    • System configuration possibilities
      • Manual System Configuration
        • „ log in and „just do it!““
          • + OK for small number of servers
          • + Quick and easy
          • - Very difficult to ensure similar configurations
          • - No auditing capability
          • - No change history
          • - No release documentation
          • - No way to deploy multiple servers quickly.
          • - No way to rebuild critical servers quickly.
    • System configuration possibilities
      • Install Time Auto-Configuration
        • „ Deploy using Kickstart, Jumpstart etc“
          • + Ensures that new systems are brought into proper state as part of installation process
          • + Possible to deploy multiple servers quickly
          • - No Validation that settings remain correct
          • - No easy way to deploy additional changes
          • - Auto-Configuration process is different between the sites  leads to different configurations
          • - No Auditing
          • - No change history
          • - No release documentation
    • Our Solution Web Portal Puppet Configuration DB CMDB/SAM DHCP/DNS .. Harmonized
    • Infineon Puppet Infrastructure
      • Puppeteer (Puppet Master of site PM’s)
      • Puppet Master per site (or based on size/load)
      • SQL Database for Web portal, Inventory
      • Configuration database (Version control & Ticketing system)
    • Configuration Database - overview
      • Consists of all Global (aka standard) configuration.
      • Consists of all Sites customizations (or global configuration overrides).
      • Access Control, Change Log
      • Using Version Control
      • Each site subscribe to a set of configurations in a certain version
      • Instruct Puppet what to implement on each system
      • Ticketing System, Milestone management (e.g. downtimes) etc
    • Modules
      • Module – A bundle of settings related to a single topic
      • At Infineon we use 3 types of modules:
        • Host Type - e.g. Puppet Master, Compute nodes…
        • Service modules - used by other modules, e.g. ssh, syslog, x11 etc.
        • Sites Modules - sites customizations
      • Every change in a module, requires a check in action, which is identified by a user, and a comment (change log)
      • A user access restriction is applied to each module and to the tagging mechanism
    • Modules
      • Host-type modules - Primary function of the host
        • Each host is associated with one host type
        • Always inherits the Host base class for common policy
      • Service types modules – shared modules
        • Avoids repeating the same code again and again
        • Breaks down the scope of each module, allowing different module owners per topic
        • Changes requires more attention as many host types might use the same service module
        • Should be plug-and-play
    • Site Modules
      • The only place a site can edit his settings (Sites can not change global modules)
      • Each site can only change his own site module
      • Site modules provide a mechanism to override the global modules in a visible and auditable way
      • Two types of overrides:
        • Configuration file override
          • Host level or site level
        • Puppet manifest override – by using class inheritance
      • As Site modules are not tagged (versioned) any change in the site specific settings, will be rolled out immediately .
    • Module Versioning
      • Versioning – The ability to refer (tag) to a certain state/PiT
        • Host Type and Service modules are associated with a version, e.g. RD-Compute Node v1.0
        • Site Modules are not tagged in order to reduce complexity
      • Why ?
        • Allows each module owner to “release” testing and stable versions of his module
        • Sites subscribes to modules in a certain version, allowing each site to select which and when changes will be applied
    • Environments
      • Environments – The ability to select a group of modules in a certain version
      • Each site is associated with at least two environments – Production and Testing:
        • Testing – an environment which refer to versioned modules under a test or productive tag.
        • Production – A selection of modules in a certain version that has already been approved for Production usage.
      • Development environment – The state where the modules are in the latest check-in version, but are not tagged (e.g. not a testing candidate yet).
      • Development  Testing  Production ( Per Host Type )
    • Site modules – complete freedom
      • Configuration file overrides
        • Admins don’t need to know the puppet language in order to change a configuration file (e.g. ssh) per host or for the whole site
        • Each configuration file override, is located in a special directory structure – visible to the application / module owner.
      • Manifest override
        • Sites can use puppet class inheritance in order to override any host-type or service type module definition
      • Any other way of changing configuration files, will be reverted by puppet
    • Global Environments
      • We also have global environments, this is needed for services which are identical across the sites but and not under the site control, for example puppetmasters, ldap servers etc.
    • Auditing
      • Each module owner should review the local sites adaptation, to find out the reasons for change
        • Improve communications
        • Improve host-type / service module quality
      • Auditing ensures that our infrastructure remains harmonized
    • Repository Layout 01.10.2009
    • Puppet Masters
      • Fully automated
      • Provides Stable and Development platforms
      • Automatically manages Puppet environments
      • No Login is allowed except for maintenance work on production boxes
      • Provides Full deployment services
      • Performance monitoring using collectd
      • We don’t use Storeconfigs  custom importers to DB.
      01.10.2009
    • Example Site environment
      • ####################################################### Setup for site BUC ########################################################
      • module_dir{"/etc/puppet/env/buc": type => "stable" }#
      • # ####################################################### setup all basic common services. ########################################
      • modules {"buc-autofs": module => "autofs", site => "buc", type => "services", version => "0.1" }#
      • modules {"buc-cups": module => "cups", site => "buc", type => "services", version => "0.1" }#
      • modules {"buc-host-base": module => "host-base", site => "buc", type => "services", version => "0.12" }#
      • modules {"buc-ldap": module => "ldap", site => "buc", type => "services", version => "0.11" }#
      • modules {"buc-monit": module => "monit", site => "buc", type => "services", version => "0.1" }#
      • modules {"buc-mysql": module => "mysql", site => "buc", type => "services", version => "0.1" }#
      • modules {"buc-redhat": module => "redhat", site => "buc", type => "services", version => "0.12" }#
      • modules {"buc-sendmail": module => "sendmail", site => "buc", type => "services", version => "0.13" }#
      • modules {"buc-ssh": module => "ssh", site => "buc", type => "services", version => "0.1" }#
      • modules {"buc-subversion": module => "subversion", site => "buc", type => "services", version => "0.11" }#
      • modules {"buc-sudo": module => "sudo", site => "buc", type => "services", version => "0.1" }#
      • modules {"buc-syslog-ng": module => "syslog-ng", site => "buc", type => "services", version => "0.1" }#
      • # ######################################################### site specific module ##################################################
      • modules {"buc": module => "bucharest", site => "buc", type => "site" }#
      • # #################################################### Supported host types in Bucharest ##########################################
      • modules {"buc-rd-compute": module => "host-rd-compute", site => "buc", version => "0.12" }#
      • modules {"buc-rd-login-server": module => "host-rd-login-server", site => "buc", version => "0.11" }#
      • ########################################################### End site BUC ##########################################################
    • Passenger vs. Mongrel
      • Mongrels requires more memory, works well with monit as a watchdog
      • Passenger requires a bit more CPU, but reduced the complexity all together
      • Java based solution?
    • External Nodes
      • Enforcing Naming standards
      • Host name implies host type
      • Store exceptions in DNS TXT records
      • Today, we would have simply get YAML output over HTTP (from out DB) and store it locally for cache / failures.
    • Certificate Authority
      • Centralized CA & Chained CA
        • Certificate Revocation List
        • Conflicting Certificate Serial Numbers
        • Initial Puppet run require that CA is available
      01.10.2009
    • What’s Missing
      • One Time actions / Emergency Operations
        • Restart a service
        • Temp disable of a service
        • ..
    • G lobal I nfineon N etwork I nstaller
      • Simple (&Single) Interface to Administrators
      • Hands off server installation
      • Automatic creation of DNS and DHCP entries.
      • Automatic host name allocations
      • DHCP Subnet management
      • IP Address management
      • Kernel Options (e.g. Serial console)
      • Service processor management (when supported by the hardware)
      • Automatic / Manual Hard disk partition
      01.10.2009
    • G lobal I nfineon N etwork I nstaller -continued
      • Switching between Execution mode (Production, Testing or development machine)
      • Systems Inventory Provide Full Hardware information (Model, Type, Serial numbers, CPU, Memory........)
      • User Management Individual access permissions per site and per activity (LDAP / AD)
      • Puppet monitoring Email notifications are dispatched at various stages of the host commissioning process
      • Puppet activity, or lack of it, is detected and displayed within the Gini console
      • Manifest execution issues are visible against the host experiencing them.
      01.10.2009
    • Foreman
      • Unattended installations – manages everything to get puppet going
        • RHE3,4,5 Debian / Ubuntu and Solaris (kickstart, preseed,jumpstart)
        • PuppetCA
        • TFTP
        • DHCP/DNS…
      • External Nodes management
        • Dynamic lookup (similar to extlookup)
      • Puppet Inventory (through facts)
        • With or without storeconfigs
      • Puppet Reports
      • For more info - http:// theforeman.org /
    • Backup slides
    • Repository Layout 01.10.2009
      • No Manual changes
      • “ Global Administrators" are the only one to control their respective parts in the manifest
      • “ Site administrators” should have a way to rollout changes immediately and override global settings if something breaks (like they used to before puppet)
      • Everything must be visible to every administrator
      • Every change must be tracked
      • Must provide a release cycle
      • Must be easy to audit
    • Change Management Global Module Owner / Team Global Local Tests Evaluations Audit Review Global Rollout Module is productive Site Communications Change Request Rollout Request Audit Request Local Test Rollout Operation Local Evaluation Implement locally Global Standard Change Request
    • Gini Screenshots 01.10.2009
    • Gini Screenshots 01.10.2009
    • Gini Screenshots 01.10.2009
    • Gini Screenshots 01.10.2009
    • Gini Screenshots 01.10.2009
    • Gini Screenshots 01.10.2009
    • What do I want to talk about
      • Team work
        • Switch from site specific solutions  transparency
        • Module owners  choose the best person for the job
          • Module team  Align with functional organization.
          • Other IT colleagues are now customers of the module owners
      • Technical stuff
        • Development and testing methods
        • Changes coordination (e.g. downtimes)
        • Automated build processes  Gini  Foreman.
    • What do I want to talk about
      • How Puppet helped us to standardize, communicate and work together
      • Where we were coming from, which problems puppet solved and which new problems puppet introduced and how we solved them
      • How to manage a large repo with many puppet admins How important is it to automate your puppetmasters as well How to coordinate when changes are actually pushed to the clients
      • Why we built Gini, and how those ideas became Foreman.