PuppetCamp - How Puppet helped us to standardize, communicate and work together

Puppet As A Framework [email_address] 10/01/2009

Infineon at a Glance The Company
Infineon provides semiconductor and system solutions, focusing on three central needs of our modern society: energy efficiency, communications and security
Revenue in FY 2008: 4.321 billion EUR
Some 26,000 employees worldwide (as of June 2009)
Strong technology portfolio with about 22,900 patents and patent applications
More than 30 major R&D locations
Germany‘s largest semiconductor company

Focus Areas and Target Markets Industrial & Multimarket Automotive Wireline Communications Wireless Solutions Chip Card & Security Energy Efficiency Communications Security 9-months FY09 revenue split Sale of WLC signed July 7

Where we are coming from
Many Locations and Data Centres
Many administrators
Different tools
Short maintenance window on our servers
Never on the same time across the different sites
Each host type might have a different maintenance window.
You don't understand our requirements
You don't understand our solutions
Our solutions wont fix your problems
My setup is different then yours
We are in different timezones
We don't want to change

Project Goals
Harmonization - Same look and feel everywhere (enables consolidation / global operation )
Simplify Servers management
Implement and maintain Global standards
Accurate Configuration Database
Rapid recovery and deployment of services globally
Eliminates duplicate / unnecessary effort
Audits / Compliance
Go home early

Typical System Lifecycle Pre Installation CMDB DNS,DHCP Network … Initial Configuration Kickstart Jumpstart Custom Scripts Fixes Updates Audits

System configuration possibilities
Manual System Configuration
„ log in and „just do it!““
+ OK for small number of servers
+ Quick and easy
- Very difficult to ensure similar configurations
- No auditing capability
- No change history
- No release documentation
- No way to deploy multiple servers quickly.
- No way to rebuild critical servers quickly.

System configuration possibilities
Install Time Auto-Configuration
„ Deploy using Kickstart, Jumpstart etc“
+ Ensures that new systems are brought into proper state as part of installation process
+ Possible to deploy multiple servers quickly
- No Validation that settings remain correct
- No easy way to deploy additional changes
- Auto-Configuration process is different between the sites  leads to different configurations
- No Auditing
- No change history
- No release documentation

Our Solution Web Portal Puppet Configuration DB CMDB/SAM DHCP/DNS .. Harmonized

Infineon Puppet Infrastructure
Puppeteer (Puppet Master of site PM’s)
Puppet Master per site (or based on size/load)
SQL Database for Web portal, Inventory
Configuration database (Version control & Ticketing system)

Configuration Database - overview
Consists of all Global (aka standard) configuration.
Consists of all Sites customizations (or global configuration overrides).
Access Control, Change Log
Using Version Control
Each site subscribe to a set of configurations in a certain version
Instruct Puppet what to implement on each system
Ticketing System, Milestone management (e.g. downtimes) etc

Modules
Module – A bundle of settings related to a single topic
At Infineon we use 3 types of modules:
Host Type - e.g. Puppet Master, Compute nodes…
Service modules - used by other modules, e.g. ssh, syslog, x11 etc.
Sites Modules - sites customizations
Every change in a module, requires a check in action, which is identified by a user, and a comment (change log)
A user access restriction is applied to each module and to the tagging mechanism

Modules
Host-type modules - Primary function of the host
Each host is associated with one host type
Always inherits the Host base class for common policy
Service types modules – shared modules
Avoids repeating the same code again and again
Breaks down the scope of each module, allowing different module owners per topic
Changes requires more attention as many host types might use the same service module
Should be plug-and-play

Site Modules
The only place a site can edit his settings (Sites can not change global modules)
Each site can only change his own site module
Site modules provide a mechanism to override the global modules in a visible and auditable way
Two types of overrides:
Configuration file override
Host level or site level
Puppet manifest override – by using class inheritance
As Site modules are not tagged (versioned) any change in the site specific settings, will be rolled out immediately .

Module Versioning
Versioning – The ability to refer (tag) to a certain state/PiT
Host Type and Service modules are associated with a version, e.g. RD-Compute Node v1.0
Site Modules are not tagged in order to reduce complexity
Why ?
Allows each module owner to “release” testing and stable versions of his module
Sites subscribes to modules in a certain version, allowing each site to select which and when changes will be applied

Environments
Environments – The ability to select a group of modules in a certain version
Each site is associated with at least two environments – Production and Testing:
Testing – an environment which refer to versioned modules under a test or productive tag.
Production – A selection of modules in a certain version that has already been approved for Production usage.
Development environment – The state where the modules are in the latest check-in version, but are not tagged (e.g. not a testing candidate yet).
Development  Testing  Production ( Per Host Type )

Site modules – complete freedom
Configuration file overrides
Admins don’t need to know the puppet language in order to change a configuration file (e.g. ssh) per host or for the whole site
Each configuration file override, is located in a special directory structure – visible to the application / module owner.
Manifest override
Sites can use puppet class inheritance in order to override any host-type or service type module definition
Any other way of changing configuration files, will be reverted by puppet

Global Environments
We also have global environments, this is needed for services which are identical across the sites but and not under the site control, for example puppetmasters, ldap servers etc.

Auditing
Each module owner should review the local sites adaptation, to find out the reasons for change
Improve communications
Improve host-type / service module quality
Auditing ensures that our infrastructure remains harmonized

Repository Layout 01.10.2009

Puppet Masters
Fully automated
Provides Stable and Development platforms
Automatically manages Puppet environments
No Login is allowed except for maintenance work on production boxes
Provides Full deployment services
Performance monitoring using collectd
We don’t use Storeconfigs  custom importers to DB.
01.10.2009

Example Site environment
####################################################### Setup for site BUC ########################################################
module_dir{"/etc/puppet/env/buc": type => "stable" }#
# ####################################################### setup all basic common services. ########################################
modules {"buc-autofs": module => "autofs", site => "buc", type => "services", version => "0.1" }#
modules {"buc-cups": module => "cups", site => "buc", type => "services", version => "0.1" }#
modules {"buc-host-base": module => "host-base", site => "buc", type => "services", version => "0.12" }#
modules {"buc-ldap": module => "ldap", site => "buc", type => "services", version => "0.11" }#
modules {"buc-monit": module => "monit", site => "buc", type => "services", version => "0.1" }#
modules {"buc-mysql": module => "mysql", site => "buc", type => "services", version => "0.1" }#
modules {"buc-redhat": module => "redhat", site => "buc", type => "services", version => "0.12" }#
modules {"buc-sendmail": module => "sendmail", site => "buc", type => "services", version => "0.13" }#
modules {"buc-ssh": module => "ssh", site => "buc", type => "services", version => "0.1" }#
modules {"buc-subversion": module => "subversion", site => "buc", type => "services", version => "0.11" }#
modules {"buc-sudo": module => "sudo", site => "buc", type => "services", version => "0.1" }#
modules {"buc-syslog-ng": module => "syslog-ng", site => "buc", type => "services", version => "0.1" }#
# ######################################################### site specific module ##################################################
modules {"buc": module => "bucharest", site => "buc", type => "site" }#
# #################################################### Supported host types in Bucharest ##########################################
modules {"buc-rd-compute": module => "host-rd-compute", site => "buc", version => "0.12" }#
modules {"buc-rd-login-server": module => "host-rd-login-server", site => "buc", version => "0.11" }#
########################################################### End site BUC ##########################################################

Passenger vs. Mongrel
Mongrels requires more memory, works well with monit as a watchdog
Passenger requires a bit more CPU, but reduced the complexity all together
Java based solution?

External Nodes
Enforcing Naming standards
Host name implies host type
Store exceptions in DNS TXT records
Today, we would have simply get YAML output over HTTP (from out DB) and store it locally for cache / failures.

Certificate Authority
Centralized CA & Chained CA
Certificate Revocation List
Conflicting Certificate Serial Numbers
Initial Puppet run require that CA is available
01.10.2009

What’s Missing
One Time actions / Emergency Operations
Restart a service
Temp disable of a service
..

G lobal I nfineon N etwork I nstaller
Simple (&Single) Interface to Administrators
Hands off server installation
Automatic creation of DNS and DHCP entries.
Automatic host name allocations
DHCP Subnet management
IP Address management
Kernel Options (e.g. Serial console)
Service processor management (when supported by the hardware)
Automatic / Manual Hard disk partition
01.10.2009

G lobal I nfineon N etwork I nstaller -continued
Switching between Execution mode (Production, Testing or development machine)
Systems Inventory Provide Full Hardware information (Model, Type, Serial numbers, CPU, Memory........)
User Management Individual access permissions per site and per activity (LDAP / AD)
Puppet monitoring Email notifications are dispatched at various stages of the host commissioning process
Puppet activity, or lack of it, is detected and displayed within the Gini console
Manifest execution issues are visible against the host experiencing them.
01.10.2009

Foreman
Unattended installations – manages everything to get puppet going
RHE3,4,5 Debian / Ubuntu and Solaris (kickstart, preseed,jumpstart)
PuppetCA
TFTP
DHCP/DNS…
External Nodes management
Dynamic lookup (similar to extlookup)
Puppet Inventory (through facts)
With or without storeconfigs
Puppet Reports
For more info - http:// theforeman.org /

Backup slides

Repository Layout 01.10.2009

No Manual changes
“ Global Administrators" are the only one to control their respective parts in the manifest
“ Site administrators” should have a way to rollout changes immediately and override global settings if something breaks (like they used to before puppet)
Everything must be visible to every administrator
Every change must be tracked
Must provide a release cycle
Must be easy to audit

Change Management Global Module Owner / Team Global Local Tests Evaluations Audit Review Global Rollout Module is productive Site Communications Change Request Rollout Request Audit Request Local Test Rollout Operation Local Evaluation Implement locally Global Standard Change Request

Gini Screenshots 01.10.2009

What do I want to talk about
Team work
Switch from site specific solutions  transparency
Module owners  choose the best person for the job
Module team  Align with functional organization.
Other IT colleagues are now customers of the module owners
Technical stuff
Development and testing methods
Changes coordination (e.g. downtimes)
Automated build processes  Gini  Foreman.

What do I want to talk about
How Puppet helped us to standardize, communicate and work together
Where we were coming from, which problems puppet solved and which new problems puppet introduced and how we solved them
How to manage a large repo with many puppet admins How important is it to automate your puppetmasters as well How to coordinate when changes are actually pushed to the clients
Why we built Gini, and how those ideas became Foreman.

PuppetCamp - How Puppet helped us to standardize, communicate and work together

by ohadlevy on Oct 11, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 15

Statistics

PuppetCamp - How Puppet helped us to standardize, communicate and work together — Presentation Transcript