Deploying datacenters with Puppet - PuppetCamp Europe 2010

Objective of this PresentationShare to the Puppet community the solution arrived by NYSE Euronext’sSystem Architecture and Engineering (SAE) team to address the technicalchallenges imposed by the Business Strategy of the company:• Consolidate various Data Centers across the globe in two brand-new state-of-the-art facilities, one in the US and another in Europe;• Adapt a Global standard process to build and configure servers and applications across multiple countries and teams;• Quickly deploy changes in large scale in an automated way.

Main Tools Used in the Solution• RHEL Kickstart – A single, very frugal kickstart profile for each RHEL release for the entire company.• RH Network Satellite – Manage packages of the OS, Third Parties and Home-grown RPMs. – Manage patches and upgrades.• Puppet – Work horse of this model – Common framework to apply and configure the Global standard server configuration and particularities of each environment and applications.

The Three Layer ApproachWe have organized our configuration needs in three levels:Base: All NYX global customizations on top of the RHEL default settings.These must be propagated across the enterprise. Example: global standard size for /usr file system, kernel parameter kernel.core_pattern = /var/crash/core.%e.%p.%hZone (or environment or network): Configuration items common to a specificnetwork or environment. Zone layer inherits the Base layer with the ability tooverride anything if necessary. Example: /etc/resolv.conf file across one given production network.Application: Any specific configuration required by any given application.This layer inherits the Zone layer and has the ability to change anything seton the previous two layers. Example: UTP application requires a 50GB /appl file system and a “qt” package installed.

The Puppet Modules behind of this approach

An example of the modules-base manifests• NYX Global Build customizes 250+ items on top of RHEL default OS• All servers include the entire modules-base via the module name ‘base’• Example:• class base::setup { include ssh::setup include banners::setup include resolver::setup include ntp::setup include sysctl::setup include profile::setup include services::setup include locale::setup include email::setup include yum::setup include crashdump::setup include bootloader::setup include rclocal::setup include sudo::setup include hardware::setup include tcpwrappers::setup include firewall::setup include etc_services::setup } 6

An example of the modules-base manifests class sysctl::setup { # Configure kernel core dumps sysctl { "kernel.core_pattern": val => “/var/crash/core.%e.%p.%h", ensure => present, } } class profile::setup { # Deploy custom /etc/bashrc file { "/etc/profile": owner => root, group => root, mode => 0644, source => "puppet:///profile/etc_profile", } } 7

An example of the modules-zones manifests The organization has 20+ zones (and growing) across the globe, includingproduction, QA, development networks, which one with own settings.Example of a zone setting:class trading::setup { $country = us $searchpath = nyx.com $domain = nyx.com $nameservers = [ "10.0.X.X", "10.0.X.X" ] $timeservers = [ "10.0.X.X version 3 prefer", "10.0.X.X" ] $snmp_ro_communities = [ "dfasdfasdfasdf" ] $snmp_rw_communities = [ "sdfasdfasdfasd 127.0.0.1" ] $snmp_dlmods = [ "cmaX /usr/lib64/libcmaX64.so" ] $trapsink = [ "sdfasdfasd" ] $syslogserver = "10.0.X.X" $postfix_mailhost = "mailrelay.nyx.com" $ilo_licensed = "true" $ilo_password = "my_password" # Including the base classes include base::setup}

An example of the modules-app manifests• 60+ applications requiring their own settings• We try to configure the application puppet manifest to be pluggable on any zoneExample of one application manifest:class sg::setup { package { ["libxslt", "libmng"]: ensure => present, } lvm::config_lvm { "tcpdump": mount_point => "/tcpdump", lvname => "ltcpdump", vgname => vg00, lvsize => "5G", fstype => "ext3", owner => “myuser", group => "users", mode => "0755", }}

How the three layers work together•The zone class usually sets variables that will be used by modules-base forconfiguration.•The zone class necessarily includes the modules-base.•All servers must belong to one zone.•The application module is coded to be pluggable to any zone (i.e. matchengine application module can be on a QA environment or productionenvironment)•The ability to build anything, anywhere and mimic Production config to QAenvironment and vice-versa.•The modules-app can override settings inherit on modules-zones. Themodules-zones can override settings inherit on modules-base.• Using puppet, we synchronize the build server process with the latestrequirements of the applications.

The Last Piece: the Node Definition FileThe node definition is the entity that matches the hostname to the zone andapplication modules. In addition, it has the networking configuration of the server. node "buildtest.nyx.com" { # Zone Class include corplan::setup # Application Class include sg::setup # Networking network::interface::setup {"bond0": ensure => present, config_type => bonded, ip => "10.0.0.100", netmask => "255.255.255.0", slave0 => "eth0", slave1 => "eth1", gateway => "10.0.0.254", arp_target => "10.0.0.254", arp_interval => "3000", restart_network => false, } }

NYX Build Server Components• DHCP/DNS Server (optional)• TFTP / PXE Server• Satellite Server or Proxy• Apache Server• Puppet Server• MySQL Server• The server resides on a separate network called build network since wedo not build any server on productions network.

Automated Installation Process - Overview PreBoot Phase Default kickstart On the Build Server, the label that register script generates a PXE Boot for 1st time executes a perl customized PXE file after script via “wget” looking up Serial Number in to register the mysql database. Custom Serial Number PXE has hostname and OS and the MAC version (RHEL4 or RHEL5). address and Server out of the reboots server. box Phase 0 Start the RHEL The kickstart post install script installation with register the new server on the PXE Boot for 2nd time the minimum satellite server and assign it to packages and the appropriate channels. Global Standard Puppet client is installed just File Systems before server is rebooted. setup and sizes. Server come back from the first reboot Continue

Automated Installation Process – Overview Cont. Phase 1 Load lean OS from disk + init script An init.d script A node definition is runs puppet for mandatory at this point and the first time and puppet client configures downloads its own server to the NYX Global configuration from base (default) configuration the puppet server. and networking.Server reboots for the thirdtime Phase 2 On Phase2 is The phase 2 execution can be Phase2 where Puppet is triggered directly from Phase1 init script executed for the init script (in this case is last time to executed on the build network) configure the or can be executed once the server to their final server is on the production zone network. (environment) and application Server might reboot or settings. execute Phase2 init.d script directly

PreBoot Phase• Very simple kickstart profile launched by default PXE label• It does nothing than executing a script via wget• It registers the server with two parameters: Serial Number and the MAC• The script will create a custom PXE boot file with given MAC that containsthe RHEL release and the hostname of the server.• Example of the kickstart pre-script: %pre SERVER=`grep url address /tmp/anaconda.log | awk {print $6}` cd /tmp && wget -q http://${SERVER}/pub/dmidecode chmod +x /tmp/dmidecode SN=`dmidecode | grep "Serial Number" | head -1 | awk {print $3}` MAC=`ifconfig | grep HWaddr | awk {print $5}` wget -q http://${SERVER}/cgi-bin/register.pl?mac=$MAC&sn=$SN reboot

Phase 0• A single kickstart profile for each RHEL release to the entire organization• Very minimum number of packages installed• Larger OS File system sizes than the RHEL Default• Server is registered to the Satellite• Puppet client is installed (puppetd)• NYX-Firstscript is set to run on the next reboot via init.d

Phase 0 – Fragment of the Kickstart profile%post --interpreter /bin/shecho "Installing puppet..."yum -y install puppet facter ruby-rdoc ruby-shadow# Puppet is install via the activation key but by default will run as a daemonecho "Chkconfiging puppet off..."chkconfig puppet off# Grab nyx-firstboot script from the satellite or proxyRHN_SERVER=`grep ^serverURL= /etc/sysconfig/rhn/up2date | cut -d/ -f3`echo "Detected Puppetmaster/RHN as: $RHN_SERVER"echo "Downloading nyx-firstboot script..."wget -O /etc/init.d/nyx-firstboot http://$RHN_SERVER/pub/glb/nyx- firstboot# Make it executable and make it start on firstbootecho "Installing nyx-firstboot..."chmod 755 /etc/init.d/nyx-firstbootchkconfig --add nyx-firstboot

Phase 1• It is run by the nyx-firstboot init script• nyx-firstboot executes puppetd twice: one to configure itself and another toset the server to the base config.• nyx-firstboot script command lines that executes puppetd# Get puppet server address, the same as the RHN Proxy/RHN SatellitePUPPET_SERVER=`grep ^serverURL= /etc/sysconfig/rhn/up2date | cut-d/ -f3`# Run puppet once to configure itself and get SSL certs auto-signedpuppetd -tov --server=$PUPPET_SERVER --tagsutilities::puppetclientbootstrap --no-noop# Run puppet again to apply all base build manifestsFACTER_sys_building=true puppetd -tov --server=$PUPPET_SERVER --tagssys_building --no-noop

Phase 2 (Last phase)The nyx-phase2 script runs puppetd again, this time puppetd fully executesthe puppet manifests which configures the server entirely to the zone andapplication. The nyx-phase2 is set to execute automatically from init.d ORtriggered automatically by nyx-firstboot.Fragment of the script:# Run puppet to apply application specific manifestspuppetd -tov --no-noop | /usr/bin/tee /tmp/puppet2bif egrep -q "err:|warning:" /tmp/puppet*; then logger -t nyx-phase2 "Found excessive errors on puppet logs!"else # Shutdown the server - to be shipped to the rack logger -t nyx-phase2 "Shutting down the system! No errors found onthe puppet logs!" /usr/bin/powerofffi

Data Center Deployment: Numbers and Lessons• One server gets completely installed in 20 minutes in unattended mode• We usually install up to 15 servers in parallel, but we had to upgrade first thepuppet master to run on Apache Web Server. The default web server has shown notto be very powerful.• Usually, one individual in one day can install up to 50 servers by himself• The most difficult part was getting the requirements of each application

Ancillary Tools for This Project• Subversion (extremely important): – we had to disseminate the culture of version control among many individuals and teams. – Each layer of the modules and node definitions have its own subversion repositoriy• Web Subversion• OpenGrok (search the manifests)• Redmine (ticket and changes control)• Tidal Scheduler – To execute puppet client once the servers are in the productionnetwork

Next Steps• Run puppetd to deploy changes across the board on the day-by-day basis. This willrequire more a cultural change and coordination across more departments.• Run puppetd in report mode daily (no-intrusive) to detect discrepancies betweenthe reality and the puppet manifests• Use puppet dashboard as a central database of Node definitions (called ExternalNode Classifier)• Create more custom facts and types in ruby (diminish the number of execs insidethe manifests).• Use of external resources to automate other configuration such as Veritas Cluster.

Thank you! Q&A

Deploying datacenters with Puppet - PuppetCamp Europe 2010

by Puppet Labs on Jul 31, 2012

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Deploying datacenters with Puppet - PuppetCamp Europe 2010 Presentation Transcript