A Presentation about Puppet that I've made at the OSSPAC conference

6,015 views
5,878 views

Published on

2 Comments
6 Likes
Statistics
Notes
No Downloads
Views
Total views
6,015
On SlideShare
0
From Embeds
0
Number of Embeds
21
Actions
Shares
0
Downloads
277
Comments
2
Likes
6
Embeds 0
No embeds

No notes for slide
  • So, how do we do handle this process today At first, we need to input the server details in a few different locations (CMDB, DHCP, DNS etc)
  • So, how do we do handle this process today At first, we need to input the server details in a few different locations (CMDB, DHCP, DNS etc)
  • Some sites have a very optimized process, Effort involved Time, new hardware, downtimes, patching/updating, installation of new services (global rollouts….) TESTING Today, there is no easy way for a global team to push changes – e.g. clearcase team needs to ask each operation team to implement the changes “auditing” is a manual process – no transparency On a global scale, we don’t really know the configuration of each site SOX etc
  • Database is currently centralized. Performance issues? DNS – A, PTR, CNAME, TXT. DHCP – BOOT host, file, Vendor classes of sparc. Replaces RARP and bootparams LOGS – Updates the Gini DB with live data LSF – Opens and closes hosts during the build TFTP – manipulates the boot link based upon the host status CERTIFICATES – cleans old certificates and allows autosign CMDB – Updates the CMDB with live data
  • A Presentation about Puppet that I've made at the OSSPAC conference

    1. 1. Infrastructure Automation Ohad Levy – Infineon Technologies [email_address]
    2. 2. Automated <ul><li>”… a process which may once have been performed manually but has been altered in some way which allows a machine or computer to either wholly or partially manipulate the process to save time” </li></ul>
    3. 3. Infrastructure <ul><li>“ Infrastructure is generally a set of interconnected structural elements that provide the framework supporting an entire structure” </li></ul>
    4. 4. Automated Infrastructure <ul><li>“ Having the basic services necessary for your infrastructure to operate largely without the aid of a keeper.” </li></ul>
    5. 5. Grow – from This: http://i68.photobucket.com/albums/i2/RealConnections/googlecomputers.jpg
    6. 6. Large Scale Environment
    7. 7. Where does it fit in?
    8. 8. Typical System Lifecycle Pre/ Installation DNS/DHCP … Initial Configuration Kickstart Jumpstart Custom Scripts Fixes Updates Audits
    9. 9. System configuration possibilities <ul><li>Manual System Configuration </li></ul><ul><ul><li>„ log in and „just do it!““ </li></ul></ul><ul><ul><ul><li>+ OK for small number of servers </li></ul></ul></ul><ul><ul><ul><li>+ Quick and easy </li></ul></ul></ul><ul><ul><ul><li>- Very difficult to ensure similar configurations </li></ul></ul></ul><ul><ul><ul><li>- No auditing capability </li></ul></ul></ul><ul><ul><ul><li>- No change history </li></ul></ul></ul><ul><ul><ul><li>- No release documentation </li></ul></ul></ul><ul><ul><ul><li>- No way to deploy multiple servers quickly. </li></ul></ul></ul><ul><ul><ul><li>- No way to rebuild critical servers quickly. </li></ul></ul></ul>
    10. 10. System configuration possibilities <ul><li>Install Time Auto-Configuration </li></ul><ul><ul><li>„ Deploy using Kickstart, Jumpstart etc“ </li></ul></ul><ul><ul><ul><li>+ Ensures that new systems are brought into proper state as part of installation process </li></ul></ul></ul><ul><ul><ul><li>+ Possible to deploy multiple servers quickly </li></ul></ul></ul><ul><ul><ul><li>- No Validation that settings remain correct </li></ul></ul></ul><ul><ul><ul><li>- No easy way to deploy changes </li></ul></ul></ul><ul><ul><ul><li>- Auto-Configuration process is different between the sites  leads to different configurations </li></ul></ul></ul><ul><ul><ul><li>- No Auditing </li></ul></ul></ul><ul><ul><ul><li>- No change history </li></ul></ul></ul><ul><ul><ul><li>- No release documentation </li></ul></ul></ul>
    11. 11. Typical System Lifecycle Pre/ Installation DNS/DHCP … Initial Configuration Kickstart Jumpstart Custom Scripts Fixes Updates Audits Puppet
    12. 12. Puppet <ul><li>What is Puppet ? </li></ul><ul><ul><li>A product designed to deploy system configurations. It is – </li></ul></ul><ul><ul><ul><li>Open source based on Ruby </li></ul></ul></ul><ul><ul><ul><li>Policy based </li></ul></ul></ul><ul><ul><ul><li>Runs every 30 minutes </li></ul></ul></ul><ul><ul><ul><li>An abstraction layer between the system administrator and the system </li></ul></ul></ul><ul><ul><ul><li>Capable to run on any UNIX operating system </li></ul></ul></ul>
    13. 13. Our Challenges <ul><li>Keep our systems &quot;harmonized“ </li></ul><ul><li>Audit / Know what's going on each system </li></ul><ul><li>Replace a server if it dies or to be able to add another server that is exactly like it </li></ul><ul><li>Similar Applications, different OS's </li></ul><ul><li>Push out changes to all the servers that need a particular change </li></ul><ul><li>Stop duplicating effort </li></ul><ul><li>Go home early </li></ul>
    14. 14. How does Puppet works?
    15. 15. Puppet Types <ul><li>A Type is a particular element that Puppet knows how to configure </li></ul><ul><li>Files (content, permissions, ownership) </li></ul><ul><li>Packages (installations, updates..) </li></ul><ul><li>Services (enabled/disabled, running/stopped) </li></ul><ul><li>Exec (run commands) </li></ul><ul><li>Full List: cron, exec, file, filebucket, group, host, interface, k5login, mailalias, maillist, mount, nagios*, package, service, sshkey, tidy, user, yumrepo, zone </li></ul>
    16. 16. Example: Managing sudoers file <ul><li>file { /etc/sudoers: </li></ul><ul><li>owner => root, </li></ul><ul><li>group => root, </li></ul><ul><li>mode => 600, </li></ul><ul><li>source => puppet://server/files/sudoer </li></ul><ul><li>} </li></ul>
    17. 17. Dependencies <ul><li>&quot; require &quot; and &quot; before &quot; / &quot; after &quot; settings ensures that types are </li></ul><ul><li>applied in the correct order </li></ul><ul><li>file { &quot;/etc/sudoers&quot;: </li></ul><ul><li>... </li></ul><ul><li>require => Package[sudo] </li></ul><ul><li>} </li></ul><ul><li>package { &quot;sudo&quot;: </li></ul><ul><li>ensure => present, </li></ul><ul><li>before => File[&quot;/etc/sudoers&quot;] </li></ul><ul><li>} </li></ul>
    18. 18. Dependencies - continued <ul><li>&quot; notify &quot; and &quot; subscribe &quot; settings can trigger cascaded updates </li></ul><ul><li>Particularly useful in services, exec </li></ul><ul><li>file { &quot;/etc/ssh/sshd_conf&quot;: </li></ul><ul><li>... </li></ul><ul><li>notify => Service[&quot;sshd&quot;] </li></ul><ul><li>} </li></ul><ul><li>service { &quot;sshd&quot;: </li></ul><ul><li> subscribe => File[&quot;/etc/ssh/sshd_conf“] </li></ul><ul><li>} </li></ul>
    19. 19. What is Facter? <ul><li>Facter gathers information about the client, which can be used as variables within puppet. </li></ul><ul><li>You can add custom facts as needed. </li></ul><ul><li>package {&quot;sshd&quot;: </li></ul><ul><li>ensure => installed, </li></ul><ul><li>name => $operatingsystem ? { </li></ul><ul><li>solaris => &quot;IFKLssh&quot;, </li></ul><ul><li>default => “openssh-server” </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
    20. 20. Example Facts <ul><li>$ sudo facter </li></ul><ul><li>architecture => amd64 </li></ul><ul><li>domain => sin.infineon.com </li></ul><ul><li>facterversion => 1.3.8 </li></ul><ul><li>fqdn => sinn1636.sin.infineon.com </li></ul><ul><li>hardwaremodel => x86_64 </li></ul><ul><li>hostname => sinn1636 </li></ul><ul><li>ipaddress => 172.20.89.113 </li></ul><ul><li>kernel => Linux </li></ul><ul><li>kernelrelease => 2.6.24-21-generic </li></ul><ul><li>lsbdistcodename => hardy </li></ul><ul><li>lsbdistdescription => Ubuntu 8.04.1 </li></ul><ul><li>lsbdistid => Ubuntu </li></ul><ul><li>lsbdistrelease => 8.04 </li></ul><ul><li>macaddress => 00:1c:25:14:26:ab </li></ul><ul><li>manufacturer => LENOVO </li></ul><ul><li>max speed => 2000 MHz </li></ul><ul><li>maximum memory module size => 4096 MB </li></ul>maximum total memory size => 8192 MB memoryfree => 988.30 MB memorysize => 1.94 GB operatingsystem => Debian operatingsystemrelease => 2.6.24-21-generic processor0 => Intel(R) Core(TM)2 Duo CPU T7300 @ 2.00GHz processor1 => Intel(R) Core(TM)2 Duo CPU T7300 @ 2.00GHz processorcount => 2 productname => 6458B43 ps => ps -ef puppetversion => 0.24.5 rubysitedir => /usr/local/lib/site_ruby/1.8 rubyversion => 1.8.6 serialnumber => L3A8632 swapfree => 912.23 MB swapsize => 956.99 MB type => Notebook uniqueid => 007f0101 vendor => LENOVO
    21. 21. What is a Class? <ul><li>A named collection of type objects </li></ul><ul><li>Can include or inherit from other classes </li></ul><ul><li>class sudo_class { </li></ul><ul><li>include foo_class </li></ul><ul><li>file { &quot;/etc/sudoers&quot;: </li></ul><ul><li>... </li></ul><ul><li>} </li></ul><ul><li>package { &quot;sudo&quot;: </li></ul><ul><li>... </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
    22. 22. Class inheritance <ul><li>class afile { </li></ul><ul><li>file { &quot;/tmp/foo&quot;: </li></ul><ul><li>ensure => file </li></ul><ul><li>source => &quot;/src/versionA&quot; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>class another_file inherits afile { </li></ul><ul><li>File [&quot;/tmp/foo&quot;] { </li></ul><ul><li>source => &quot;/src/versionB&quot; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
    23. 23. What is a Node ? <ul><li>A configuration block matching a client </li></ul><ul><li>Can contain types, classes </li></ul><ul><li>&quot;default&quot; node matches any client without a node block </li></ul><ul><li>node &quot;ohad.myself&quot; { </li></ul><ul><li>include sudo_class </li></ul><ul><li>include other_class </li></ul><ul><li>} </li></ul>
    24. 24. External Nodes <ul><li>Node definitions can be defined outside of puppet – e.g. LDAP, external script… </li></ul><ul><li>Ideal when you have too many nodes.. </li></ul>
    25. 25. Classes and definitions <ul><li>Classes are groups of resources. </li></ul><ul><li>Definitions are similar to classes, but they can be instantiated multiple times with different arguments on the same node. </li></ul><ul><li>class apache2 { </li></ul><ul><li>define simple-vhost ( $admin = &quot;webmaster&quot;, $aliases, $docroot) { </li></ul><ul><li>file { &quot;/etc/apache2/sites-available/$name&quot;: </li></ul><ul><li>mode => &quot;644&quot;, </li></ul><ul><li>require => [ Package[&quot;apache2&quot;], Service[&quot;apache2&quot;] ], </li></ul><ul><li>content => template (&quot;apache/vhost.conf&quot;), </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
    26. 26. Classes and definitions - Continued <ul><li>node mywebserver { </li></ul><ul><li>include apache2 </li></ul><ul><li>apache2::simple-vhost { &quot;test.example.com&quot;: </li></ul><ul><li>docroot => &quot;/var/www/test“ </li></ul><ul><li>} </li></ul><ul><li>apache2::simple-vhost { &quot;debian.example.com&quot;: </li></ul><ul><li>docroot =>&quot;/var/www/debian“, </li></ul><ul><li>alias => [“debiantest.example.com”,”debian”], </li></ul><ul><li>admin => “system@example.com” </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
    27. 27. vhost.conf template <ul><li>Puppet uses Ruby's ERB template system: </li></ul><ul><li><VirtualHost *> </li></ul><ul><li>ServerAdmin <%= admin %> </li></ul><ul><li>DocumentRoot <%= docroot %> </li></ul><ul><li>ServerName <%= name %> </li></ul><ul><li><% aliases.each do |al| -%> </li></ul><ul><li>ServerAlias <%= al %> </li></ul><ul><li><% end -%> </li></ul><ul><li>ErrorLog &quot;|/usr/bin/cronolog /var/log/apache/<%= </li></ul><ul><li>name %>/%Y-%m/error-%d&quot; </li></ul><ul><li>CustomLog &quot;|/usr/bin/cronolog /var/log/apache/<%= </li></ul><ul><li>name %>/%Y-%m/access %d&quot; sane </li></ul><ul><li></VirtualHost> </li></ul>
    28. 28. Template output <ul><li>more /etc/apache2/sites-available/debian.example.com </li></ul><ul><li><VirtualHost *> </li></ul><ul><li>ServerAdmin [email_address] </li></ul><ul><li>DocumentRoot /var/www/debian </li></ul><ul><li>ServerName debian.example.com </li></ul><ul><li>ServerAlias debiantest.example.com </li></ul><ul><li>ServerAlias debian </li></ul><ul><li>ErrorLog |/usr/bin/cronolog </li></ul><ul><li>/var/log/apache/debian.example.com/%Y-%m/error-%d“ </li></ul><ul><li>CustomLog &quot;|/usr/bin/cronolog </li></ul><ul><li>/var/log/apache/debian.example.com/%Y-%m/access-%d“ sane </li></ul><ul><li></VirtualHost> </li></ul>
    29. 29. OS API - It also works the other way around: <ul><li>$ ralsh user levyo </li></ul><ul><li>user { 'levyo': </li></ul><ul><li>groups => ['tidc'], </li></ul><ul><li>ensure => 'present', </li></ul><ul><li>password => 'x', </li></ul><ul><li>shell => '/bin/bash', </li></ul><ul><li>uid => '49960', </li></ul><ul><li>comment => 'Ohad Levy, +65 xxxx xxxx', </li></ul><ul><li>gid => '11100', </li></ul><ul><li>home => '/home/levyo' </li></ul><ul><li>} </li></ul>
    30. 30. Puppet Modules <ul><li>Modules allow you to group both the logic and the files for an application together. </li></ul><ul><li>Puppet automatically searches its module path to find modules. </li></ul><ul><li>Modules can contain four types of files, each of which must be stored in a separate subdirectory: </li></ul><ul><ul><li>Manifests – must be stored in manifests/, and if you create manifests/init.pp then that file will be loaded if you import that moudle name directly (e.g. import “mymodule”) </li></ul></ul><ul><ul><li>Templates – must be stored under templates/, put in your manifest some where template(“mymodule/mytemplate”) </li></ul></ul><ul><ul><li>Files – stored in files/, can be accessed as: source => puppet://”mymodule”/myfile </li></ul></ul><ul><ul><li>Plugins – Additional types, provides or facts </li></ul></ul>
    31. 31. File server and File Bucket <ul><li>Puppet also includes a file server which you can use for transferring files from the server to the client. </li></ul><ul><li>If you configure it, puppet can also save a backup of each file that is changed on the client to the server. The backups go in a filebucket and can be retrieved later. </li></ul>
    32. 32. Getting Started <ul><li>Install puppet (yum,pkg-get,…) or manually install ruby, facter and puppet </li></ul><ul><li>Setup a puppet server (puppetmaster / puppet-server). </li></ul><ul><li>Write a manifest for your node (classes etc) </li></ul><ul><li>Start puppet master on the server </li></ul><ul><li>Start puppetd on the client </li></ul>
    33. 33. Some more info <ul><li>Puppet uses SSL for all communications, therefor it includes a CA, you can automatically sign CSR or use puppetca tool to mange them. </li></ul><ul><li>Storeconfig, option to save machine states(facts, configuration runs) and special facts (e.g. SSH keys) in a database. </li></ul><ul><li>Reporting, puppet has a few reporting options, most common are emails with changes, RRD files, yaml files and puppetshow web interface. </li></ul><ul><li>Puppet HA, load balancing etc </li></ul>
    34. 34. What are the benefits ? <ul><li>Costs </li></ul><ul><ul><li>Based on open source tools – No license cost </li></ul></ul><ul><ul><li>Reduced Labor  Reduced Cost </li></ul></ul><ul><li>Productivity </li></ul><ul><ul><li>Reproducibility, Accuracy and homogenous environment </li></ul></ul><ul><ul><li>(Reduce needed knowledge to operate our environment) </li></ul></ul><ul><ul><li>Allow easy ramp up of new sites and services </li></ul></ul><ul><li>Change Management </li></ul><ul><ul><li>Testing Processes, Auditing </li></ul></ul><ul><ul><li>Global control over servers </li></ul></ul><ul><ul><li>Harmonization  Consolidation </li></ul></ul><ul><li>… </li></ul>
    35. 35. Conclusions <ul><li>We're all stuck on the hamster wheel </li></ul><ul><li>Makes easy stuff easier, hard stuff possible </li></ul><ul><li>Harmonize! (not just today, also in the future) </li></ul><ul><li>Enable the business for consolidation </li></ul><ul><li>Similar projects </li></ul><ul><ul><li>cfengine </li></ul></ul><ul><ul><li>bcfg2 </li></ul></ul><ul><li>Additional Resources </li></ul><ul><ul><li>http://reductivelabs.com/trac/puppet </li></ul></ul><ul><ul><li>http://reductivelabs.com/trac/puppet/wiki/LanguageTutorial </li></ul></ul><ul><ul><li>http://reductivelabs.com/trac/puppet/wiki/CompleteConfiguration </li></ul></ul><ul><li>#puppet on irc.freenode.org </li></ul>
    36. 36. Backup Slides Additional information
    37. 37. Configuration DB - Terminology <ul><li>Module – A bundle of settings related to a single application </li></ul><ul><li>At Infineon we use 3 types of modules: </li></ul><ul><ul><li>Host Type - e.g. Login Servers, Compute nodes… </li></ul></ul><ul><ul><li>Service - module which is needed by other modules, e.g. ssh, x11 etc. </li></ul></ul><ul><ul><li>Sites Modules - sites customizations </li></ul></ul><ul><li>Versioning – The ability to refer (tag) to a certain state/PiT </li></ul><ul><ul><li>Host Type and Service modules are associated with a version, e.g. Login Server v1.0 </li></ul></ul><ul><ul><li>Site Modules are not tagged in order to reduce complexity </li></ul></ul><ul><li>* Every change in a module, requires a check in action, which is identified by a user, and a comment (change log) </li></ul><ul><li>* A user access restriction should be applied to each module and to the tagging mechanism </li></ul>
    38. 38. Configuration DB - Terminology <ul><li>Environments – The ability to select a group of modules in a certain version </li></ul><ul><li>Each site is associated with at least two environments – Production and Testing: </li></ul><ul><ul><li>Testing – an environment which refer to versioned modules under a test tag. </li></ul></ul><ul><ul><li>Production – A selection of modules in a certain version that has already been approved for Production usage. </li></ul></ul><ul><li>Development environment – The state where the modules are in the latest check-in version, but are not tagged (e.g. not a testing candidate yet). </li></ul><ul><li>Development  Testing  Production (Per Host Type) </li></ul>
    39. 39. Configuration DB - Terminology <ul><li>Global modules / owners </li></ul><ul><ul><li>Each owner is responsible for his module (or modules) </li></ul></ul><ul><ul><li>A special attentions is required for Service module as many other modules might depend on them </li></ul></ul><ul><li>Site modules </li></ul><ul><ul><li>As Site modules are not tagged (versioned) any change in the site specific settings, will be rolled out immediately. </li></ul></ul>
    40. 40. Risks & Challenges <ul><li>If the right processes are not in place, it would be possible to implement the wrong changes globally </li></ul><ul><li>Learning Curve </li></ul><ul><ul><li>How our platform works </li></ul></ul><ul><ul><li>How to use version control </li></ul></ul><ul><ul><li>How to work in the new processes/workflows </li></ul></ul><ul><li>Fear (Am I obsolete?, will I continue to learn?) </li></ul>
    41. 41. Why Did we choose Puppet <ul><li>Open source </li></ul><ul><li>More flexible than other open source alternatives </li></ul><ul><li>Really works (Declarative language, Abstracts configuration as resources, Allows relationships, Transactional, Modular) </li></ul><ul><li>Provides an API to the OS, making it very simple to support other OS’s in the future </li></ul><ul><li>Puppet is now being adopted in many projects </li></ul><ul><li>For more info see http://reductivelabs.com/trac/puppet/wiki/CfengineVsPuppet </li></ul><ul><li>http://reductivelabs.com/trac/puppet/wiki/WhosUsingPuppet </li></ul>
    42. 42. Datacenters Challenges <ul><li>Harmonization </li></ul><ul><li>Simplify Servers management </li></ul><ul><li>Implement and maintain Global standards </li></ul><ul><li>Accurate Configuration Database </li></ul><ul><li>Rapid recovery and deployment of services globally </li></ul><ul><li>Eliminates duplicate / unnecessary effort </li></ul><ul><li>Same look and feel everywhere (enables consolidation / global operation ) </li></ul><ul><li>Audits / Compliance </li></ul>
    43. 43. Infineon Puppet Infrastructure <ul><li>Puppeteer (Puppet Master of site PM’s) </li></ul><ul><li>Puppet Master per site (or based on size/load) </li></ul><ul><li>SQL Database for Web portal, Inventory </li></ul><ul><li>Configuration database (Version control & Ticketing system) </li></ul>
    44. 44. Gini’s Features <ul><li>AD/LDAP Authentication </li></ul><ul><li>DHCP/DNS Connectors (API, + MS(yacks) DHCP, MS(yacks) DNS and infoblox) </li></ul><ul><li>Puppet Host type management </li></ul><ul><li>Disk Layout Management </li></ul><ul><li>Host Hardware management (needed for Solaris DHCP jumpstart, and general host info) </li></ul><ul><li>Multi OS management (e.g. RH, CentOs, Solaris.... in their various versions) </li></ul><ul><li>User role management (who is allowed to install, in which site, which host type etc) </li></ul><ul><li>Subnet and IP Management (auto assignment of IP address, DHCP reservation DNS records etc) </li></ul><ul><li>Multi Domain / site management </li></ul><ul><li>Hands-free auto OS installation (currently Kickstart and Jumpstart (sparc) using DHCP) </li></ul><ul><li>Management of Puppet Environments, choose an env per host (e.g. we have different production states for different sites, or for different host types). </li></ul><ul><li>Management of puppet certificates (also support chained certificates). </li></ul><ul><li>Import of Machine Facts (we cant use store config due to the large setup of our env) </li></ul><ul><li>Nice Graphs (with Gruff) </li></ul><ul><li>Puppet modules selection per host(for those who needs more than the default host type) </li></ul><ul><li>SP (Service Processor, iLOM etc) management </li></ul><ul><li>TFTP management </li></ul><ul><li>Status indications for machine state (pre installation, during puppet first run, etc) </li></ul><ul><li>... and probably a lot of other small stuff that I cant think of now </li></ul>
    45. 45. Our Solution Web Portal Puppet Configuration DB Inventory DHCP/DNS .. Harmonized
    46. 46. Where does GINI live? GINI DB DNS TFTP CERTIFICATES CMDB DHCP LOGS LSF Future work

    ×