Deploying Foreman in Enterprise Environments


Published on

Nils Domrose, inovex GmbH
Config Management Camp
3 and 4 February 2014
Gent, Belgium

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Deploying Foreman in Enterprise Environments

  1. 1. Deploying Foreman in Enterprise Environments best practices and lessons learned… Nils Domrose Gent, February, 3 2014
  2. 2. About me ▸ senior linux systems engineer at inovex GmbH ▸ worked as a network engineer, software developer and systems engineer ▸ using foreman for about 1 year ▸ using bare-metal deployment for ages ▸ life is short – let’s focus on interesting stuff! #irc __endy__ @endyman 2
  3. 3. About inovex We use technology to make our customers happy. And ourselves. ▸ ▸ offices in [‘Pforzheim’, ‘Karlsruhe’, ‘Cologne’, ‘Munich’] ▸ we have open positions... 1 SHAPE Consulting 2 BUILD Application Development 3 RUN IT Engineering & Operations 4 TRACK Business Intelligence 5 TEACH Academy 3
  4. 4. Agenda ▸ ▸ ▸ ▸ ▸ the chicken & egg problem foreman components we are the borg fail! foreman HA best practices 4
  5. 5. The Chicken & Egg Problem ...deploying your deployment infrastructure ▸ some shell scripts to push installer and answers.yaml to new system (installer is subject to rpm/deb packaging) ▸ run a custom kafo installer ▸ setup repos ▸ configure nics (bonding) ▸ setup pacemaker & corosync ▸ setup drbd ▸ setup dhcp server ▸ setup postgreSQL ▸ setup puppetDB ▸ run foreman kafo installer
  6. 6. The Chicken & Egg Problem build your first kafo installer ▸ kafo? ▸ kafo! - imagine master-less puppet plus config data in yaml (like hiera) ▸ can’t tell wether it’s the chicken or the egg or both but it’s cool  ▸ stores parameters for your modules in answers.yaml ▸ simply copy your modules into the installer’s modules directory ▸ or even use puppet-librarian to manage your modules (poor man’s git-submodules)
  7. 7. The Chicken & Egg Problem kafo - getting started ▸ Never give up, never surrender!: depending on your kafo gem it might not (yet) be as easy as: root@foreman:~# cd /var/tmp/ root@foreman:/var/tmp# i=my-installer; mkdir $i; cd $i; kafofy $i ▸ It’s more like: i=my-installer; mkdir $i && cd $i && kafofy $i && mkdir /etc/$i && cat > /etc/$i/$i.yaml << EOF > --> :log_dir: /var/log/$i > :log_name: configuration.log > :log_level: info > :no_prefix: false > :mapping: {} > :answer_file: /etc/$i/answers.yaml > :installer_dir: $(pwd) > :modules_dir: $(pwd)/modules > :default_values_dir: /tmp > :colors: true > :kafo_modules_dir: /usr/lib/ruby/vendor_ruby/kafo/modules > EOF
  8. 8. The Chicken & Egg Problem get the modules ▸ delete the modules directory root@foreman:/var/tmp/my-installer# rm -Rf modules/ ▸ create a Puppetfile root@foreman:/var/tmp/my-installer# cat Puppetfile forge "" mod 'puppetlabs/stdlib' mod 'ntp', :git => 'git://' ▸ run the librarian root@foreman:/var/tmp/my-installer# librarian-puppet install --clean root@foreman:/var/tmp/my-installer# ll modules total 16 drwxr-xr-x 4 root root 4096 Jan 29 12:25 ./ drwxr-xr-x 7 root root 4096 Jan 29 12:17 ../ drwxr-xr-x 7 root root 4096 Jan 29 12:25 ntp/ drwxr-xr-x 6 root root 4096 Jan 29 12:24 stdlib/ ▸ next: package it, install it
  9. 9. Foreman Components foreman ▸ ▸ ▸ ▸ ▸ rails application apache mod_passenger Webinterface REST API unattended resources – rendered templates database REST API unattended resources foreman frontend apache mod_passenger Port 80 HTTP Port 443 HTTPS 9
  10. 10. Foreman Components smart-proxy ▸ WEBrick based REST server to manage supporting infrastructure Port 8443 HTTPS WEBrick server puppet CA puppet apache mod_passenger Port 8140 HTTPS BMC DHCP TFTP DNS ipmitool ISC dhcpd tftpd nsupdate Port 67 UDP Port 69 UDP 10
  11. 11. Foreman Components placement of foreman components ▸ ▸ ▸ ▸ flows at each side to local puppet and smart proxy all VLANs directly access puppet ca all VLANs directly access foreman unattended resources we need DNS only once (so let’s limit TSIG key distribution) VLAN D Site2 mgmt VLAN2 fw VLAN C Site2 Smartproxy, puppet, DHCP VLAN A Site1 mgmt VLAN1 fw Foreman Smartproxy, puppet, CA, DNS, DHCP VLAN B Site1 11
  12. 12. Foreman Components optimizing comms flows ▸ limit access to the smartproxy and foreman to local clients only ▸ open dedicate flow from proxy to foreman and puppet VLAN D Site2 mgmt VLAN2 fw VLAN C Site2 smartproxy, puppet, DHCP, Proxy VLAN A Site1 mgmt VLAN1 fw Foreman smartproxy, puppet, CA, DNS, DHCP VLAN B Site1 12
  13. 13. Foreman Components optimizing comms flows ▸ install apache mod_rewrite on remote smartproxy ▸ rewrite/proxy unattended resources ▸ rewrite/proxy puppet CA resources ... rewrite_rule => [ ‘^/unattended/(.*)$$1 [P]’, ] rewrite_rule => [ ‘^/([^/]+/certificate.*)$$1’, ] ... 13
  14. 14. Foreman Components foreman customization ▸ ▸ ▸ we must identify proxied servers foreman url must be customized templates must be adapted ▸ enable tokens (default in 1.4) ▸ create parameter in hostgroups for each segment (unattended_url in 1.4 does not seam to help here…) ▸ adapt templates 14
  15. 15. Resistance is futile Integrating foreman into corporate infrastructure Integrating foreman into corporate infrastructure is: ▸ not a technical issue ▸ mostly even not a security issue ▸ It’s about convincing people to do things differently ▸ It’s about responsibilities ▸ It’s about fear ▸ It’s about laziness 15
  16. 16. Resistance is futile The one with the wrong <enter vendor name> DHCP product The corporate DHCP does not provide a proper API and changes are distributed across the DHCP infrastructure in a slow, asynchronous manner… Deploy your own DHCP server: - You must be able to create and delete leases instantly - You only need it to bootstrap your servers. Later a static NIC configuration can be applied using your favorite cfgmgmt. No, it’s not that static – we can use ENC variables to update IP addresses. - Repeat: I won’t create my leases manually and I won’t wait for changes to be replicated until I power-on my server 16
  17. 17. Resistance is futile The one with the DHCP caused Zombiecalypse You are about to deploy your own DHCP but people complain that your DHCP will nuke the corporate datacenter and will cause an epic loss of revenue… Ask for dedicated VLANs and for ip-helper configs on routers and firewalls - Your DHCP is limited to your VLANs only - You only need one DHCP server - It’s not an evil plan to make the guy running the corporate DHCP redundant 17
  18. 18. Resistance is futile The one with static DNS You kindly ask for TSIG keys to create your A and PTR records automatically but the DNS is managed manually and there is no TSIG key although dynamic updates are enabled… Make a deal: you won’t talk about the missing TSIG keys and get your key or keytab and permissions right after the guy fixed the security issue. Ask for dedicated zones and for credentials which allow you to manage your zones using nsupdate - Alternatively ask for delegation of your zones and setup a dedicated DNS server (protected by keys) 18
  19. 19. FAIL! Foreman HA ▸ use pacemaker/corosync to provide basic resilience for postgreSQL and puppetDB, puppetCA ▸ If you scale out, you might setup a nice pg-pool2 cluster ▸ puppetmasters could be ran active active as well as the foreman rails application (keep memcached in mind) ▸ on pure smartproxies no pacemaker cluster is required. A simple failover using VRRP (via keepalived) should be sufficient – if you replicate the tftp and dhcp data (for puppet you’ll have to do the exercise anyhow.) HA IP(s) HA IP(s) puppetDB puppetDB Apache mod_passenger Apache mod_passenger postgreSQL postgreSQL DRBD DRBD pacemaker pacemaker corosync node1 node2 19
  20. 20. Best practices ▸ use separate IP addresses for each service so you can split up stuff later on ▸ If you can’t use DHCP the bootdisk plugin might be an option (currently .deb package 1.2.3 is broken for 1.4 but version plugin source is ready for 1.4) ▸ LDAP or kerberos5 (1.4) for user authentication ▸ use Locations for filtering ▸ classes are not tight to locations so you better use the same set of classes and import only once from you master puppet master  ▸ use the column plugin for better overview ▸ ENC and smart variables are nice – ever configured dhcp pool as yaml hash? Use smart variables as switches, and for simple datatypes only - do the complex stuff in service modules - ideally by using data in modules 20
  21. 21. Wishlist ...secret guide to world domination ▸ integrate the reverse proxy feature into smart-proxy (there is a ticket for that somewhere) ▸ implement iso image provisioning based vm installation in conjunction with the bootdisk plugin ▸ support more configuration management stacks ▸ better support for discovery, ipmi and hardware provisioning (like HW-Raid) 21
  22. 22. Thank You! Contact Nils Domrose Senior Systems Engineer inovex GmbH Office cologne Schanzenstr. 6-20 51063 Köln 22