Your SlideShare is downloading. ×
Puppet at Opera Sofware - PuppetCamp Oslo 2013
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Puppet at Opera Sofware - PuppetCamp Oslo 2013

1,207

Published on

A bit of history, frustration-driven development, and why and how we started looking into Puppet at Opera Software. What we're doing, successes, pain points and what we're going to do with Puppet and …

A bit of history, frustration-driven development, and why and how we started looking into Puppet at Opera Software. What we're doing, successes, pain points and what we're going to do with Puppet and Config Management next.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,207
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
11
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Puppet at Opera Puppet Camp Oslo 2013 cosimo@opera.com
  • 2. devs sysadmin
  • 3. devs sysadmin DevSys?
  • 4. FDDFrustration Driven Development
  • 5. # LVS main config file## Last modified:# 2012-12-10 Commented out all wlb servers, as they havent been in use …# 2012-XX-XX Tons of shifting around servers, upgrading and problems (Everyone)# 2011-04-01 Removed all old b#-servers (N.....)# 2010-03-24 Bye bye bigma. (M..../Cosimo)# 2010-03-03 Restore pre Feb 26th config that seems to ensure stability (Cosimo)# When adding bigboy/bigcat, bad site lockups happen# 2010-03-03 Reducing weight on b12 as it is less powerfull (M....)# 2010-02-26 re-adding bigdog, and lowering bigunc, also vamping up b12 to 100%# 2010-02-26 Bigdog is crashing, removing from lvs (M......)# 2010-02-03 Enabled f8 and b7, first b7, then some hours later f8 … (N......)# 2010-01-19 Bigant ready to rock and roll! (Cosimo)# 2010-01-13 Removed bigpa, fatgirl from database pool (Cosimo)# 2010-01-07 Added b8 to backend pool (Cosimo)# 2010-01-05 Added bigant to the My Opera databases (Cosimo)# 2009-11-22 Added bigdog to the My Opera databases (Cosimo)# 2009-11-18 Added b7 and f8 as back-end servers (M.....)# 2009-11-18 Removed p23-02 backend, moved to auth (Cosimo)# 2009-11-12 Removing b7 and f8 from Mysql Load balancers (Cosimo)# 2009-11-11 Added Lenny backend p23-02 (Cosimo)# 2009-10-11 phased-in InnoDB-powered bigma in production (Cosimo)# 2009-09-23 phased-in InnoDB-powered bigma in production (Cosimo)# 2009-06-27 switched master from bigma to bigsis (w-mlb) o/ (N.....)# 2009-06-23 shifting load away from bigbro. its dying? (Cosimo)# 2009-03-18 pushing bigbro as much as we can, to test it out (Cosimo)global_defs { lvs_id MY_LVS …}
  • 6. innodb_buffer_pool_size = 128M # was 64M # was 16M # was 32M
  • 7. The Pilot – Goals● New deployment procedure● Sane configuration files● Configuration management
  • 8. CM Tools Evaluation (2009) CFEngine 2 BCfg2 Puppet 0.25.4 LCFG
  • 9. CM Tools Evaluation CFEngine 2 BCfg2 Puppet 0.25.4 → 2.6.2 → 2.7.14 LCFG
  • 10. The very beginning...commit 9c54321f51bf969940b63b48d055743ac504035eAuthor: Cosimo Streppone <cosimo@opera.com>Date: Thu Jan 14 13:21:40 2010 +0000 Generic puppet recipes. To be continued.
  • 11. Our approach
  • 12. A “conservative” approach, surely • Keep it simple. No concat/append/modify • As few dependencies as possible • Stability and reliability is critical • No pulls from github or external URLs • We dont use puppet for deployment • Even realize() gets me into panic mode
  • 13. Three Years In• Modules repository, with 60+ mods• Some custom facter plugins• Shared projects conventions & structure• Shared deployment procedures and libs• Good server baseline configuration• Our team, ~200 nodes• Opera Mini Ops team, thousands of nodes
  • 14. Datacenters
  • 15. Its Modules all the way down... Tomcat base_packages Apache PowerDNS Bash Statsd MuninPostfix Cassandra security_upgrades Ssh RRDCached Django Solr 4.0 RabbitMQ Varnish
  • 16. Projects structureMaster config file /config/production.jsonRole-specific files /config/role/<role>/Puppet manifests /config/puppet/Deployment scripts /deploy/
  • 17. Master configuration file{ "master_rev" : "20130129", "application" : "geodns", "environment" : "production", "domain" : "localdomain", "contact" : "cosimo@opera.com", "puppet_vars" : { # Available in manifests "some-password" : "hola/amigos" }, "systems" : { # List of all hostnames and their roles "node01" : { "puppet_class" : [ "geodns::backend" ] }, "node02" : { "puppet_class" : [ "geodns::frontend" ], "puppet_vars" : { … }, }, … }
  • 18. Puppet master layout/etc/puppet → puppet.conf (master configuration file) fileserver.conf files → {auth, geodns, opcdn} (local project files) modules → (shared generic modules) {ntp, apache, varnish, nginx, ...} manifests → (generic and project specific manifests) classes/ {basenode, backend, frontend}.pp classes/ <project> / <anything goes, project-specific>
  • 19. Puppet master - site.pp /etc/puppet/manifests/site.pp $server = "puppetmaster.opera.com" import "os/*.pp" import "classes/*.pp" # generic classes import "classes/*/*.pp" # project classes node default { include basenode } filebucket { "main": server => $server } File { ignore => [.svn, .git, CVS ], backup => "main", }
  • 20. Puppet master – no nodes.pp/etc/puppet/puppet.conf external_nodes = /etc/puppet/bin/puppet-node-classifier node_terminus = exec/etc/puppet/manifests/nodes/geodns-production.json { "application" : "geodns", "environment" : "production", "domain" : "localdomain", "systems" : { "node01" : { "puppet_class" : [ "geodns::backend" ], }, … } }
  • 21. Facter $ facter --puppet architecture => amd64 datacenter => nerv domain => opera.com facterversion => 1.5.7 fqdn => node01.int.opera.com hardwareisa => unknown hardwaremodel => x86_64 hostname => node01 id => root interfaces => eth0,eth1 ipaddress => 1.2.3.4 ipaddress_eth0 => 1.2.3.4 …
  • 22. Facter – custom plugins facter/datacenter.rb Facter.add("datacenter") do setcode do datacenter = "unknown" # Get current ip address from Facters own db ipaddr = Facter.value(:ipaddress) if ipaddr.match("^1.2.3.") datacenter = "dc1" elsif ipaddr.match(...) … end end end
  • 23. Facter – custom plugins case $datacenter { "dc1" : { include opera::datacenters::dc1 } "dc2" : { include opera::datacenters::dc2 } "dc3" : { include opera::datacenters::dc3 } … default: { include opera::datacenters::base } }
  • 24. Basenode class class basenode { include opera # Opera-specific data-center based settings case $datacenter { "dc1" : { include opera::datacenters::dc1 } … default: { include opera::datacenters::base } } include apt-opera include base_packages include locales include logcheck include munin include nagios include cron include perl include python include puppet include ntp include timezone … }
  • 25. Bootstrap script autosign + some preinstalled packages + internal apt repository + a bit of shell scripting
  • 26. Real world examples – 1 Project class geodns::backend { include opera::admins::devops include security-upgrades include powerdns include geoip::city include memcache package { [ libjson-xs-perl, … ]: ensure => present } bash::prompt { /root/.bashrc: description => geodns, color => red, } munin::plugin::custom { geodns_: } munin::plugin { [ geodns_country, geodns_errors, … ]: plugin_name => geodns_, } }
  • 27. Real world examples – 2 Varnish varnish::config { "project-varnish-config": vcl_conf => "tvstore.vcl", storage_type => "malloc", storage_size => "512M", listen_port => 8100, sess_workspace => 131072, ttl => 60, thread_pools => 2, thread_min => 400, thread_max => 3000, # Needed for GeoIP support in varnish: # http://stackoverflow.com/questions/5906603/ cc_command => "exec cc -fpic -shared -Wl,-x -L/usr/include/GeoIP.h -lGeoIP -o %o %s" }
  • 28. Real world examples – 3 Munin include munin::server file { /etc/munin/munin-conf.d/project-settings.conf: … }
  • 29. Real world examples – 4 Solr include solr4 solr4::core { core1: config => .../core1/solrconfig.xml, properties => .../core1/solrcore.properties, schema => .../core1/schema.xml, } solr4::config { solr-search-config: cores => [core1, … ], }
  • 30. Pain points AKA wish-list
  • 31. Speed! ~60 s runtime → ~600 resources TOO SLOW!
  • 32. Resources that dont go awaynotice: /Stage[main]/Django/Package[Django]/ensure: ensure changed 1.4.3 to 1.4.2notice: /Stage[main]/Package[cython]/ensure: creatednotice: /Stage[main]/Java::Sun_java6/Exec[debconf-set-selections-sun-java6-bin] /returns: executed successfullynotice: /Stage[main]/Java::Sun_java6/Exec[debconf-set-selections-sun-java6-jre] /returns: executed successfully
  • 33. Shared resources cron::logcleanup { … } • Used by both Apache and Nginx modules • Getting conflicts if you pull both
  • 34. Shared environmentMany projects run under the same master.A syntax error anywhere blocks everyone.
  • 35. Testing Would be awesome to be able to test our modules and manifests. Locally. Without a puppetmaster.
  • 36. Future directions
  • 37. Things wed like to look into... • PuppetDB • Better systems inventory • Better Nagios integration • Testing manifests and modules
  • 38. Q&A
  • 39. @cstrep cosimo@opera.com https://github.com/cosimo/http://w w w.streppone.it/cosimo/blog/

×