When traditional configuration management is to slow for your needs

787 views

Published on

Puppet, Chef, too slow ? Consul + Consul Template to the Rescue, my LinuxCon Europe 2015 Dublin Talk.

Published in: Technology

When traditional configuration management is to slow for your needs

  1. 1. When your configurationWhen your configuration management is too slow tomanagement is too slow to reconfigure your infrareconfigure your infra Kris Buytaert @krisbuytaert
  2. 2. KrisKris BuytaertBuytaert ● I used to be a Dev,I used to be a Dev, ● Then Became an OpThen Became an Op ● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source Consultant @inuits.euConsultant @inuits.eu ● Everything is an effing DNS ProblemEverything is an effing DNS Problem ● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore ● Organising too many confs , #devopsdays,Organising too many confs , #devopsdays, #loadays, #cfgmgmtcamp ...#loadays, #cfgmgmtcamp ... ● Evangelizing devopsEvangelizing devops
  3. 3. #devops=~C(L)AMS#devops=~C(L)AMS ● CultureCulture ● (Lean)(Lean) ● AutomationAutomation ● Monitoring and MeasurementMonitoring and Measurement ● SharingSharing Damon Edwards and John WillisDamon Edwards and John Willis Gene KimGene Kim
  4. 4. Infrastructure as CodeInfrastructure as Code
  5. 5. Infrastructure as CodeInfrastructure as Code ● Treat configuration automation as codeTreat configuration automation as code ● Development best practicesDevelopment best practices • Model your infrastructureModel your infrastructure • Version your cookbooks / manifestsVersion your cookbooks / manifests • Test your cookbooks/ manifestsTest your cookbooks/ manifests • Dev/ test /uat / prod for your infraDev/ test /uat / prod for your infra ● Model your infrastructureModel your infrastructure ● A working service = automated ( Application Code + Infrastructure Code + Security +A working service = automated ( Application Code + Infrastructure Code + Security + Monitoring )Monitoring ) ● IAC -ne scriptingIAC -ne scripting
  6. 6. 3 types of files3 types of files ● Part of a packagePart of a package • rpm -qfrpm -qf ● Managed by configuration management toolsManaged by configuration management tools • ““### This file is managed by ..”### This file is managed by ..” ● ““User” generated dataUser” generated data • Actual user dataActual user data • Logs ..Logs ..
  7. 7. Idempodent InfrastructureIdempodent Infrastructure ● Configuration management ensures stateConfiguration management ensures state ● Periodically,Periodically, ● Repeatedly,Repeatedly, ● Manual changes are revertedManual changes are reverted
  8. 8. Puppet for CfgMgmtPuppet for CfgMgmt ● Puppet is opensource,Puppet is opensource, ● Puppet makes your infra idempodent,Puppet makes your infra idempodent, ● Puppet scales,Puppet scales, ● Puppet runs every X minutes, orPuppet runs every X minutes, or ● Puppet is triggered on demandPuppet is triggered on demand ● A puppetrun takes between 3s and 10minA puppetrun takes between 3s and 10min
  9. 9. Typical ArchTypical Arch ● Puppet masterPuppet master ● Puppet agentsPuppet agents ● PuppetdbPuppetdb
  10. 10. Stored Configs andStored Configs and Exported resourcesExported resources
  11. 11. Sometimes,Sometimes, ● That's not fast enough.That's not fast enough. • Think adding new nodes to a loadbalancedThink adding new nodes to a loadbalanced setupsetup • Removing nodes from a load balanced setupRemoving nodes from a load balanced setup • Exported Resource + Collected resourceExported Resource + Collected resource roundtrip is to slowroundtrip is to slow
  12. 12. Sometimes,Sometimes, ● That's not enough.That's not enough. • Containrz spinning up faster than nodesContainrz spinning up faster than nodes • AWS, when you don't want AWS LockInAWS, when you don't want AWS LockIn
  13. 13. CattleCattle
  14. 14. ““In a not so distant future”In a not so distant future” ● People will build small minimal functionPeople will build small minimal function applications that constantly talk to eachotherapplications that constantly talk to eachother ● They will want to know where the otherThey will want to know where the other applications liveapplications live ● They will want to register their applications andThey will want to register their applications and broadcast that to their peersbroadcast that to their peers
  15. 15. So you want service registration of your healthySo you want service registration of your healthy (containerized) applications ?(containerized) applications ?
  16. 16. Consul.ioConsul.io
  17. 17. Consul.ioConsul.io ● ““Service discovery”Service discovery” ● Failure detectionFailure detection ● Using Gossip build on top of SerfUsing Gossip build on top of Serf ● Random node 2 node communicationRandom node 2 node communication ● A HashiCorp projectA HashiCorp project
  18. 18. Setting upSetting up
  19. 19. List membersList members
  20. 20. Defining a ServiceDefining a Service
  21. 21. Query ServicesQuery Services
  22. 22. YamaYama ● Yet another Mcollective AlternativeYet another Mcollective Alternative
  23. 23. git@github.com:solarkennedy/puppet-consul.gitgit@github.com:solarkennedy/puppet-consul.git Puppet-consulPuppet-consul
  24. 24. Consul_templateConsul_template ● Dynamically refils your templates based onDynamically refils your templates based on changes in consulchanges in consul ● Performs actions you definePerforms actions you define ● Go Template formatGo Template format
  25. 25. Puppet-consul_templatePuppet-consul_template ● git@github.com:KrisBuytaert/puppet-git@github.com:KrisBuytaert/puppet- consul_template.gitconsul_template.git ● Might be merged into the puppet-consulMight be merged into the puppet-consul modulemodule
  26. 26. PuppetizePuppetize
  27. 27. Live demoLive demo
  28. 28. Live DemoLive Demo ● 3 nodes3 nodes ● 1x Nginx template rewritten by consul1x Nginx template rewritten by consul ● 2x apache2x apache ● Monitoring_plugins based checkMonitoring_plugins based check ● Vagrantproject :Vagrantproject :
  29. 29. Other use casesOther use cases ● Monitoring + AlertingMonitoring + Alerting ● Service registrationService registration
  30. 30. Other approachesOther approaches ● HaProxy ?HaProxy ? ● LinuxHA / Corosync / .... ?LinuxHA / Corosync / .... ? ● ZookeeperZookeeper ● EtcdEtcd ●
  31. 31. Remember idempotenceRemember idempotence ● Consul seemed instableConsul seemed instable ● Puppet using old ruby json libPuppet using old ruby json lib ● Writing of json config was not idempodentWriting of json config was not idempodent ● ● Reconfiguring consul every single timeReconfiguring consul every single time ● Fixed now.Fixed now.
  32. 32. ContactContact Kris BuytaertKris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ http://www.inuits.be/http://www.inuits.be/ InuitsInuits Duboistraat 50Duboistraat 50 2060 Antwerpen2060 Antwerpen BelgiumBelgium 891.514.231891.514.231 +32 475 961221+32 475 961221

×