Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Delegated Configuration with Multiple Hiera Databases - PuppetConf 2014

2,693 views

Published on

Delegated Configuration with Multiple Hiera Databases - Robert Terhaar, Atlantic Dynamic

Published in: Technology
  • Be the first to comment

Delegated Configuration with Multiple Hiera Databases - PuppetConf 2014

  1. 1. Delegated Config with Multiple Hiera Databases Robert Terhaar rob@atlanticdynamic.com Atlantic Dynamic NYC PuppetConf 2014 - September 24
  2. 2. Company & Personal Bio • Build custom cloud, automation, deployment, and management systems for: • Finance // Bio-Tech // Start-ups // Advertising • Sysadmin since 1998 • Puppet user since 2007 • Based in NYC 🗽
  3. 3. Hiera Databases with Multiple Delegated Config
  4. 4. What is Hiera?
  5. 5. Hiera is a framework for hierarchically organizing data, and abstracting it from your puppet manifests.
  6. 6. With Hiera, you can externalize your data, and easily understand how configuration data is assigned to your servers.
  7. 7. What data belongs in Hiera?
  8. 8. Keep your secrets in Hiera
  9. 9. https://github.com/TomPoulton/hiera-eyaml Secrets in Hiera
  10. 10. Store your business- logic in Hiera
  11. 11. Store all OS- specific config in params.pp
  12. 12. Hiera is not params.pp
  13. 13. Business Logic (Hiera) vs. OS-Specific Config (params.pp) • Servers in production: use database IP 10.0.0.1 • In us-east1: use NTP server 167.88.119.29 • On RHEL7: SELINUX=enforcing • Package names for Apache on Debian/RHEL: apache2/httpd • 1 CPU = default to 1 worker
 but on 4 CPUs = default to 5 workers
  14. 14. A Basic Hiera Example
  15. 15. Hiera Basics $servers = hiera('ntp::servers') As a Puppet Function
  16. 16. Hiera Basics Function in a parameterized class class ntp( $servers = hiera('ntp::servers'), ) { < ntp config goes here… > }
  17. 17. Hiera Basics Implicit Lookup w/ Data Bindings class ntp( $servers, ) { < ntp config goes here… > }
  18. 18. Hiera Basics $servers = hiera(‘ntp::servers’, ) As a Puppet Function, w/ Default ‘pool.ntp.org’
  19. 19. $ cat /etc/puppet/hiera.yaml --- :backends: - yaml ! :logger: console ! :hierarchy: - "fqdn/%{fqdn}" - "role/%{role}" - "lifecycle/%{lifecycle}" - "location/%{location}" - common ! :yaml: :datadir: /etc/puppet/hieradb
  20. 20. $ tree /etc/puppet/hieradb ! "## lifecycle $   "## dev.yaml $   "## production.yaml $   &## staging.yaml "## location $   &## us-east1.yaml &## os "## rhel6.yaml &## rhel7.yaml
  21. 21. Hiera supports multiple storage backends
  22. 22. • eyaml • http (REST) • mysql • postgres • redis • mongodb • json • yaml • and more…
  23. 23. $ cat /etc/puppet/hiera.yaml --- :backends: ! ! ! :logger: console ! :hierarchy: - "fqdn/%{fqdn}" - "role/%{role}" - "lifecycle/%{lifecycle}" - "location/%{location}" - common ! :yaml: :datadir: /etc/puppet/hieradb ! :postgres: :datadir: /etc/puppet/hieradb :host: <hostname> :user: <username> :pass: <password> :database: <database> - yaml - postgres
  24. 24. $ tree /etc/puppet/hieradb ! "## lifecycle $   "## dev.yaml $   "## $   "## production.yaml $   &## staging.yaml "## location $   &## us-east1.yaml &## os "## rhel6.yaml &## rhel7.yaml production.sql
  25. 25. But, let’s not focus on the tools… yet
  26. 26. Designing the solution
  27. 27. “Good design is as little design as possible” https://www.vitsoe.com/us/about/good-design
  28. 28. Design and Architecturehttp://commons.wikimedia.org/wiki/File:Fallingwater_%28Kaufmann_Residence_by_Frank_Lloyd_Wright%29_-_26_June_2012.jpg
  29. 29. “Architecture is the stuff that's hard to change later. And there should be as little of that stuff as possible.” 
 - Martin Fowler http://martinfowler.com/ieeeSoftware/whoNeedsArchitect.pdf
  30. 30. Semantics ! Architecture: Concrete Bricks ! Design: LEGO® Blocks
  31. 31. The Setup
  32. 32. Everyone ElseDevOps Team Node resource resource resource resource Node resource resource resource resource environments modules hiera puppetdb manifests templates ! - fqdn/%{::fqdn}! - lifecycle/%{::lifecycle}! - location/%{::location}! - os/%{::osfamily}! - common Puppet Master
  33. 33. Everyone ElseDevOps Team hiera ! - fqdn/%{::fqdn}! - lifecycle/%{::lifecycle}! - location/%{::location}! - os/windows.yml! - common
  34. 34. Everyone Else! ! ntp::servers = [! “server1.corp”,! “server2.corp”,! ] DevOps Team hiera ! - fqdn/%{::fqdn}! - lifecycle/%{::lifecycle}! - location/%{::location}! - os/windows.yml! - common
  35. 35. Delegation
  36. 36. Step 1
  37. 37. Develop the Problem Statement.
  38. 38. We need a way to delegate access to a few Hiera keys.
  39. 39. We need a way to delegate access to a few Hiera keys.
  40. 40. Colleagues who are not in the “DevOps Team” need to manage a few pre-defined parameters. (but only on a subset of servers)
  41. 41. Everyone Else! ! ntp::servers = [! “server1.corp”,! “server2.corp”,! ] Special People Club hiera ! - fqdn/%{::fqdn}! - lifecycle/%{::lifecycle}! - location/%{::location}! - os/windows.yml! - common Colleagues who are not in the “DevOps Team” need to manage a few pre-defined parameters. (but only on a subset of servers)
  42. 42. Step 2
  43. 43. Gathering Requirements
  44. 44. Requirement: The Solution Must Fly
  45. 45. Don’t over- engineer
  46. 46. Don’t over- engineer your solution
  47. 47. Don’t over- engineer your requirements document
  48. 48. Requirement “Types” http://en.wikipedia.org/wiki/Requirements_analysis#Types_of_Requirements
  49. 49. • What are we building (1-2 sentence overview) • What are the basic goals? (write them down!) • How will we know when it’s done? • What assumptions are we are making? • What are some risks? The Requirements Document
  50. 50. • Get feedback from… • your boss • the client • your colleagues • other stakeholders The Requirements Document
  51. 51. Iterate the doc
  52. 52. Own the doc
  53. 53. Be Realistic & Prioritize
  54. 54. Feedback, Surprises
  55. 55. The Results
  56. 56. What are we building? ! We are building a data import system for Hiera which allows secure delegated access to end users. The system filters data, and can import data from various external systems.
  57. 57. • Import filtered data from various sources to a database. ! • That database is secondary Hiera backend datastore. ! • Adding additional import sources should be simple. ! • Easy to understand where keys are imported from. Goals
  58. 58. How will we know when it’s done? The first version will be complete once we: • Build a prototype • Document the solution • Test importing data from a few sources • Create a deployment plan • Deploy to production
  59. 59. Step 3
  60. 60. Brainstorm, and Prototype
  61. 61. The solution without design
  62. 62. Everyone Else! ! UPDATE windows! SET value=‘[“server1.corp”, “server2.corp”]! WHERE key=‘ntp::servers’; DevOps Team hiera We need a way to delegate access to a few Hiera keys. PostgreSQL &! hiera-postgresql- backend ! - fqdn/%{::fqdn}! - lifecycle/%{::lifecycle}! - location/%{::location}! - os/windows.sql! - common windows.sql! ———! ntp::servers: SELECT value FROM windows WHERE key=‘ntp::servers’; Single SQL Hiera Backend
  63. 63. The designed solution
  64. 64. Importer ! App Puppet! Master Node Node resource resource resource resource resource resource resource resource DevOps Team Delegated Hiera DB Primary Hiera DB filter Everyone Else Import Plugin 1 Import Plugin 2 External Data Source 2 White List! (What keys & namespaces are allowed) Authoritative Hiera Data External Data Source 1 Simple data import script (run via cron) The slightly better solution (logical diagram)
  65. 65. Custom Hiera Backend The slightly better solution (with implementation detail) Importer ! App Puppet! Master Node Node resource resource resource resource resource resource resource resource DevOps Team Delegated Hiera DB Primary Hiera DB filter Everyone Else Import Plugin 1 Import Plugin 2 External Data Source 2 External Data Source 1 Python import script, with pluggable import backends PostgreSQL DB CMDB API, and LDAP .yaml files stored in git
  66. 66. Results
  67. 67. Intentional design, and stakeholder feedback will lead to a better solution.
  68. 68. DevOps Team Everyone Else
  69. 69. Useful Resources Good Design: https://www.vitsoe.com/us/about/good-design Learn More about Hiera: http://garylarizza.com/blog/2013/12/08/when-to-hiera/ Postgres Hiera Backend: https://github.com/adrianlzt/hiera-postgres-backend Hiera Encryption (eyaml): https://github.com/TomPoulton/hiera-eyaml Requirements: http://en.wikipedia.org/wiki/Requirements_analysis
  70. 70. Robert Terhaar rob@atlanticdynamic.com Atlantic Dynamic - NYC PuppetConf 2014 - September 24 Questions? Thank You!

×