Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Solving real world data
problems with Jerakia
Craig Dunn, Config Management Camp, Ghent 2016
• Best practice
• Code base design
• Workflow mangement
• Scaling Puppet
• Installation and support
• Module writing
• Thro...
• Puppet user since 2008
• IT consultant for 15+ years
• Active community member
• The “Roles and Profiles” guy
• Problem s...
A brief history of Puppet
In the beginning…
• Over complex code
• Unsharable modules
• Making simple changes required alot of skill.
The embedded da...
class ntp {
if $env == ‘dev’ {
$server = ‘dev.ntp.local’
} else {
if $hostname == ‘gateway’ {
$server = ‘pool.ntp.org’
} e...
And then…
Hiera
The dawn of the data separation era
• Separation of data from code
• Module authors could write sharable re-usable code
• Code was less complex and more reada...
Hierarchical Search
Pluggable
• Pluggable interchangable backends
• Data can be sourced from multiple formats
• hiera-eyaml
• hiera-mysql
• hi...
Managing our data is now
a critical part of configuration management
Infrastructure grows and requirements
get more complex
• Different teams and customers require different hierarchies
• A particular application needs to source data from a diffe...
Introducing Jerakia
jerakia.io
Jerakia
• Data lookup tool
• Open source
• Extendable framework
• Solving the most complex edge cases
Jerakia
• Can be used as a Hiera backend
• Can be wired directly into Puppet as a data binding terminus
• Drop in replacem...
Why Jerakia?
One design goal…
Flexibility
• Lookup behaviour written in Ruby DSL
• Almost everything is pluggable
• Inter-changable data sources
• Easy integration
...
$ gem install jerakia
$ puppet module install crayfishx/jerakia
• A request is received containing a key and a namespace
• A policy is chosen to perform the request
• One or more lookups...
An Example Jerakia Policy File
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:fo...
An Example Jerakia Policy File
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:fo...
An Example Jerakia Policy File
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:fo...
An Example Jerakia Policy File
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:fo...
• Lookups are contained within policies
• A policy can contain multiple lookups
• A lookup always contains at least a data...
Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookup
Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookup
Request consists ...
Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookup
Information to be...
Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookup
Lookup plugins ca...
Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookup
A pluggable data ...
Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookup
Data returned fro...
Lookup methods
confine / exclude
Invalidates a lookup unless/if the criteria is met
confine request.namespsace[0], "apache"
confine reques...
Stop
Do not proceed to the next lookup if this lookup is valid
lookup :special do
…
confine request.namespsace[0], "apache...
Datasources
• Easily pluggable and extendable
• File and HTTP datasources shipped out-of-the-box
Datasources
datasource :name, { :option => “value”… }
Datasource definition
lookup :main do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => ...
lookup :main do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"host/#{scope[:cert...
lookup :main do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"host/#{scope[:cert...
/var/lib/jerakia/env/dev/apache.yaml
/var/lib/jerakia/env/dev/apache.d/www_corp_com.yaml
/var/lib/jerakia/env/dev/apache.d...
Data Layout
:searchpath => [
"host/#{scope[:certname]}",
"env/#{scope[:environment]}",
]
# cat /var/lib/jerakia/env/dev/ap...
Plugins
• Access to request and scope
• Can read or modify on-the-fly
• Re-usable
• Cleaner code in policy files
class Jerakia::Lookup::Plugin
module Mything
def do_something
…
end
end
end
Writing plugins
• Written as Ruby extensions
•...
lookup :main, :use => :mything do
plugin.mything.do_something
…
end
Using plugins
• Plugins are loaded into the lookup
• R...
lookup :main, :use => :hiera do
plugin.hiera.rewrite_lookup
datasource :file, {
:docroot => "/var/lib/jerakia",
:format =>...
Output filters
• Pluggable
• Specified in the lookup
• Parses data returned from the datasource
Output filters
• Two are currently shipped
• Encryption (provided by eyaml*)
• Strsub
*https://github.com/TomPoulton/hiera-...
Output filters
lookup :main do
…
output_handler :encryption
end
Output filters
lookup :main do
…
output_handler :encryption
end
Example User Story
• Team in Ireland manage PHP/Apache
• Autonomous team that don’t manage infra
• Their optimal hierarchy...
policy :default do
lookup :main, do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [...
policy :default do
lookup :ireland do
datasource :file, {
:format => :yaml,
:docroot => "/var/external/data/ie",
:searchpa...
policy :default do
lookup :ireland do
datasource :file, {
:format => :yaml,
:docroot => "/var/external/data/ie",
:searchpa...
policy :default do
lookup :ireland do
datasource :file, {
:format => :yaml,
:docroot => "/var/external/data/ie",
:searchpa...
Command line
$ jerakia lookup port —namespace apache
$ jerakia help lookup
Usage:
jerakia lookup [KEY]
Options:
c, [--conf...
Integration with Puppet
—-
:backends:
- jerakia
[master]
. . .
data_binding_terminus = jerakia
Roadmap &
Contributing
Upcoming in 0.5
• Data Schemas
• Better REST client/server
• Deep merge behaviour
• Lookup plugin “load method”
Contributions wanted
• Code maturity
• Caching
• Features
• Bugfixes
• Documentation
• #jerakia (freenode) Sponsored by
Jerakia 1.0
Thank you
Questions?
jerakia.io
@crayfishx
Solving real world data problems with Jerakia
Solving real world data problems with Jerakia
Upcoming SlideShare
Loading in …5
×

Solving real world data problems with Jerakia

1,156 views

Published on

This is the talk I gave at Config Management Camp 2016 in Ghent introducing Jerakia as a lookup tool that can be used in place of, or along side of hiera to solve some of the edge cases around data separation

Published in: Technology
  • Be the first to comment

Solving real world data problems with Jerakia

  1. 1. Solving real world data problems with Jerakia Craig Dunn, Config Management Camp, Ghent 2016
  2. 2. • Best practice • Code base design • Workflow mangement • Scaling Puppet • Installation and support • Module writing • Throughout Europe www.enviatics.com
  3. 3. • Puppet user since 2008 • IT consultant for 15+ years • Active community member • The “Roles and Profiles” guy • Problem solver • Lives in Málaga, Spain. • …. and hotels • Daddy! www.craigdunn.org Craig Dunn @crayfishx
  4. 4. A brief history of Puppet
  5. 5. In the beginning… • Over complex code • Unsharable modules • Making simple changes required alot of skill. The embedded data era
  6. 6. class ntp { if $env == ‘dev’ { $server = ‘dev.ntp.local’ } else { if $hostname == ‘gateway’ { $server = ‘pool.ntp.org’ } else { $server = ‘prod.ntp.local’ } } … }
  7. 7. And then…
  8. 8. Hiera The dawn of the data separation era
  9. 9. • Separation of data from code • Module authors could write sharable re-usable code • Code was less complex and more readable • The Forge became useful • Managing data became a lot easier
  10. 10. Hierarchical Search
  11. 11. Pluggable • Pluggable interchangable backends • Data can be sourced from multiple formats • hiera-eyaml • hiera-mysql • hiera-http • hiera-redis • hiera-consul
  12. 12. Managing our data is now a critical part of configuration management
  13. 13. Infrastructure grows and requirements get more complex
  14. 14. • Different teams and customers require different hierarchies • A particular application needs to source data from a different place • Control access to sub-sets of data for teams within an organisation • Dynamically generate the lookup hierarchy at runtime • Group together application specific data into separate files • Manage encrypted data from any data source • Global hiera.yaml file creates restrictions
  15. 15. Introducing Jerakia jerakia.io
  16. 16. Jerakia • Data lookup tool • Open source • Extendable framework • Solving the most complex edge cases
  17. 17. Jerakia • Can be used as a Hiera backend • Can be wired directly into Puppet as a data binding terminus • Drop in replacement for Hiera, or not.
  18. 18. Why Jerakia?
  19. 19. One design goal…
  20. 20. Flexibility
  21. 21. • Lookup behaviour written in Ruby DSL • Almost everything is pluggable • Inter-changable data sources • Easy integration • Hiera compatible*
  22. 22. $ gem install jerakia
  23. 23. $ puppet module install crayfishx/jerakia
  24. 24. • A request is received containing a key and a namespace • A policy is chosen to perform the request • One or more lookups are called to act on the request • A response is sent back to the requestor • Container for lookups • Written in Ruby DSL • Different policies for different apps Policy File
  25. 25. An Example Jerakia Policy File policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end
  26. 26. An Example Jerakia Policy File policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end
  27. 27. An Example Jerakia Policy File policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end
  28. 28. An Example Jerakia Policy File policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } end end
  29. 29. • Lookups are contained within policies • A policy can contain multiple lookups • A lookup always contains at least a data source Lookups
  30. 30. Scope Handler Request Lookup Plugins Data Source Output Filter Response Data Anatomy of a Jerakia lookup
  31. 31. Scope Handler Request Lookup Plugins Data Source Output Filter Response Data Anatomy of a Jerakia lookup Request consists of a lookup key, a namespace and some metadata
  32. 32. Scope Handler Request Lookup Plugins Data Source Output Filter Response Data Anatomy of a Jerakia lookup Information to be used in determining how data is looked up
  33. 33. Scope Handler Request Lookup Plugins Data Source Output Filter Response Data Anatomy of a Jerakia lookup Lookup plugins can read and modify the scope and request objects
  34. 34. Scope Handler Request Lookup Plugins Data Source Output Filter Response Data Anatomy of a Jerakia lookup A pluggable data source is used to lookup data
  35. 35. Scope Handler Request Lookup Plugins Data Source Output Filter Response Data Anatomy of a Jerakia lookup Data returned from the datasource is passed to a pluggable output filter
  36. 36. Lookup methods
  37. 37. confine / exclude Invalidates a lookup unless/if the criteria is met confine request.namespsace[0], "apache" confine request.namespsace[0], [ /website_.*/, "apache", "php" ]
  38. 38. Stop Do not proceed to the next lookup if this lookup is valid lookup :special do … confine request.namespsace[0], "apache" stop end lookup :main do …
  39. 39. Datasources • Easily pluggable and extendable • File and HTTP datasources shipped out-of-the-box
  40. 40. Datasources datasource :name, { :option => “value”… }
  41. 41. Datasource definition lookup :main do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "host/#{scope[:certname]}", "env/#{scope[:environment]}", "common", ] } end /var/lib/jerakia/env/dev/apache.yaml
  42. 42. lookup :main do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "host/#{scope[:certname]}", "env/#{scope[:environment]}", "common", ] } end /var/lib/jerakia/env/dev/apache.yaml Datasource definition
  43. 43. lookup :main do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "host/#{scope[:certname]}", "env/#{scope[:environment]}", "common", ] } end /var/lib/jerakia/env/dev/apache.yaml Datasource definition
  44. 44. /var/lib/jerakia/env/dev/apache.yaml /var/lib/jerakia/env/dev/apache.d/www_corp_com.yaml /var/lib/jerakia/env/dev/apache.d/www_acme_net.yaml /var/lib/jerakia/env/dev/apache.d/www_fake_org.yaml Fragments • Introduced in 0.4 • If a .d directory is found, files within are concatenated • One document is returned
  45. 45. Data Layout :searchpath => [ "host/#{scope[:certname]}", "env/#{scope[:environment]}", ] # cat /var/lib/jerakia/env/dev/apache.yaml —- port: 80 # cat /var/lib/jerakia/env/dev.yaml —- apache::port: 80 Hiera Jerakia
  46. 46. Plugins • Access to request and scope • Can read or modify on-the-fly • Re-usable • Cleaner code in policy files
  47. 47. class Jerakia::Lookup::Plugin module Mything def do_something … end end end Writing plugins • Written as Ruby extensions • Can be placed in the plugin dir • Or shipped as rubygems
  48. 48. lookup :main, :use => :mything do plugin.mything.do_something … end Using plugins • Plugins are loaded into the lookup • Referenced as plugin.name.method lookup :main, :use => [ :mything, :foo ] do … end
  49. 49. lookup :main, :use => :hiera do plugin.hiera.rewrite_lookup datasource :file, { :docroot => "/var/lib/jerakia", :format => :yaml, :searchpath => [ "env/#{scope[:environment]}", "common", ] end The hiera plugin • Provides compatibility to hiera filesystem layouts • Shipped with Jerakia # cat /var/lib/jerakia/env/dev.yaml —- apache::port: 80
  50. 50. Output filters • Pluggable • Specified in the lookup • Parses data returned from the datasource
  51. 51. Output filters • Two are currently shipped • Encryption (provided by eyaml*) • Strsub *https://github.com/TomPoulton/hiera-eyaml
  52. 52. Output filters lookup :main do … output_handler :encryption end
  53. 53. Output filters lookup :main do … output_handler :encryption end
  54. 54. Example User Story • Team in Ireland manage PHP/Apache • Autonomous team that don’t manage infra • Their optimal hierarchy is different from “ours” • “We” need to service them from Puppet • They must not modify infra services • “We” also manage PHP/Apache for other clients
  55. 55. policy :default do lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end Our main lookup is responsible for the entire infrastructure
  56. 56. policy :default do lookup :ireland do datasource :file, { :format => :yaml, :docroot => "/var/external/data/ie", :searchpath => [ "project/#{scope[:project]}", "common", ] } end lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end Lookup for the Ireland team added above the main lookup with separate docroot and searchpath
  57. 57. policy :default do lookup :ireland do datasource :file, { :format => :yaml, :docroot => "/var/external/data/ie", :searchpath => [ "project/#{scope[:project]}", "common", ] } confine scope[:location], "ie" confine request.namespace[0], [ "apache", "php", ] end lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end Only use this lookup if the requestor location is IE and the namespace is apache or php
  58. 58. policy :default do lookup :ireland do datasource :file, { :format => :yaml, :docroot => "/var/external/data/ie", :searchpath => [ "project/#{scope[:project]}", "common", ] } confine scope[:location], "ie" confine request.namespace[0], [ "apache", "php", ] stop end lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end If this lookup is valid then do not proceed to the main lookup, even if data is not found.
  59. 59. Command line $ jerakia lookup port —namespace apache $ jerakia help lookup Usage: jerakia lookup [KEY] Options: c, [--config=CONFIG] # Configuration file p, [--policy=POLICY] # Lookup policy # Default: default n, [--namespace=NAMESPACE] # Lookup namespace t, [--type=TYPE] # Lookup type # Default: first s, [--scope=SCOPE] # Scope handler # Default: metadata [--scope-options=key:value] # Key/value pairs to be passed to the scope handler m, [--merge-type=MERGE_TYPE] # Merge type # Default: array l, [--log-level=LOG_LEVEL] # Log level v, [--verbose], [--no-verbose] # Print verbose information D, [--debug], [--no-debug] # Debug information to console, implies --log-level debug d, [--metadata=key:value] # Key/value pairs to be used as metadata for the lookup Lookup [KEY] with Jerakia
  60. 60. Integration with Puppet —- :backends: - jerakia [master] . . . data_binding_terminus = jerakia
  61. 61. Roadmap & Contributing
  62. 62. Upcoming in 0.5 • Data Schemas • Better REST client/server • Deep merge behaviour • Lookup plugin “load method”
  63. 63. Contributions wanted • Code maturity • Caching • Features • Bugfixes • Documentation • #jerakia (freenode) Sponsored by
  64. 64. Jerakia 1.0
  65. 65. Thank you Questions? jerakia.io @crayfishx

×