1.
Tyler Langlois, October 12th 2017
Software Engineer, Elastic
@leothrix, github: tylerjl
Custom Types and
Providers: Modeling Modern
REST Interfaces and Beyond

2.
2
Obligatory “About Me” Slide
• Been with company since 2014
• Co-maintainer of Elastic Puppet modules (primarily Elasticsearch and
Kibana)
• Puppet-ing in one way or another over my whole professional career
• Brought too many Elastic stickers that need to be given away (please
partake)
• Talk to me about Elasticsearch/Logstash/Kibana/Beats!
Infrastructure/Operations/Software Engineer @ Elastic

3.
3
Who is This Presentation For?
Developers who work with Puppet modules
Puppet users who want to dip into native type/provider development
“What in the %@#$ is the Elasticsearch module doing”
Operators who want to automate against APIs
Hopefully empowers you to implement custom resources on your own

4.
4
WHAT DOES THIS TALK’S
TITLE EVEN MEAN?

5.
5

6.
6
Managing Resources
with Raw APIs
Example: CloudFormation
• Pro:
• Infrastructure resources are data
• Extensible
• Con:
• Managing changes
• Grokking huge chunks of JSON

7.
7
Modeling Resources with
Raw APIs
Example: Terraform
• Pro:
• Readable
• Manageable
• Lifecycle + changes
• Interoperability between other
systems

8.
8
Modeling Resources in Puppet
A DSL to Model
Disparate Resources
A Graph to Manage
Relationships
A Concept of
Changes to
Manage Lifecycles
ls, stat,
chmod,
chown
sysv,
systemd,
upstart
deb, rpm,
pkg
}

9.
9
Modeling Resources in Puppet
Abstraction is Powerful
file { “/tmp/foo”:
source => “puppet:///foo”,
} ->
package { “foo”:
source => “/tmp/foo”
} ~>
service { “foo”:
ensure => “running”,
}

10.
10
Modeling Resources in Puppet
?
Elasticsearch
Logstash
Other
REST
APIs
}

11.
11
Modeling Resources in Puppet
Extending the idea to APIs
elasticsearch::template { “logstash”:
content => {
“template” => “*”,
“settings” => {
“number_of_replicas” => 0
}
}
} ->
service { “es-app”:
ensure => “running”
}

12.
12
Modeling Resources in Puppet
• State Changes
• Instead of comparing changes with GET responses and template files,
compare during a no-op
• A change in state can form dependencies and refresh events into other
resources
• Trickling changes up via reports lends better visibility
Benefits

13.
13
Modeling Resources in Puppet
• State Changes
• More finely-grained control
• Most resources can be represented as Puppet hashes, so Hiera can
be fully leveraged
• Communicating via full Ruby HTTP libraries means CA files, auth, and
more are easier to control
• TESTS!
Benefits

14.
14
Modeling Resources in Puppet
• State Changes
• More finely-grained control
• Some existing API-based resources:
• Kubernetes module (swagger-generated)
• Google Cloud
• Following examples will be low-level (i.e. with just native Ruby HTTP
libraries)
• …hopefully, will help you write your own for $system

15.
15
Let’s (briefly) talk
about Puppet Types
and Providers

16.
16
Types, Providers, and their Resources
Underlying
Resource
Puppet
Provider
Puppet
Type
• Has some way to change a
property
• Its state is introspectable and
discoverable
• Uniquely identified
• How Ruby interacts with
actual commands/system
properties
• Knows how to discover the
properties of resources
• Normalized provider API to
Puppet DSL
• Somewhat typed, catalog
compilation
• Abstraction over providers

17.
17
Types, Providers, and their Resources: service
• systemctl/service/rc
commands
• Startup visibility with enable/
chkconfig/etc.
• Primarily shell-based for state
• One provider for each init
system
• Ruby knows which shell
commands to invoke to start,
stop, enable, etc.
• Unified API to start, enable,
and restart a general
service resource
• Abstraction over provider-
specific implementations
• What we see in a manifest
Underlying
Resource
Puppet
Provider
Puppet
Type

18.
18
Types, Providers, and their Resources: elasticsearch
• REST API endpoints
• Objects modeled in JSON
• Individual endpoints via
_template, _ingest, etc.
• One provider base class, one
provider per resource type
• Using native Ruby HTTP APIs
are high-level enough
• Better alternative than `exec
{ “curl”:`
• Resource properties
expressed in Puppet DSL
hashes
• We don’t make API calls, we
declare desired state
Underlying
Resource
Puppet
Provider
Puppet
Type

19.
19
Then:

20.
20
Now:

21.
Case Study:
Elasticsearch
Pipelines
curl vs. Puppet

22.
22
Ingest Pipelines

23.
23
Ingest Pipelines

24.
24
Ingest Pipelines
• All pipelines are uniquely identified by a name (like defined or native types!)
• Endpoints to manage pipelines:
• GET to retrieve JSON object enumerating all pipelines
• Note: can also retrieved based by name alone
• PUT to create with JSON body
• Note that we’re using unauthenticated APIs right now
Key observations

25.
25
Ingest Pipelines: Puppet Type

26.
26
Ingest Pipelines: Puppet Type (Implementation)

27.
27
Ingest Pipelines: Puppet Type (Implementation)
…what the included abstraction does

28.
28
Ingest Pipelines: Puppet Provider (Implementation)

29.
29
Ingest Pipelines: Puppet Provider (details)
…what the parent class does

30.
30
Ingest Pipelines: Puppet Provider (details)
…what the parent class does

31.
31
Ingest Pipelines: Puppet Tests

32.
32
Ingest Pipelines
• That’s most of it!
• Test-driven development + rspec makes it smooth
• Bulk is abstracted; the beefy parts are in parent classes and reused by
templates, indices, etc.
• Native types and providers ≠ scary
Summary

33.
33
Fitting REST Resources Into Puppet
Considerations
`exists?` versus `prefetch`
Leveraging type-level tools
HTTP
API availability
1
2
3
4

34.
34

35.
35
An Example of Returning a Hash to Prefetch
Automatically Gathering Resources
uri = URI(“http://localhost:9200/_template”)
http = Net::HTTP.new uri.host, uri.port
req = Net::HTTP::Get.new uri.request_uri
response = http.request req
JSON.parse(response.body).map do |object_name, api_object|
{
:name => object_name,
:ensure => :present,
:content => api_object,
:provider => name
}
end

36.
36
Advantages
• puppet resource functionality
• Minimizes chatter with API endpoints
• i.e., checking for existence versus properties, etc.
• Call flush only when necessary
• Additional API freebies (i.e., centralized access in flush(), etc.)
Prefetching resources versus vanilla exists?

37.
37

38.
38
Fitting REST Resources Into Puppet
Considerations
`exists?` versus `prefetch`
Leveraging type-level tools
HTTP
API availability
1
2
3
4

39.
39
Response Content vs. Request Content
Usually never 1:1 mappings
{
"logstash": {
"order": 0,
"version": 60001,
"index_patterns": [
"logstash-*"
],
. . .
elasticsearch::template {
'logstash':
content => {
'template' => '*',
'settings' => {
. . .
vs.

40.
40
Types To the Rescue
• A resource’s desired state is almost never the plain response for a query
against the resource
• Example: kubernetes Deployment versus the state of a Deployment
• munge can help unify the resource versus JSON for comparability
• insync? can be enhanced to understand which fields are being explicitly
controlled by a user
• e.g., I want {“foo”: “bar”} set, I don’t care about what’s in
{“another”: “field”}
• Used pretty heavily in puppet-elasticsearch
Managing response data

41.
41
Example: Setting Default Fields
Elasticsearch template
# Set default values for templates
munge do |value|
{
'order' => 0,
'aliases' => {},
'mappings' => {}
}.merge(value)
end

42.
42
Example: Unifying Formatting
Elasticsearch template
# Normalize then compare the Puppet hash and json
def insync?(is)
Puppet_X::Elastic.deep_implode(is) ==
Puppet_X::Elastic.deep_implode(should)
end
{ “foo”:
{ “bar”: “value” }
}
{
“foo.bar”: “value”
}

43.
43
Fitting REST Resources Into Puppet
Considerations
`exists?` versus `prefetch`
Leveraging type-level tools
HTTP
API availability
1
2
3
4

44.
44
HTTP In Providers

45.
45
HTTP In Providers
• Native HTTP libraries let us more easily control and pass:
• TLS certificate authorities and verification booleans
• HTTP basic auth credentials
• Failure cases (timeouts, 4xx/5xx response codes, etc.)
• In this case with Elasticsearch, error responses can return JSON
messages for more helpful Puppet failures

46.
46
Fitting REST Resources Into Puppet
Considerations
`exists?` versus `prefetch`
Leveraging type-level tools
HTTP
API availability
1
2
3
4

47.
47
API Availability
• What happens if:
• An API-based REST resource requires an API to be up, not just a
daemon?
• A resource should block until one is available?
• An unrelated resource needs that API as well?
Weird edge cases when controlling APIs as opposed to hosts

48.
48
API Availability
• es_instance_conn_validator doesn’t resolve until a connection can
be made

49.
Some observations
after a couple years…

50.
50
Results From the Field
• One parent class makes
creating more easy
• Supported REST-based
resources include:
• indices
• templates
• pipelines
• + more
Extensibility
• rspec + webmock for
great testing
• ES docs + specs first
have made some
implementations first try
successes
• Good mocks make some
acceptance tests
unnecessary (faster CI!)
Reliability
• Much easier to extend to
new OS’s (i.e., Windows)
• Greater control has made
some tasks (like 3.x →
4.x module update)
smooth
+ more

51.
51
Questions?
• github.com/elastic
Thank You!