Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
TIM CINEL • BUILD WHISPERER • ATLASSIAN • @TIMCINEL
Writing Custom Types to
Manage Web-Based Applications
OUR SITUATION
WORKED EXAMPLE
DEMO
This Talk
NEXT STEPS
Our Situation
Atlassian BambooSonatype Nexus
Atlassian’s Build Engineering team uses Puppet
to manage manage many services, including…
•...
Managing config files has drawbacks…
• Nexus’s config file changes
are picked up after restart
• Changes require brief servi...
How do we address
these drawbacks?
Custom Types.
Concepts Refresher
Concept and terminology refresher
• Type
• Name
• Attributes
Can be considered as
instances of types
Native Type
• Ruby im...
What Are Custom Types?
user Type

(Default)
useradd Provider
for user Type
Manifest
bamboo_agent Type

(Custom)
rest Provi...
What Are Custom Types?
user Type

(Default)
useradd Provider
for user Type
Manifest
bamboo_agent Type

(Custom)
rest Provi...
Custom Types can be
created to manage
web-based services.
Custom
Types
The Good Fine Configuration
No need to restart or reload services
to pick up configuration changes.
No More Dr...
Custom
Types
The Bad Authentication
Somebody has to create and
maintain the custom types
More Dependencies
The module cont...
It’s not OK to manually configure
Bamboo instances
20
bamboo_rest
Automated away
These properties used to be
configured on each server manually.
Not anymore. Thanks Custom Types...
Nexus instances
18
It’s not OK to allow config to drift on
nexus_rest
Seamless changes
Changes to these properties used to
cause Nexus to restart, resulting in
down time.
Not anymor...
Configuration
Only
These modules are only responsible
for configuring installed services via
REST APIs.
Installation of the...
IMPLEMENT A TYPE
SET UP
WRITE A PROVIDER
ADDING MORE PROPERTIES
Worked Example
INTERACTION BETWEEN TYPES
IMPLEMENT A TYPE
SET UP
WRITE A PROVIDER
ADDING MORE PROPERTIES
Worked Example
INTERACTION BETWEEN TYPES
Helpful Tools
• Puppet
• Ruby
• Bundler
• irb
• curl and Firebug
• Docker
"Tools" (CC BY-NC-ND 2.0) by jlgriffiths
What makes Nexus 3 a good
candidate?
• Completely different API to Nexus 2
• Easy to run using Docker
• A repository manag...
Our Nexus 3 Types
Hosted Maven repository.
Attributes
• name
• online
• blobstore
Abstract storage used by
repositories.
A...
Discover
Service
Endpoints
Useful Endpoints
• Query all resources
• Create new resource
• Delete existing resource
• Updat...
Query Repositories
Nexus Endpoint Example
File Structure
• lib/ - All code
• lib/puppet/ Puppet API code
• lib/puppet/provider/<type>/<provider>.rb

Provider definit...
IMPLEMENT A TYPE
SET UP
WRITE A PROVIDER
ADDING MORE PROPERTIES
Worked Example
INTERACTION BETWEEN TYPES
Creating a new type
It’s as simple as adding these two lines.
tag: step-01
Creating a new type
It’s as simple as adding these two lines.
tag: step-01
Creating a new type
It’s as simple as adding these two lines.
tag: step-01
It’s not quite ready.
At least we have our dev loop now.
tag: step-01
Make sure RUBYLIB contains the path
to lib/ before running puppet
It’s not quite ready.
At least we have our dev loop now....
puppet resource and puppet
apply are your friends
It’s not quite ready.
At least we have our dev loop now.
tag: step-01
puppet resource and puppet
apply are your friends
It’s not quite ready.
At least we have our dev loop now.
tag: step-01
Add namevar
A type’s namevar acts like a primary key
tag: step-02
Add namevar
A type’s namevar acts like a primary key
tag: step-02
tag: step-02
`puppet resource` won’t work until we
implement a provider
tag: step-02
`puppet apply` works because currently
nexus3_repository is not ensurable
tag: step-02
Inline documentation and ensurableness
Non-ensurable types (like exec) do not have persistent state.
tag: step-03
Inline documentation and ensurableness
Non-ensurable types (like exec) do not have persistent state.
tag: step-03
Inline documentation and ensurableness
Adding inline documentation is easy.
tag: step-03
Oops. We broke it again.
Looks like we need to implement a provider.
ensurable resources have state, so their
provider mus...
Oops. We broke it again.
Looks like we need to implement a provider.
ensurable resources have state, so their
provider mus...
IMPLEMENT A TYPE
SET UP
WRITE A PROVIDER
ADDING MORE PROPERTIES
Worked Example
INTERACTION BETWEEN TYPES
Before we start writing the provider…
We could do with some helper classes so we don’t get bogged
down in implementation d...
Before we start writing the provider…
We could do with some helper classes so we don’t get bogged
down in implementation d...
Before we start writing the provider…
We could do with some helper classes so we don’t get bogged
down in implementation d...
Before we start writing the provider…
We could do with some helper classes so we don’t get bogged
down in implementation d...
A basic provider
tag: step-05
A basic provider
the provider name is usually the name of the
underlying tool used to manage resources
tag: step-05
A basic provider
if implemented, self.instances is
used to enumerate all resources of type
tag: step-05
A basic provider
exists? provides logic for determining
the presence of a resource
tag: step-05
Aw yeah.
now that we’ve implemented a basic
provider, puppet resource is working
tag: step-05
Aw yeah.
now that we’ve implemented a basic
provider, puppet resource is working
tag: step-05
Aw yeah.
now that we’ve implemented a basic
provider, puppet resource is working
tag: step-05
Ability to create and destroy
tag: step-06
Ability to create and destroy
method used when Puppet determines
that a resource should be created
tag: step-06
Ability to create and destroy
method used when Puppet wants to
destroy a resource
tag: step-06
Now we’re getting somewhere.
tag: step-06
Now we’re getting somewhere.
purges any discovered resources
that are not in the manifest
tag: step-06
Now we’re getting somewhere.
ensures that repo_1 and number-
two are present
tag: step-06
Now we’re getting somewhere.
destroyed unmanaged resources
tag: step-06
Now we’re getting somewhere.
created new resources
tag: step-06
Now we’re getting somewhere.
tag: step-06
Well, sort of.
resources are being created again
even though they already exist
tag: step-06
Implement prefetch
`prefetch` must be defined if your `exists?` and getter methods don’t
query the system.
tag: step-07
Implement prefetch
`prefetch` must be defined if your `exists?` and getter methods don’t
query the system.
tag: step-07
Yep, now we’ve got it.
tag: step-07
Yep, now we’ve got it.
no resources recreated
tag: step-07
IMPLEMENT A TYPE
SET UP
WRITE A PROVIDER
ADDING MORE PROPERTIES
Worked Example
INTERACTION BETWEEN TYPES
Adding an attribute to the Type
tag: step-08
Adding an attribute to the Type
nexus3_repository type now has an online attribute
tag: step-08
Adding an attribute to the Type
input value restriction
tag: step-08
Adding an attribute to the Type
input value normalisation
tag: step-08
Adding an attribute to the Provider
tag: step-08
Adding an attribute to the Provider
fetch the new attribute value from system and normalise it
tag: step-08
Adding an attribute to the Provider
submit the new attribute to the system on create and update
tag: step-08
Adding an attribute to the Provider
tag: step-08
Adding an attribute to the Provider
property getters and setters
tag: step-08
Adding an attribute to the Provider
flush can be used to implement update logic
tag: step-08
Aw yeah.
tag: step-08
Aw yeah.
puppet resource is reporting online status
tag: step-08
Aw yeah.
puppet apply updated the online attribute
tag: step-08
Aw yeah.
puppet apply updated the online attribute
tag: step-08
Aw yeah.
puppet apply updated the online attribute
tag: step-08
Adding another attribute to the Type
tag: step-09
Adding another attribute to the Type
custom input validation logic
tag: step-09
Adding another attribute to the Provider
tag: step-09
Adding another attribute to the Provider
immutable property using fail method
tag: step-09
puppet resource is reporting blobstore value
tag: step-09
puppet resource is reporting blobstore value
tag: step-09
changing the blobstore value for an existing repository is failing
tag: step-09
changing the blobstore value for an existing repository is failing
tag: step-09
That’s
nexus3_repository
done.
"Silbury Hill" (CC BY 2.0) by rightee
We’re over the hump.
IMPLEMENT A TYPE
SET UP
WRITE A PROVIDER
ADDING MORE PROPERTIES
Worked Example
INTERACTION BETWEEN TYPES
We’ve created another Type
This one is called nexus3_blobstore
We won’t walk through it like we did for nexus3_repository....
Mostly good. But…
tag: step-10
tag: step-10
tag: step-10
nexus3_repositorys instances are being created before
nexus3_blobstore instances. not good.
tag: step-10
nexus3_repository instances are being created before
nexus3_blobstore instances. not good.
tag: step-10
Relationships between Types
Types can define soft require relationships with arbitrary resources
tag: step-11
Relationships between Types
Types can define soft require relationships with arbitrary resources
if puppet manages this con...
Relationships between Types
Types can define soft require relationships with arbitrary resources
the associated nexus3_blob...
Better.
tag: step-11
blobstore created before each dependent repository
tag: step-11
Even better.
tag: step-11
allow Puppet to manage the config file
tag: step-11
prefetch fails due to missing provider configuration file
tag: step-11
Puppet creates config file, then succeeds to create resources in the
right order using our custom types and providers
tag:...
IMPLEMENT A TYPE
SET UP
WRITE A PROVIDER
ADDING MORE PROPERTIES
Worked Example
INTERACTION BETWEEN TYPES
Done 👌
https://bitbucket.org/TCinel/puppet-camp-example/
Demo
Next Steps
Areas for
Improvement
Publish to Forge
Incorporate better error handling,
logging.
Tests
Unit tests with RSpec, acceptance...
Puppet SourceShit Garry SaysPuppet Types 

and Providers http://garylarizza.com/blog/
Other Resources
Now go, and write
some custom types
already!
Thank you
TIM CINEL • BUILD WHISPERER • ATLASSIAN • @TIMCINEL
https://bitbucket.org/TCinel/puppet-camp-example/
Upcoming SlideShare
Loading in …5
×

PuppetConf 2016: Writing Custom Types to Manage Web-Based Applications – Tim Cinel, Atlassian

203 views

Published on

Here are the slides from Tim Cinel's PuppetConf 2016 presentation called Writing Custom Types to Manage Web-Based Applications. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa

Published in: Technology
  • Be the first to comment

  • Be the first to like this

PuppetConf 2016: Writing Custom Types to Manage Web-Based Applications – Tim Cinel, Atlassian

  1. 1. TIM CINEL • BUILD WHISPERER • ATLASSIAN • @TIMCINEL Writing Custom Types to Manage Web-Based Applications
  2. 2. OUR SITUATION WORKED EXAMPLE DEMO This Talk NEXT STEPS
  3. 3. Our Situation
  4. 4. Atlassian BambooSonatype Nexus Atlassian’s Build Engineering team uses Puppet to manage manage many services, including… • Repositories • Security • Proxy features • Variables • Global defaults • Security
  5. 5. Managing config files has drawbacks… • Nexus’s config file changes are picked up after restart • Changes require brief service outage • Changes made using Nexus and Bamboo UI during runtime not detected by Puppet • Risk of configuration drift • Most Bamboo configuration is in its database, not configuration files • Ability to manage service is limited Uptime Drift Limited
  6. 6. How do we address these drawbacks? Custom Types.
  7. 7. Concepts Refresher
  8. 8. Concept and terminology refresher • Type • Name • Attributes Can be considered as instances of types Native Type • Ruby implementation • Attributes • Provider(s) Allows the representation of a type of resource in Puppet DSL • Ruby implementation • Type Providers interact with target system to manage resources Resource Type Provider
  9. 9. What Are Custom Types? user Type
 (Default) useradd Provider for user Type Manifest bamboo_agent Type
 (Custom) rest Provider for bamboo_agent Type Manifest
  10. 10. What Are Custom Types? user Type
 (Default) useradd Provider for user Type Manifest bamboo_agent Type
 (Custom) rest Provider for bamboo_agent Type Manifest
  11. 11. Custom Types can be created to manage web-based services.
  12. 12. Custom Types The Good Fine Configuration No need to restart or reload services to pick up configuration changes. No More Drift Custom types can directly query the application state runtime. Fewer Restarts If it’s configurable, a custom type can manage it. No need to manually tweak service configuration.
  13. 13. Custom Types The Bad Authentication Somebody has to create and maintain the custom types More Dependencies The module containing the custom types, custom providers and libraries Engineering Cost Configuration APIs often require credentials
  14. 14. It’s not OK to manually configure Bamboo instances 20
  15. 15. bamboo_rest Automated away These properties used to be configured on each server manually. Not anymore. Thanks Custom Types! https://forge.puppetlabs.com/atlassian/bamboo_rest
  16. 16. Nexus instances 18 It’s not OK to allow config to drift on
  17. 17. nexus_rest Seamless changes Changes to these properties used to cause Nexus to restart, resulting in down time. Not anymore. Thanks Custom Types! https://forge.puppetlabs.com/atlassian/nexus_rest
  18. 18. Configuration Only These modules are only responsible for configuring installed services via REST APIs. Installation of these services is managed by another module.
  19. 19. IMPLEMENT A TYPE SET UP WRITE A PROVIDER ADDING MORE PROPERTIES Worked Example INTERACTION BETWEEN TYPES
  20. 20. IMPLEMENT A TYPE SET UP WRITE A PROVIDER ADDING MORE PROPERTIES Worked Example INTERACTION BETWEEN TYPES
  21. 21. Helpful Tools • Puppet • Ruby • Bundler • irb • curl and Firebug • Docker "Tools" (CC BY-NC-ND 2.0) by jlgriffiths
  22. 22. What makes Nexus 3 a good candidate? • Completely different API to Nexus 2 • Easy to run using Docker • A repository manager is a good fit with the Puppet resource model Nexus 3 We’e going to create custom types for Nexus 3 in our worked example.
  23. 23. Our Nexus 3 Types Hosted Maven repository. Attributes • name • online • blobstore Abstract storage used by repositories. Attributes • name • path nexus3_repository nexus3_blobstore
  24. 24. Discover Service Endpoints Useful Endpoints • Query all resources • Create new resource • Delete existing resource • Update existing resource Before starting work on a custom type, we should confirm the necessary endpoints are available.
  25. 25. Query Repositories Nexus Endpoint Example
  26. 26. File Structure • lib/ - All code • lib/puppet/ Puppet API code • lib/puppet/provider/<type>/<provider>.rb
 Provider definition • lib/puppet/type/<type>.rb
 Type definition • lib/puppet_x helper code
  27. 27. IMPLEMENT A TYPE SET UP WRITE A PROVIDER ADDING MORE PROPERTIES Worked Example INTERACTION BETWEEN TYPES
  28. 28. Creating a new type It’s as simple as adding these two lines. tag: step-01
  29. 29. Creating a new type It’s as simple as adding these two lines. tag: step-01
  30. 30. Creating a new type It’s as simple as adding these two lines. tag: step-01
  31. 31. It’s not quite ready. At least we have our dev loop now. tag: step-01
  32. 32. Make sure RUBYLIB contains the path to lib/ before running puppet It’s not quite ready. At least we have our dev loop now. tag: step-01
  33. 33. puppet resource and puppet apply are your friends It’s not quite ready. At least we have our dev loop now. tag: step-01
  34. 34. puppet resource and puppet apply are your friends It’s not quite ready. At least we have our dev loop now. tag: step-01
  35. 35. Add namevar A type’s namevar acts like a primary key tag: step-02
  36. 36. Add namevar A type’s namevar acts like a primary key tag: step-02
  37. 37. tag: step-02
  38. 38. `puppet resource` won’t work until we implement a provider tag: step-02
  39. 39. `puppet apply` works because currently nexus3_repository is not ensurable tag: step-02
  40. 40. Inline documentation and ensurableness Non-ensurable types (like exec) do not have persistent state. tag: step-03
  41. 41. Inline documentation and ensurableness Non-ensurable types (like exec) do not have persistent state. tag: step-03
  42. 42. Inline documentation and ensurableness Adding inline documentation is easy. tag: step-03
  43. 43. Oops. We broke it again. Looks like we need to implement a provider. ensurable resources have state, so their provider must have logic to determine state tag: step-03
  44. 44. Oops. We broke it again. Looks like we need to implement a provider. ensurable resources have state, so their provider must have logic to determine state tag: step-03
  45. 45. IMPLEMENT A TYPE SET UP WRITE A PROVIDER ADDING MORE PROPERTIES Worked Example INTERACTION BETWEEN TYPES
  46. 46. Before we start writing the provider… We could do with some helper classes so we don’t get bogged down in implementation details. tag: step-04
  47. 47. Before we start writing the provider… We could do with some helper classes so we don’t get bogged down in implementation details. helper classes in puppet_x tag: step-04
  48. 48. Before we start writing the provider… We could do with some helper classes so we don’t get bogged down in implementation details. providers only need to use this class method tag: step-04
  49. 49. Before we start writing the provider… We could do with some helper classes so we don’t get bogged down in implementation details. lazily load application config from file tag: step-04
  50. 50. A basic provider tag: step-05
  51. 51. A basic provider the provider name is usually the name of the underlying tool used to manage resources tag: step-05
  52. 52. A basic provider if implemented, self.instances is used to enumerate all resources of type tag: step-05
  53. 53. A basic provider exists? provides logic for determining the presence of a resource tag: step-05
  54. 54. Aw yeah. now that we’ve implemented a basic provider, puppet resource is working tag: step-05
  55. 55. Aw yeah. now that we’ve implemented a basic provider, puppet resource is working tag: step-05
  56. 56. Aw yeah. now that we’ve implemented a basic provider, puppet resource is working tag: step-05
  57. 57. Ability to create and destroy tag: step-06
  58. 58. Ability to create and destroy method used when Puppet determines that a resource should be created tag: step-06
  59. 59. Ability to create and destroy method used when Puppet wants to destroy a resource tag: step-06
  60. 60. Now we’re getting somewhere. tag: step-06
  61. 61. Now we’re getting somewhere. purges any discovered resources that are not in the manifest tag: step-06
  62. 62. Now we’re getting somewhere. ensures that repo_1 and number- two are present tag: step-06
  63. 63. Now we’re getting somewhere. destroyed unmanaged resources tag: step-06
  64. 64. Now we’re getting somewhere. created new resources tag: step-06
  65. 65. Now we’re getting somewhere. tag: step-06
  66. 66. Well, sort of. resources are being created again even though they already exist tag: step-06
  67. 67. Implement prefetch `prefetch` must be defined if your `exists?` and getter methods don’t query the system. tag: step-07
  68. 68. Implement prefetch `prefetch` must be defined if your `exists?` and getter methods don’t query the system. tag: step-07
  69. 69. Yep, now we’ve got it. tag: step-07
  70. 70. Yep, now we’ve got it. no resources recreated tag: step-07
  71. 71. IMPLEMENT A TYPE SET UP WRITE A PROVIDER ADDING MORE PROPERTIES Worked Example INTERACTION BETWEEN TYPES
  72. 72. Adding an attribute to the Type tag: step-08
  73. 73. Adding an attribute to the Type nexus3_repository type now has an online attribute tag: step-08
  74. 74. Adding an attribute to the Type input value restriction tag: step-08
  75. 75. Adding an attribute to the Type input value normalisation tag: step-08
  76. 76. Adding an attribute to the Provider tag: step-08
  77. 77. Adding an attribute to the Provider fetch the new attribute value from system and normalise it tag: step-08
  78. 78. Adding an attribute to the Provider submit the new attribute to the system on create and update tag: step-08
  79. 79. Adding an attribute to the Provider tag: step-08
  80. 80. Adding an attribute to the Provider property getters and setters tag: step-08
  81. 81. Adding an attribute to the Provider flush can be used to implement update logic tag: step-08
  82. 82. Aw yeah. tag: step-08
  83. 83. Aw yeah. puppet resource is reporting online status tag: step-08
  84. 84. Aw yeah. puppet apply updated the online attribute tag: step-08
  85. 85. Aw yeah. puppet apply updated the online attribute tag: step-08
  86. 86. Aw yeah. puppet apply updated the online attribute tag: step-08
  87. 87. Adding another attribute to the Type tag: step-09
  88. 88. Adding another attribute to the Type custom input validation logic tag: step-09
  89. 89. Adding another attribute to the Provider tag: step-09
  90. 90. Adding another attribute to the Provider immutable property using fail method tag: step-09
  91. 91. puppet resource is reporting blobstore value tag: step-09
  92. 92. puppet resource is reporting blobstore value tag: step-09
  93. 93. changing the blobstore value for an existing repository is failing tag: step-09
  94. 94. changing the blobstore value for an existing repository is failing tag: step-09
  95. 95. That’s nexus3_repository done.
  96. 96. "Silbury Hill" (CC BY 2.0) by rightee We’re over the hump.
  97. 97. IMPLEMENT A TYPE SET UP WRITE A PROVIDER ADDING MORE PROPERTIES Worked Example INTERACTION BETWEEN TYPES
  98. 98. We’ve created another Type This one is called nexus3_blobstore We won’t walk through it like we did for nexus3_repository. tag: step-10
  99. 99. Mostly good. But… tag: step-10
  100. 100. tag: step-10
  101. 101. tag: step-10
  102. 102. nexus3_repositorys instances are being created before nexus3_blobstore instances. not good. tag: step-10
  103. 103. nexus3_repository instances are being created before nexus3_blobstore instances. not good. tag: step-10
  104. 104. Relationships between Types Types can define soft require relationships with arbitrary resources tag: step-11
  105. 105. Relationships between Types Types can define soft require relationships with arbitrary resources if puppet manages this config file, then all nexus3_repository resources will depend on it tag: step-11
  106. 106. Relationships between Types Types can define soft require relationships with arbitrary resources the associated nexus3_blobstore resource will become a dependency (if it exists) tag: step-11
  107. 107. Better. tag: step-11
  108. 108. blobstore created before each dependent repository tag: step-11
  109. 109. Even better. tag: step-11
  110. 110. allow Puppet to manage the config file tag: step-11
  111. 111. prefetch fails due to missing provider configuration file tag: step-11
  112. 112. Puppet creates config file, then succeeds to create resources in the right order using our custom types and providers tag: step-11
  113. 113. IMPLEMENT A TYPE SET UP WRITE A PROVIDER ADDING MORE PROPERTIES Worked Example INTERACTION BETWEEN TYPES Done 👌
  114. 114. https://bitbucket.org/TCinel/puppet-camp-example/
  115. 115. Demo
  116. 116. Next Steps
  117. 117. Areas for Improvement Publish to Forge Incorporate better error handling, logging. Tests Unit tests with RSpec, acceptance tests with Beaker and some CI. Robustness Make the module and its glorious types available to the community.
  118. 118. Puppet SourceShit Garry SaysPuppet Types 
 and Providers http://garylarizza.com/blog/ Other Resources
  119. 119. Now go, and write some custom types already!
  120. 120. Thank you TIM CINEL • BUILD WHISPERER • ATLASSIAN • @TIMCINEL https://bitbucket.org/TCinel/puppet-camp-example/

×