Your SlideShare is downloading. ×
Beyond the Node: Arkestration with Noah
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Beyond the Node: Arkestration with Noah

5,757

Published on

Slides from my presentation at PuppetConf 2011.

Slides from my presentation at PuppetConf 2011.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,757
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
33
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Speaker Notes Next slide - plan
  • What problem is noah trying to help solve?
  • In the beginning we did things manually. This was generally regarded as a very bad idea.
  • We took notes! (Yes this is on an actual server right now as we speak)
  • We wrote handbooks!
  • Then we started writing Shell scripts!
  • This script is actually broken.
  • Then we got smart!
  • In the general scheme of things, we've evolved. We treat our infrastructure as code. We've adopted behaviors and practices that developers have been doing for years. We're green, ship it!
  • At DevOpsDay Mt. View, a revelation was made.
  • The response was generally overwhelmingly supportive....
  • Ideas were “refined”
  • This is the heart of “orchestration”. This is the problem space that needs addressing right now.
  • Orchestraton means many things however. Luke mentioned “Command and Control” for instance. Impedence mismatch – mention Capistrano and boatload of “deploy” tools. Something is missing. Mechanisms – traditional orchestration
  • Just a quick example use case to demonstrate “orchestration”
  • Znodes are easily grokable. We get “paths”. Programatically friendly. Watches are cool. Downsides:
  • Have always needed something LIKE ZK. Haven't always had the “luxury” of language bindings. Haven't always needed the full feature set. Another unofficial goal is that it work in a “disconnected” world.
  • Going over the basics of interacting with Noah. All of this is on the wiki.
  • Host – node, server Service – http/https/ftp OSI L3 analogy. An open port. Host/Service inspired by Nagios Application – tomcat, apache, rails app. OSI L7 analogy Configuration – httpd.conf, server.xml, database.yml
  • Ephemerals – Arbitrary blob of information Tag – tags...not much to say here Links – similar to symlinks. Create a custom namespace Watches – Async pluggable callbacks
  • Explain the general idea behind programatic friendly paths. Operate with standard HTTP verbs – GET, PUT, DELETE
  • Explain the general idea behind programmatic friendly paths.
  • Common fields in all GETs
  • This is a GET to /hosts/
  • Remember that services must be bound to hosts.
  • Note the asterix here
  • JSON version is default in most cases: Accept: application/json to ensure this representation application/octet will ensure that you get a proper mime typed version.
  • Reminder of the paths
  • Note that services has a requirement on hostname in the path.
  • This is what you PUT to each path in previous slide Maybe have to jump back and forth.
  • Same paths as put. No payload required.
  • Explain a bit how ZK watches work Mention race condition.
  • Made up of endpoints and patterns. Persistence will eventually be tunable Supersets If you declare a watch for endpoint on: /hosts You can't declare a watch for endpoint on: /hosts/foo_host Subsets are the reverse
  • This creates a watch for anything under /applications. When something “changes” under /applications, the change will be send to the endpoint.
  • Again, watches have the same common fields
  • Here we're going to add a tag to that new application
  • Note that every example listed does not actually exist YET
  • Creator unrelated to the reciever is important. Allows proxy management You can write your own at the cost of fragility
  • James wrote these. Need to be updated to newest Noah API. Mention that this was supposed to be done before this presentation but there was some “confusion” about the schedule - Functions pull data from/put in Noah - noah_data fact pulls configurations from Noah Cover the mappings
  • Taking Noah out of the mix for a moment, what's a traditional workflow for managing applications with Puppet?
  • Problems I've seen Different repos Dev interaction What happens when we add a new setting? How many people run puppet in daemon mode or via cron as opposed to on demand? Rampination.
  • Adding Noah in the mix Looks a bit more complicated. Noah becomes your System of Record Talk about Volatile settings
  • This is where I disagree with Luke a little bit. There are valid use cases for “dynamic” configuration. JMX exists.
  • Explain CI
  • We currently do this for deploys at VA using a curl script in a bash step You can either trigger the deploy itself or wrap it around a puppet run. Both can be done by CI. Upshot is that you get more templated CI jobs.
  • This is similar to the original example use case.
  • The nagios event handler, again, can be implemented as a curl script. If there's a watch attached, you can trigger any number of actions. You can have endpoints in your applications that make it dynamically respond.
  • This is an immediate win
  • Ghetto job dependencies Easier than dealing with mysql command lines and escaping.
  • ZK and Doozer are a bit more complex. Require persistent connections. NoSQL and Nesoi are nice because they're http and json!
  • Transcript

    • 1. Beyond the Node Advanced 'Arkestration' with Noah PuppetConf 2011
    • 2. .plan
    • 7.  
    • 8. A Brief History of Time (with apologies to Hawking)
    • 9. Configuration Management
      • Manual
    • 10. Configuration Management
    • 11. Configuration Management
    • 12. Configuration Management
    • 14. Configuration Management
    • 15. Configuration Management
    • 18. CFEngine
    • 19. Chef
    • 20. Puppet
    • 21. Configuration Management
    • 24. So we've solved this, right?
    • 25. “ I think..and I don't know if anyone would agree, that configuration management is a solved problem at this point, right?”
    • 26. “ I think..and I don't know if anyone would agree, that configuration management is a solved problem at this point, right?” WTF?
    • 27. "The point I want to make..Configuration management is not a solved problem...and it's dangerous to make the mistake to think that the way we do things now is the best way to do them..." - Andrew Clay Shafer
    • 28. “ what I was attempting to say ... is that the current crop of configuration management tools have reached a usable point where they do enough (for now). What we’re seeing as questions now are 'How do I think beyond the single node where this tool is running?'”
    • 29. “ what I was attempting to say (epic fail, I might add) is that the current crop of configuration management tools have reached a usable point where they do enough (for now). What we’re seeing as questions now are ' How do I think beyond the single node where this tool is running? '”
    • 30. Orchestration and Noah Noah attempts to do two things:
      • Bridge an impedance mismatch between operational, developmental and application configuration
      • 31. Provide mechanisms for coordination between applications, nodes, services, configuration management and other infrastructure aspects
    • 32. Example Use Case 1) Capacity Reached. Tell Noah 2) Noah tells provisioning system 3) Capacity allocated. Tell Noah 4) Noah triggers Puppet on LB 5) Noah triggers Puppet on Nagios Noah Nagios Puppet (app) Load Balancer
    • 33. “ Inspired” by
    • 34. "ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications."
    • 35. Features
      • Hierarchical name space of data registers similar to a Unix filesystem ( znodes )
      • 36. Leader election, locking, configuration, sequences
      • 37. Distributed and highly available
      • 38. Watches
      Downsides
      • Persistent connections required
      • 39. Official bindings are C and Java
      • 40. Somewhat "complex" to setup and run
    • 41. Noah and ZooKeeper Noah is essentially a re-imagining of ZK, taking the things I liked:
      • Znode concept
      • 42. Watches
      and making it work over HTTP using JSON
    • 43. Noah Basics
    • 44. The Primitives
      • Hosts
      • 45. Has status ( up , down , pending )
      • 46. Can have Services
      • 47. Services
        Has status ( up , down , pending )
      • Must belong to a Host
      • Applications
      • 48. Can have Configurations
      • 49. Configurations
      • 50. Can belong to many Applications
    • 51. The Not So Primitives
      • Ephemerals
      • 52. Identified by a “path”
      • 53. Tags
      • 54. Can be applied to any Primitive or Ephemeral
      • 55. Links
        Can be applied to any Primitive, Ephemeral or Tag
      • Watches
      • 56. Can be applied at any hierarchy or specific object
    • 57. URL Patterns
      • Hosts
        • / hosts / app01 / services / httpd
      • Services
        • / services / httpd / app02
      • Applications
        • / applications / my_app / configurations / log_level
      • Configurations
        • / configurations / log_level
    • 58. GET ting Data
    • 59. { "foo": { "created_at" : "2011-04-20 04:50:41 UTC", "updated_at" : "2011-04-20 04:50:41 UTC", "tags" : [ ], "id" : "4af53f8d-36d2-2158-c4a1-514ff3578048", "links" : [ ], "services": { }, "status": "up" }, "bar": { } }
    • 60. GET ting Data
      • Hosts
    • 61. { "foo" : { "created_at": "2011-04-20 04:50:41 UTC", "updated_at": "2011-04-20 04:50:41 UTC", "tags": [ ], "id": "4af53f8d-36d2-2158-c4a1-514ff3578048", "links": [ ], "services": { “ mysql”:”pending”, “ httpd”:”up” }, "status": "up" }, "bar": { } } Host Name
    • 62. { "foo": { "created_at": "2011-04-20 04:50:41 UTC", "updated_at": "2011-04-20 04:50:41 UTC", "tags": [ ], "id": "4af53f8d-36d2-2158-c4a1-514ff3578048", "links": [ ], "services" : { “ mysql”:”pending”, “ httpd”:”up” }, "status": "up" }, "bar": { } } Services
    • 63. GET ting Data
    • 65. { "foo_svc" : { "foo" : { "created_at": "2011-04-20 05:09:44 UTC", "updated_at": "2011-04-20 05:09:44 UTC", "tags": [ ], "id": "53504526-a5d7-a604-5fe8-1f8fd2e1aef0", "links": [ ], "status": "down" } }, "both": { } } Service name
    • 66. { "foo_svc" : { "foo" : { "created_at": "2011-04-20 05:09:44 UTC", "updated_at": "2011-04-20 05:09:44 UTC", "tags": [ ], "id": "53504526-a5d7-a604-5fe8-1f8fd2e1aef0", "links": [ ], "status": "down" } }, "both": { } } Host Name
    • 67. GET ting Data
    • 70. { "myrailsapp1" : { "created_at": "2011-04-21 06:57:42 UTC", "updated_at": "2011-04-21 06:57:42 UTC", "tags": [ "sample_data", "production" ], "id": "8774fead-e63a-8e5c-0453-1b5330bbd04c", "links": [ "/my_sample_organization" ], "configurations": { "database.yml": { "format": "yaml", "body": "development:n database: development_databasen adapter: mysqln username: dev_usern password: dev_passwordn" } } }, "myrestapp1": { } } App Name
    • 71. { "myrailsapp1": { "created_at": "2011-04-21 06:57:42 UTC", "updated_at": "2011-04-21 06:57:42 UTC", "tags": [ "sample_data", "production" ], "id": "8774fead-e63a-8e5c-0453-1b5330bbd04c", "links": [ "/my_sample_organization" ], "configurations" : { "database.yml": { "format": "yaml", "body": "development:n database: development_databasen adapter: mysqln username: dev_usern password: dev_passwordn" } } }, "myrestapp1": { } } Configs
    • 72. GET ting Data
    • 76. { "barconf1" : { "created_at": "2011-04-27 01:20:37 UTC", "format": "string", "body": "barbody1", "updated_at": "2011-04-27 01:20:37 UTC", "tags": [ ], "id": "aa183dce-30ec-2d7e-c6f9-f3385161a7e6", "links": [ ] }, "barconf2": { } }
    • 77. GET ting Data
    • 82. GET ting Data
      • Hosts
      • 83. Services
      • 84. Applications
      • 85. Configurations *
      • 86. Ephemerals *
      • 87. /ephemerals/my/config/setting returns the data stored there.
    • 88. PUT ting Data
      • / hosts /host_name
      • 89. / services /service_name/host_name
      • 90. / applications /application_name
      • 91. / configurations /configuration_name
      • 92. / ephemerals /arbitrary/path/to/whatever
    • 93. PUT ting Data
      • / hosts / host_name
      • 94. / services / service_name / host_name
      • 95. / applications / application_name
      • 96. / configurations / configuration_name
      • 97. / ephemerals / arbitrary / path / to / whatever
    • 98. PUT ting Data
      • Host {“status”:”up”}
      • 99. Service {“status”:”up”,”host_status”:”up”}
      • 100. Application {“name”:”app_name”}
      • 101. Configuration {“format”:”format_type”,”body”:”data”}
      • 102. Ephemeral raw data
    • 103. DELETE ing Data
      • /hosts/ host_name
      • 104. /services/ service_name / host_name
      • 105. /applications/ application_name
      • 106. /configurations/ configuration_name
      • 107. /ephemerals/ arbitrary / path / to / whatever
    • 108. Tagging
      • Append “tag” to any PUT url with PUT method
      • 109. e.g. ' /applications/some_application/tag'
      • 110. {“tags”:”sometag”} or {“tags”:[“tag1”,”tag2”]}
      • 111. Call DELETE instead of PUT to remove tags
      • 112. GET ' /tags ' for all tags or ' /tag/tagname ' for specific tag
    • 113. Linking
      • Append “link” to any PUT url with PUT method
      • 114. e.g. ' /applications/some_application/link'
      • 115. {“link_name”:”my_organization”}
      • 116. Call DELETE instead of PUT to remove links *
      • 117. GET ' /link_name ' for all linked objects *
      • 118. Designed to allow custom namespaces
    • 119. Watches
    • 120. What are Watches?
      • A bit different than ZK watches
      • 121. Async
      • 122. Persistent *
      • 123. Pluggable (URI pattern based)
      • 124. Hierarchical (superset and subset)
      • 125. Append ' /watch ' to almost any url path
      • 126. {“endpoint”:”http://some_host:port/ping_me”}
      • 127. ' /watch ' root endpoint as well
      • 128. {
      • 129. “ endpoint”:”http://some_host/ping”,
      • 130. ” pattern”:”//noah/applications/app_name”
      • 131. }
    • 132. Create a Watch $ curl -XPUT -d '{“pattern”:”//noah/applications”,"endpoint":" http://host/ping "}' http://noahserver/watches
    • 133. Perform some operation $ curl -XPUT -d '{"name":"fooapp2"}' http://noahserver/applications/fooapp2
    • 134. Message sent to Endpoint { "id":"be9f6a45-0d49-3f00-59a8-c6ad61da24e1", "tags":[], "links":[], "name":"fooapp2" , "created_at":"2011-09-15 03:07:39 UTC", "updated_at":"2011-09-15 03:07:39 UTC", "configurations":{}, "action":"create" , "pubcategory":"//noah/applications/fooapp2" }
    • 135. Perform another operation $ curl -XPUT -d '{"tags":"footag2"}' http://noahserver/applications/fooapp2/tag
    • 136. Message sent to Endpoint { "id":"be9f6a45-0d49-3f00-59a8-c6ad61da24e1", "tags":["footag"] , "links":[], "name":"fooapp2" , "created_at":"2011-09-15 03:07:39 UTC", "updated_at":"2011-09-15 03:15:03 UTC", "configurations":{}, "action":"update" , "pubcategory":"//noah/applications/fooapp2" }
    • 137. Watches are “plugin” based You can write your own plugins with Ruby.
      • URI Pattern *
      • 138. amqp://user:pass@host:port/exchange
      • 139. rundeck: //token@host :port/job_id
      • 140. puppet://hostname/
      • 141. statsd://host:port/metric.path.name.action
      • 142. Currently based on Event Machine
      • 143. Future based on Actors and worker pools
    • 144. Watch Notes
      • Creator of the watch is unrelated to receiver
      • 145. Don't have to use the “official” watch daemon
      • 146. You can watch the watches (PubSub)
    • 147. Basic Puppet Integration
      • http://github.com/jamtur01/puppet-noah
      • 148. ENC, Functions and Facts
      • 149. Hosts -> Nodes
      • 150. Host's Services -> Classes
      • 151. Configurations -> Facts (string types only)
    • 152. So where does it fit? Noah Puppet Apps Scripts Monitoring Deploy CI
    • 153. Noah + Puppet + Applications Noah Puppet Applications
    • 154. Common P+A Use Case
      • Puppet knows Nodes and Classes
      • 155. App has configuration file
      • 156. Configuration file controlled by Puppet
      • 157. Puppet client run required to update file
    • 158. Common P+A Issues
      • Puppet repo is not app repo
      • 159. Devs must learn ERB
      • 160. Devs might even have to learn Puppet
      • 161. Introducing new settings and communication
      • 162. Daemon-mode is exception rather than rule
    • 163. N+P+A Version
      • Noah knows Nodes and Classes
      • 164. Move volatile settings into Noah
      • 165. Configuration file controlled by Puppet
      • 166. App gets volatile settings from Noah *
      • 167. Future Puppet runs update Noah
      • 168. Noah tells App *
      • 169. Truth vs. Reality
      • 170. My DB master is foo
      • 171. My DB master right NOW is bar
    • 172. Volatile Settings?
      • Not all configuration is created equal
      • 173. “ Elasticity ” - Pool of worker node EC2 ips
      • 174. Ad-hoc configuration – Logging levels
      • 175. Environment aware – Nodes of class “foo”
    • 176. N+P+A Benefits
      • Devs don't HAVE to learn ERB
      • 177. Devs don't HAVE to learn Puppet
      • 178. Ops and Dev friendly bridge point
      • 179. No unintended changes released
    • 180. Integration with CI Noah Puppet CI Deploy
    • 181. Integration with CI
      • Jenkins asks Noah for nodes of class “foo”
      • 182. Jenkins triggers Puppet (or deploy) on appropriate nodes
      • 183. Jenkins is always “current”
    • 184. Integration with Monitoring Noah Puppet Monitoring App
    • 185. Integration with Monitoring
      • Puppet manages Nagios configs
      • 186. Nagios updates Noah status (event handler)
      • 187. Noah optionally fires watches based on status change
      • 188. Endpoint is application. App “reconfigures”
    • 189. Integration with Scripts Noah Scripts (node 1) Scripts (node 2) Scripts (node 3)
    • 190. Integration with Scripts
      • Provide “locking” and “coordination” of cron jobs across hosts
      • 191. Stop dealing with lock files on host
      • 192. Use ephemerals and curl
    • 193. Gotchas
      • Noah itself is NOT distributed *
      • 194. Data redundancy based on Redis
      • 195. No ACLs *
      • 196. Async
    • 197. Other options
    • 202. Questions?
    • 203. Noah
      • Github: http://github.com/lusis /Noah
      • 204. IRC: #noah
      • 205. Blog(s): http://blog.lusis.org/blog/categories/noah
            • http://lusislog.blogspot.com/search/label/noah
    • 206. John E. Vincent
      • Twitter: @lusis
      • 207. Github: http://github.com/lusis
      • 208. IRC: lusis
      • 209. LinkedIn: http://linkedin.com/in/lusis
      • 210. Blog(s): http://blog.lusis.org
            • http://lusislog.blogspot.com
    • 211. Thank You!

    ×