PuppetDB: Sneaking Clojure into Operations

1,437 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,437
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
12
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

PuppetDB: Sneaking Clojure into Operations

  1. 1. The life and times ofPuppetDB Clojure/West 2013
  2. 2. deepakgiridharagopaldeepak@puppetlabs.com@grim_radical [github twitter freenode]
  3. 3. Let’s talk about...
  4. 4. Immutability is great!
  5. 5. Immutability allowsfor invariants, whichhelp you reason aboutcorrectness
  6. 6. Immutability preventsspooky action at adistance
  7. 7. Immutability fostersmodular, composableabstractions
  8. 8. (not a tough sell to Clojurists*) *clojurians?
  9. 9. That’s great fordeveloment, but how about operations?
  10. 10. Immutability forinfrastructure?Because operations is in thesame boat as development
  11. 11. Everyone who’s gottheir app running ona fleet of servers hasexperienced spookyaction at a distance
  12. 12. Known, good state iscritical for reliableupgrades
  13. 13. A lack of predictabilityin your systemsruins automation andabstraction
  14. 14. The problem is that: Systems are inherently mutable!But ideally: Systems should behave as though they weren’t!
  15. 15. façade of immutability. Immutability
  16. 16. Computer systems are in many waysopen systems, providing the keys tothe vault if one is so inclined to grabthem. But in order to foster an air ofimmutability in our own systems,its of utmost importance to create afaçade of immutability. Immutability -- The Joy of Clojurerequires that we layer over andabstract the parts of our system thatprovide unrestrained mutability.
  17. 17. Describe how you’dlike your systems tolook, and Puppetdoes all the hardwork for you!
  18. 18. file { “/etc/issue”: content => “Got an issue? Here’s a tissue!”,}file { “/etc/motd”: content => template(“Welcome to $hostname!”),}
  19. 19. file { "/etc/sudoers": owner => root, group => root, mode => 440, source => "puppet:///modules/sudo/sudoers"}
  20. 20. package { ntp: ensure => installed,}service { ntpd: ensure => running, enable => true, subscribe => File[/etc/ntp.conf],}file { /etc/ntp.conf: ensure => file, require => Package[ntp], source => "puppet:///modules/ntp/ntp.conf",}
  21. 21. class ntp { package { ntp: ensure => installed, } service { ntpd: ensure => running, enable => true, subscribe => File[/etc/ntp.conf], } file { /etc/ntp.conf: ensure => file, require => Package[ntp], source => "puppet:///modules/ntp/ntp.conf", }}
  22. 22. node “webserver.mydomain.com” { include ntp}node “appserver.mydomain.com” { include ntp}node “database.mydomain.com” { include ntp}
  23. 23. class ssh { @@sshkey { $hostname: type => dsa, key => $sshdsakey } Sshkey <<| |>>}
  24. 24. File “/tmp/foo/bar” User “deepak” Dir “/tmp/foo” Dir “/tmp”
  25. 25. Dir “/tmp” User “deepak” Dir “/tmp/foo” File “/tmp/foo/bar”
  26. 26. Dir “/tmp” User “deepak” Dir “/tmp/foo” File “/tmp/foo/bar”
  27. 27. Idempotent, andonly does what’snecessary
  28. 28. Compensates for theinherent mutabilityof systems
  29. 29. Combats spookyaction at a distancewith automaticrepair
  30. 30. Brings predictabilityto your systems
  31. 31. A foundation ofpredictability andreliability lets youperform higher-leveloperations on yourinfrastructure
  32. 32. V UME OL
  33. 33. every resource every parameter every relationship every fact for every nodeupdated all the time
  34. 34. Users leverage thisdata to do higher-order things withtheir infrastructure
  35. 35. key distribution monitoring clustered servicesmaster/slave replication load balancers shared filesystems firewall rules ...
  36. 36. Infrastructure ascode
  37. 37. Infrastructure asdata
  38. 38. User demand: Store as much data as we can! Much better queryability!Oh yeah, but: Don’t slow down the system! Don’t compromise reliability!
  39. 39. We can rebuild it, wehave the technology!
  40. 40. Speed is important Parsing, validating, and manipulating incoming data is computationally expensive
  41. 41. Speed is importantThe slower central storage is, theless agile sysadmins can be. That can cost money and uptime!
  42. 42. Reliability isimportant If it’s a critical part of managinginfrastructure, it’s got to be solid!
  43. 43. Deployment isimportant Should be easy to package, install, configure, use, monitor, and upgrade.
  44. 44. Wait, isn’t Puppetwritten in Ruby?
  45. 45. Lessons learned fromwriting the rest ofour software in RubyIt’s...not speedy. Object creation,method calls, garbage collection, etc. “Magical” APIs amplify the problem.
  46. 46. Lessons learned fromwriting the rest ofour software in Ruby Can only use one core! Workarounds compromise performance or simplicity
  47. 47. Lessons learned fromwriting the rest ofour software in Ruby Mutable state all over the damn place!
  48. 48. Lessons learned fromwriting the rest ofour software in Ruby Many struggles with the runtime. :(
  49. 49. -- Jeff Gagliardi
  50. 50. 1. It’s fast
  51. 51. 2. JVM libraries & Tools
  52. 52. 3. State & Parallelism
  53. 53. PuppetDB
  54. 54. PuppetDB Definitely Better!
  55. 55. Fast, safe storageof catalogs, facts,and events like, *way* faster!
  56. 56. HTTP APIsfor resource, fact,node, report retrieval plenty of data, just a “curl” away!
  57. 57. Storage &Querying
  58. 58. Storage &Querying
  59. 59. CQRS
  60. 60. CommandQueryResponsibilitySeparation use a different model to update information than the model you use to read information
  61. 61. Writes
  62. 62. CQRSwrite pipeline async, parallel, MQ-based, with automatic retry
  63. 63. { :command "replace catalog" :version 2 :payload {...}}
  64. 64. DelayedUUID /commands MQ Parse Process Dead Letter Office
  65. 65. (defmulti process-command!  (fn [{:keys [command version] :or {version 1}} _]    [command version]))(defmethod process-command! ["replace catalog" 1] [command options]  (replace-catalog* command options))
  66. 66. (defmulti process-command!  (fn [{:keys [command version] :or {version 1}} _]    [command version]))(defmethod process-command! ["replace catalog" 2]  [command options]  (replace-catalog* command options))(defmethod process-command! ["replace catalog" 1]  [command options] (-> command (update-in [:payload] catalog-v1->v2)   (replace-catalog* options))
  67. 67. Commandprocessors must beretry-aware expect failure, because it *will* happen.
  68. 68. (demo)
  69. 69. Reads
  70. 70. Queriesare expressed in theirown “language” domain specific, AST-based query language
  71. 71. ["and", ["=", "type", "User"], ["=", "title", "nick"]]
  72. 72. ["and", ["=", ["fact", "operatingsystem"], "Debian"], ["<", ["fact", "uptime_seconds"], 10000]]
  73. 73. ["and", ["=", "name", "ipaddress"], ["in", "certname", ["extract", "certname", ["select-resources", ["and", ["=", "type", "Class"], ["=", "title", "Apache"]]]]
  74. 74. ["or", ["=", "certname", "foo.com"], ["=", "certname", "bar.com"], ["=", "certname", "baz.com"]]
  75. 75. We use core.matchto walk the tree,compiling it to SQL
  76. 76. AST-based API letsusers write their ownlanguages ah, you’ve got to love open source!
  77. 77. (Package[httpd] and country=fr)or country=usPackage["mysql-server"]and architecture=amd64 Erik Dalén, Spotify https://github.com/dalen/puppet-puppetdbquery
  78. 78. AST-based API letsus more safelymanipulate queries
  79. 79. (def query-app  (app    [&]    {:get (fn [{:keys [params] :as request}]             (perform-query (params "query")))}))(def resources-app  (app   []   query-app   [type title &]   (comp query-app (partial http-q/restrict-resource-query-to-type type) (partial http-q/restrict-resource-query-to-title title))   [type &]   (comp query-app (partial http-q/restrict-resource-query-to-type type))))
  80. 80. (defn restrict-resource-query-to-type  [type query]  (conj ["and"          ["=" "type" type]]        query))
  81. 81. Storage
  82. 82. Relational Database,embedded orPostgreSQL because they’re actually pretty fantastic at ad-hoc queries, aggregation, windowing, etc. while maintaining safety
  83. 83. Relational Database,embedded orPostgreSQLwe use arrays, recursive queries, indexing inside complex structures
  84. 84. Relational Database,embedded orPostgreSQL schema is oriented towards the types of queries we encounter
  85. 85. Deployment
  86. 86. PuppetDB Server DLO DB Workers HTTP MQ
  87. 87. PuppetDB Server Workers DLO DB HTTP MQ
  88. 88. PuppetDB Server Workers DLOHTTP DBProxy(SSL) HTTP MQ
  89. 89. Codez
  90. 90. (defn process-commands!  "Connect to an MQ an continually, sequentially process commands. If the MQ consumption timeout is reached without any new data, the function will terminate."  [connection endpoint discarded-dir options-map]  (let [producer (mq-conn/producer connection)        publish (partial mq-producer/publish producer endpoint)        on-message (produce-message-handler publish discarded-dir options-map)        mq-error (promise)        consumer (mq-conn/consumer connection                                     {:endpoint endpoint                                      :on-message on-message                                      :transacted true                                      :on-failure #(deliver mq-error (:exception %))})]    (mq-cons/start consumer)    ;; Block until an exception is thrown by the consumer thread    (try      (deref mq-error)      (throw @mq-error)      (finally        (mq-cons/close consumer)))))
  91. 91. (defmacro with-error-delivery  "Executes body, and delivers an exception to the provided promise if one is thrown."  [error & body]  `(try     ~@body     (catch Exception e#       (deliver ~error e#))))
  92. 92. (defn filter-mbean  "Converts an mbean to a map. For attributes that cant be converted to JSON, return a string representation of the value."  [mbean]  {:post [(map? %)]}  (into {} (for [[k v] mbean]             (cond              ;; Nested structures should themselves be filtered              (map? v)              [k (filter-mbean v)]              ;; Cheshire can serialize to JSON anything that              ;; implements the JSONable protocol              (satisfies? JSONable v)              [k v]              :else              [k (str v)]))))
  93. 93. (defmacro with-test-broker  "Constructs and starts an embedded MQ, and evaluates `body` inside a `with-open` expression that takes care of connection cleanup and MQ tear-down. `name` - The name to use for the embedded MQ `conn-var` - Inside of `body`, the variable named `conn-var` contains an active connection to the embedded broker. Example: (with-test-broker "my-broker" the-connetion ;; Do something with the connection (prn the-connection)) "  [name conn-var & body]  `(let [dir# (fs/absolute-path (fs/temp-dir))         broker-name# ~name         conn-str# (str "vm://" ~name)         ^BrokerService broker# (mq/build-embedded-broker broker-name# dir#)]     (.setUseJmx broker# false)     (.setPersistent broker# false)     (mq/start-broker! broker#)     (try       (with-open [~conn-var (mq/connect! conn-str#)]         ~@body)       (finally         (mq/stop-broker! broker#)         (fs/delete-dir dir#)))))
  94. 94. Results
  95. 95. Thousands ofproductiondeploymentsSmall shops with a dozen hosts,large shops with thousands ofhosts, intercontinentaldeployments...
  96. 96. Thousands of deployments,Hundreds of threads per install,Zero deadlocks,Zero bugs involving clojure state companion Ruby code has ~10x the defect rate
  97. 97. Users dig the uberjar,makes deploymentstraightforward andless error-prone
  98. 98. Nobody cares that it’sClojure. Users justwant stuff to work.
  99. 99. Monitoring andinstrumentation is abig deal. Users wanteasy ways toconsume metrics.
  100. 100. If you build it, theywill come.
  101. 101. Open sourcehttp://github.com/puppetlabs/puppetdb
  102. 102. deepakgiridharagopaldeepak@puppetlabs.com@grim_radical [github twitter freenode] We’re hiring!

×