Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Grand Puppet Sub-Systems Tour - Nicholas Fagerlund, Puppet Labs

889 views

Published on

The Grand Puppet Sub-Systems Tour - Nicholas Fagerlund, Puppet Labs

Published in: Technology
  • Be the first to comment

The Grand Puppet Sub-Systems Tour - Nicholas Fagerlund, Puppet Labs

  1. 1. 2014 Presented by The Grand Puppet Sub-Systems Tour Nick Fagerlund Technical Writer / Weird Bugs | Puppet Labs @nfagerlund
  2. 2. Presented by PIPES
  3. 3. Presented by What does Puppet DO? • Gather system information • Compile a catalog • Apply catalog • Report
  4. 4. Presented by What does Puppet DO? AGENT • Gather system information • Apply catalog • Report MASTER • Compile a catalog
  5. 5. STOP 1: Command Line and the Presented by
  6. 6. Presented by Someone ran puppet agent
  7. 7. Presented by ??? • /opt/puppet/bin/puppet • Puppet::Util::CommandLine.new.execute • Puppet::Application.find • Puppet::Application::Agent.new • Puppet::Application::Agent#setup • Puppet::Application::Agent#run_command
  8. 8. Presented by #run_command # lib/puppet/application/agent.rb def run_command if options[:fingerprint] fingerprint else # It'd be nice to daemonize later, but we have to daemonize before # waiting for certificates so that we don't block daemon = daemonize_process_when(Puppet[:daemonize]) ! wait_for_certificates ! if Puppet[:onetime] onetime(daemon) else main(daemon) end
  9. 9. Presented by onetime() and main() # lib/puppet/application/agent.rb begin exitstatus = daemon.agent.run rescue => detail Puppet.log_exception(detail) end ! daemon.stop(:exit => false) ! if not exitstatus exit(1) elsif options[:detailed_exitcodes] then exit(exitstatus) else exit(0) end # lib/puppet/application/agent.rb def main(daemon) if Puppet[:listen] setup_listen(daemon) end Puppet.notice "Starting Puppet client version #{Puppet.version}" ! daemon.start end
  10. 10. Presented by Obtaining the daemon # lib/puppet/application/agent.rb def daemonize_process_when(should_daemonize) daemon = Puppet::Daemon.new(Puppet::Util::Pidlock.new(Puppet[:pidfile])) daemon.argv = @argv daemon.agent = @agent ! daemon.daemonize if should_daemonize ! daemon end
  11. 11. STOP 2: The Daemon and the Presented by
  12. 12. Presented by DAEMON LAND?! Nope Bye
  13. 13. Presented by Puppet::Agent • Run a thing
  14. 14. Presented by Puppet::Daemon • Run a thing that runs the other thing
  15. 15. Lots of layers of abstraction?? • Historical reasons! Presented by
  16. 16. Presented by STOP 3: The Configurer
  17. 17. Presented by Puppet::Configurer
  18. 18. Puppet::Configurer#run_internal Presented by # lib/puppet/configurer.rb def run_internal(options) # We create the report pre-­‐populated with default settings for # environment and transaction_uuid very early, this is to ensure # they are sent regardless of any catalog compilation failures or # exceptions. options[:report] ||= Puppet::Transaction::Report.new("apply", nil, @environment, @transaction_uuid) report = options[:report] init_storage ! Puppet::Util::Log.newdestination(report) … … …
  19. 19. Puppet::Configurer#run_internal Presented by # lib/puppet/configurer.rb … query_options = get_facts(options) unless query_options …
  20. 20. Puppet::Configurer#run_internal Presented by # lib/puppet/configurer/fact_handler.rb … facts = Puppet::Node::Facts.indirection.find(Puppet[:node_name_value], :environment => @environment) …
  21. 21. Puppet::Configurer#run_internal Presented by # lib/puppet/configurer.rb … unless catalog = prepare_and_retrieve_catalog(options, query_options) return nil end …
  22. 22. Puppet::Configurer#run_internal Presented by # lib/puppet/configurer.rb … apply_catalog(catalog, options) …
  23. 23. Presented by Where next? Retrieve Apply
  24. 24. STOP 4: The Transaction Presented by
  25. 25. Presented by (Puppet::Resource::Catalog)
  26. 26. Presented by Puppet::Resource::Catalog Catalog Resource Other Stuff
  27. 27. Presented by Puppet::Configurer (again) # lib/puppet/configurer.rb def convert_catalog(result, duration) catalog = result.to_ral catalog.finalize catalog.retrieval_duration = duration catalog.write_class_file catalog.write_resource_file catalog end
  28. 28. Presented by RAL Catalog
  29. 29. Presented by RAL Catalog # lib/puppet/resource/catalog.rb def to_catalog(convert) result = self.class.new(self.name, self.environment_instance) result.version = self.version map = {} resources.each do |resource| next if virtual_not_exported?(resource) next if block_given? and yield resource newres = resource.copy_as_resource newres.catalog = result if convert != :to_resource newres = newres.to_ral end # We can't guarantee that resources don't munge their names # (like files do with trailing slashes), so we have to keep track # of what a resource got converted to. map[resource.ref] = newres result.add_resource newres end
  30. 30. Presented by RAL Catalog Puppet::Resource ! ↓ ! (something) << Puppet::Type
  31. 31. Presented by RAL Catalog ???
  32. 32. Presented by RAL Catalog # lib/puppet/configurer.rb def apply_catalog(catalog, options) report = options[:report] report.configuration_version = catalog.version ! benchmark(:notice, "Finished catalog run") do catalog.apply(options) end ! report.finalize_report report end
  33. 33. Puppet::Resource::Catalog#apply Presented by # lib/puppet/resource/catalog.rb def apply(options = {}) Puppet::Util::Storage.load if host_config? ! transaction = create_transaction(options) ! begin transaction.report.as_logging_destination do transaction.evaluate end rescue Puppet::Error => detail !
  34. 34. Presented by Transaction ???
  35. 35. Presented by Transaction • Ral catalog • Prioritizer • Report
  36. 36. Presented by Transaction • Ral catalog • Prioritizer • Report • Event manager • Resource harness • Relationship graph
  37. 37. Presented by Transaction • Ral catalog • Prioritizer • Report • Event manager • Resource harness • Relationship graph • relationship_graph.traverse
  38. 38. Presented by Transaction • Ral catalog • Prioritizer • Report • Event manager • Resource harness • Relationship graph • relationship_graph.traverse • *ollies out*
  39. 39. Presented by STOP 5: The Indirector
  40. 40. Presented by Back to Puppet::Configurer • #prepare_and_retrieve_catalog
  41. 41. Presented by Back to Puppet::Configurer # lib/puppet/configurer.rb result = Puppet::Resource::Catalog.indirection.find( Puppet[:node_name_value], query_options.merge(:ignore_cache => true, :environment => @environment, :fail_on_404 => true) )
  42. 42. Presented by The Indirector ! Puppet::Resource::Catalog.indirection.find()
  43. 43. Presented by The Indirector ← ↑ ↓ ?
  44. 44. Presented by The Indirector • Globally configured • Abstract • Gets stuff for you
  45. 45. Presented by The Indirector # lib/puppet/resource/catalog.rb extend Puppet::Indirector indirects :catalog, :terminus_setting => :catalog_terminus
  46. 46. Presented by The Indirector • find • search • save • destroy
  47. 47. Presented by The Indirector • find • search • save • destroy • terminus_class • cache_class • lib/puppet/indirector/…
  48. 48. Presented by Puppet Agent • Catalog => REST • Report => REST • Facts => Facter
  49. 49. Presented by Puppet Apply • Catalog => Compiler • Report => Processor • Facts => Facter
  50. 50. Presented by Avoiding the cache # lib/puppet/configurer.rb result = Puppet::Resource::Catalog.indirection.find( Puppet[:node_name_value], query_options.merge(:ignore_cache => true, :environment => @environment, :fail_on_404 => true) )
  51. 51. Presented by The Indirector • Why?
  52. 52. Presented by The Indirector • Why? • Global? Rigid?
  53. 53. Presented by The Indirector • Why? • Global? Rigid? • IT MAKES SENSE IN CONTEXT
  54. 54. Presented by The Indirector • …Catalog.find( <FACTS> ) • REST terminus • HTTP POST request to master
  55. 55. STOP 6: The Puppet Master’s Presented by
  56. 56. Presented by Rack Web server ↓ Rack server ↓ “App” object that responds to #call() ↓ Actual application logic
  57. 57. Presented by config.ru $0 = "master" ! ARGV << "-­‐-­‐rack" ! ARGV << "-­‐-­‐confdir" << "/etc/puppet" ARGV << "-­‐-­‐vardir" << "/var/lib/puppet" ! require 'puppet/util/command_line' run Puppet::Util::CommandLine.new.execute !
  58. 58. Presented by Starting master w/ Rack # lib/puppet/application/master.rb def start_rack_master require 'puppet/network/http/rack' ! announce_start_of_master ! return Puppet::Network::HTTP::Rack.new() end
  59. 59. Presented by The app object # lib/puppet/network/http/rack.rb class Puppet::Network::HTTP::Rack def call(env) request = Rack::Request.new(env) response = Rack::Response.new Puppet.debug 'Handling request: %s %s' % [request.request_method, request.fullpath] ! begin Puppet::Network::HTTP::RackREST.new.process(request, response) rescue => detail # Send a Status 500 Error on unhandled exceptions. response.status = 500 response['Content-­‐Type'] = 'text/plain' response.write 'Internal Server Error: "%s"' % detail.message # log what happened Puppet.log_exception(detail, "Puppet Server (Rack): Internal Server Error: Unhandled Exception: "%s"" % detail.message) end response.finish
  60. 60. Presented by #process # lib/puppet/network/http/handler.rb def process(request, response) new_response = Puppet::Network::HTTP::Response.new(self, response) ! request_headers = headers(request) request_params = params(request) request_method = http_method(request) request_path = path(request) … !
  61. 61. Presented by #process # lib/puppet/network/http/handler.rb if route = @routes.find { |route| route.matches?(new_request) } route.process(new_request, new_response)
  62. 62. lib/puppet/network/http/api/v1.rb Presented by
  63. 63. lib/puppet/network/http/api/v1.rb • GET => indirection.find() • POST => indirection.find() (for catalogs) • (There was an issue with some servers rejecting GET parameters that were too long. Funny story about that.) • DELETE => indirection.destroy() • PUT => indirection.save() • GET to magical plural name => indirection.search() Presented by
  64. 64. Presented by BIG PICTURE Configurer calls catalog.indirection.find() ↓ HTTP POST request to master ↓ Request handler calls catalog.indirection.find() ↓ Compiler terminus handles it ↓
  65. 65. Presented by STOP 7: The Compiler
  66. 66. Presented by MOSTLY SKIPPING THIS Lexer ↓ Parser ↓ AST ↓ Evaluator ↓
  67. 67. Presented by THIS IS THE TEST “Compiler terminus to the catalog indirection” ! ???
  68. 68. Presented by THIS IS THE TEST GOOD WORK EVERYBODY!
  69. 69. Presented by Questions?
  70. 70. Presented by THANKS!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! •Hope we’ll see you next year in Portland!

×