Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Applying software engineering to configuration management

4,173 views

Published on

"Applying software engineering to configuration management" at Cfgmgmtcamp 2015

http://cfgmgmtcamp.eu/

  • Be the first to comment

Applying software engineering to configuration management

  1. 1. Applying software engineering to configuration management Bart Vanbrabant, PhD iMinds-Distrinet / Impera
  2. 2. ● Research started with PoDIM by Thomas Delaet ● LISA 07: Best paper award with PoDIM ● Master thesis 2008: apply PoDIM concepts to Puppet ● Security in cfg mgmt side step ● LISA 10: A survey of system configuration tools ● Summer 2010: First prototype of Impera tool ● June 2014: My PhD “A framework for integrated configuration management of distributed systems” ● Late 2014: Open Source release on Github and University spin-off: https://github.com/impera-io About
  3. 3. Why configuration management? ● Automate configuration (deployment, configuration and re-configuration) of software ● Keep all configuration parameters consistent to avoid errors ● Cloud computing has increased scale and dynamics ● Continuous deployment increases dynamics even more ● Application in cloud are distributed systems (SOA)
  4. 4. Managed applications ● Applications are business driver of why the operations are there in the first place ● Application architecture: o components o services that interact o logical architecture o abstract deployment architecture
  5. 5. Configuration management ● Current tools focus on managing individual components on machines o files o packages o services ● Abstraction level mismatch between o how the application is architected and developed o how it is operated ● Both configuration management and monitoring are in function of system resources
  6. 6. Abstraction blend ● Operations already a blend of abstraction levels ● Mixing very high level with very low level o e.g. large website on Amazon o own web servers (managed in function of low-level resources) o combined with ELB (services, managed through API) ● Blend will only increase o mash up of as-a-service o missing components self-hosted on virtual machines
  7. 7. One tool (model) to rule them all ● Inconsistencies in configuration cause errors and downtime ● Many interdependencies between services and components ● One integrated configuration model ● Highest and lowest abstraction level in a single model (and everything in between)
  8. 8. Software engineering (SE) ● SE has been raising the level of abstraction ● Our approach: apply concepts used in SE to cfg mgmt: o object-oriented programming o encapsulation o interfaces o component-based software development o refinement o …
  9. 9. Impera tool ● Declarative configuration model ● Object-oriented modelling ● Relations as first class citizen ● Separation of type (interface) and implementation (refinement) to build up abstractions ● Re-usable modules ● Plug-in mechanism: o transformation in configuration model o export to 3rd party o resource handlers ● Transparent orchestration
  10. 10. Entities and relations entity Server: number listen_port end
  11. 11. Entities and relations entity Server: number listen_port end entity WebServer extends ip::services::Server, web::Container string document_root end
  12. 12. Entities and relations entity Server: number listen_port end entity WebServer extends ip::services::Server, web::Container string document_root end std::Host host [1] -- [0:] Server servers Client clients [0:] -- [1:] Server server
  13. 13. Entities and relations entity Server: number listen_port end entity WebServer extends ip::services::Server, web::Container string document_root end std::Host host [1] -- [0:] Server servers Client clients [0:] -- [1:] Server server index WebServer(host, listen_port)
  14. 14. Entities and relations ... entity WebServer extends ip::services::Server, web::Container string document_root end std::Host host [1] -- [0:] Server servers Client clients [0:] -- [1:] Server server index WebServer(host, listen_port) typedef HttpWebServer as WebServer(listen_port=80)
  15. 15. Instantiating a model srv1 = std::Host(name=”server-1”, os=fedora::fedora21) webserver = WebServer(host=serv1, document_root=”/var/www”)
  16. 16. Refinement implementation apacheFedoraWebServer for WebServer: p_apache = std::Package(host=host, name=”httpd”, state=”installed”) f_cfg = std::ConfigFile(host=host, path=”/etc/httpd/conf/httpd.conf”, content=std::template(“apache/httpd.conf”), reload=true, requires=p_apache) f_svc = std::Service(host=host, name=”httpd”, state=”running”, onboot=true, requires=[f_cfg, p_apache]) end
  17. 17. implementation apacheFedoraWebServer for WebServer: p_apache = std::Package(host=host, name=”httpd”, state=”installed”) f_cfg = std::ConfigFile(host=host, path=”/etc/httpd/conf/httpd.conf”, content=std::template(“apache/httpd.conf”), reload=true, requires=p_apache) f_svc = std::Service(host=host, name=”httpd”, state=”running”, onboot=true, requires=[f_cfg, p_apache]) end implement WebServer using apacheFedoraWebServer when std::familyof(host.os, "fedora") Refinement
  18. 18. Refinement implementation apacheFedoraWebServer for WebServer: p_apache = std::Package(host=host, name=”httpd”, state=”installed”) f_cfg = std::ConfigFile(host=host, path=”/etc/httpd/conf/httpd.conf”, content=std::template(“apache/httpd.conf”), reload=true, requires=p_apache) f_svc = std::Service(host=host, name=”httpd”, state=”running”, onboot=true, requires=[f_cfg, p_apache]) end implement WebServer using apacheFedoraWebServer when std::familyof(host.os, "fedora")
  19. 19. Instantiating a model srv1 = std::Host(name=”server-1”, os=fedora::fedora21) webserver = WebServer(host=serv1, document_root=”/var/www”) Result: ● ensure that the httpd package is installed ● ensure that the httpd.conf file has the content generated using a template ● ensure that the httpd server is running and starts at boot
  20. 20. Building a model ● Configuration modules with: o model:  types: entities, relations, …  refinement o templates o files o plugins: model, resource handlers, dependency managers, … ● Initial model instantiates types ● Refine model until ready
  21. 21. Refinement
  22. 22. Refinement
  23. 23. Derive configuration ● Implicit configuration ● Derive configuration from integrated configuration model o firewall rules based on network topology, network connection and security policy o monitoring configuration o backup o … ● Get this configuration for “free”
  24. 24. Derive configuration ● Use types from generic modules to model required information o e.g. logging::LogFile(host=host, path=”/var/log/httpd/access_log”, type=”apache”) o Can be used to configure:  logstash  syslog  awstats  backup  logrotate  ...
  25. 25. Plug-ins ● Declarative configuration model ● Integration of imperative code with plug-ins ● Python 3 plug-ins: o typed parameters:  accept parameters from configuration model  navigate configuration model  read-only view o typed return value: results from imperative code ● Impera schedules plug-in invocation and supports backtracking
  26. 26. Plug-ins ● Model type system is meta-programmed into Python ● Each instance is represented by a Python object ● Access to configuration model by native code is mediated with proxy wrappers ● Template is a special type of plug-in with implicit arguments
  27. 27. Plug-in example demo/plugins/__init__.py: @plugin def get_ipaddress(host: “ip::Host”) -> “ip::address”: # lookup IP in a database return db.lookup_ip(host.name)
  28. 28. Plug-in example demo/plugins/__init__.py: @plugin def get_ipaddress(host: “ip::Host”) -> “ip::address”: # lookup IP in a database return db.lookup_ip(host.name) main.cf implementation lookupIP for ip::Host: self.ipaddres = demo::get_ipaddress(self)) end implement ip::Host using lookupIP server1 = ip::Host(name=”server-1”, os=fedora::fedora21)
  29. 29. Orchestration ● Incremental roll-out of configuration changes ● Prune incomplete parts of the model ● Plug-ins can indicate that a parameter is unknown ● Orchestration is derived automatically based on availability of configuration parameters
  30. 30. Plug-in example demo/plugins/__init__.py: @plugin def get_ipaddress(host: “ip::Host”) -> “ip::address”: # lookup IP in a database ip = db.lookup_ip(host.name) if ip is None: return Unknown(source=host) return ip
  31. 31. Orchestration examples ● Cloud o IP address of VM only available when VM is provisioned at cloud provider o Deploy service updates only when VM is available ● OpenStack deployment o Complex service dependencies o e.g. network service requires uuid allocated by identity service in its configuration files
  32. 32. Deployment architecture
  33. 33. Deploying changes ● two modes available ● client, server, agent o push and pull o fact renewal o enforcing dependencies over machine boundaries ● deploy o deploy directly, machine per machine o local or using ssh o runs embedded agent and server
  34. 34. Demo ● Deploy a Drupal website on Amazon ● Use varnish as loadbalancer for 2 web servers ● Add a web server ● Switch to ELB
  35. 35. Demo Configuration demo project: https://github.com/bartv/cfgmgmt15-demo
  36. 36. Demo lb-1 db-1 webhost-1 webhost-2
  37. 37. Demo ELB @ AWS db-1 webhost-1 webhost-2
  38. 38. Demo ELB @ AWS db-1 webhost-1 webhost-2 webhost-3
  39. 39. Status ● Started in 2010 ● 5 master students ● Large hybrid cloud demonstrator (OpenStack and Amazon) ● Manage our own OpenStack: o deployment o accounts, tenants, networks, routers, … ● Open source on Github ● Basic documentation + tutorial
  40. 40. Roadmap ● Integrate monitoring (management) ● State management and migration ● Increase compilation speed ● Improve documentation ● Automated testing for core and modules ● …
  41. 41. Contact Bart Vanbrabant bart@impera.io @bartvanbrabant https://distrinet.cs.kuleuven.be Impera tool: ● https://github.com/impera-io/impera ● modules https://github.com/impera-io/* ● http://impera.readthedocs.org/en/latest/ ● https://impera.io | @impera_io

×