Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

batou - multi(component|host|environment|.*) deployment

1,387 views

Published on

Talk given at EuroPython 2013

Published in: Technology, Sports
  • Be the first to comment

batou - multi(component|host|environment|.*) deployment

  1. 1. BATOU multi-(component¦host¦environment¦.*) deployment Wednesday, 3.July 13
  2. 2. @theuni Wednesday, 3.July 13
  3. 3. Wednesday, 3.July 13
  4. 4. Wednesday, 3.July 13
  5. 5. Wednesday, 3.July 13
  6. 6. AUTOMATING DEPLOYMENTS IS HARD Wednesday, 3.July 13
  7. 7. HOW DOES CONVERGENCE HELP? Wednesday, 3.July 13
  8. 8. HOW DOES THIS WORK WITH BATOU? Wednesday, 3.July 13
  9. 9. SOME PERSPECTIVE Wednesday, 3.July 13
  10. 10. Wednesday, 3.July 13
  11. 11. IT'S NOT THAT BAD. Wednesday, 3.July 13
  12. 12. Wednesday, 3.July 13
  13. 13. service deployment Fabric, Capistrano, ... system configuration Puppet, Chef, ... provisioning kickstart, Razor, imaging ... Wednesday, 3.July 13
  14. 14. FTP bash mkzopeinstance zc.buildout fabric Wednesday, 3.July 13
  15. 15. CONVERGENCE Wednesday, 3.July 13
  16. 16. "Everything that follows is a result of what you see here." (Dr. Alfred Lanning; I, Robot) Wednesday, 3.July 13
  17. 17. SIMPLE os.mkdir('foo') with open('foo/bar', 'w') as myfile: myfile.write('asdf') os.chmod('foo/bar', 0755) Wednesday, 3.July 13
  18. 18. •unexpected system state •can't resume •unnecessary updates os.mkdir('foo') with open('foo/bar', 'w') as myfile: myfile.write('asdf') os.chmod('foo/bar', 0755) SIMPLISTIC Wednesday, 3.July 13
  19. 19. CORRECT(?) if not os.path.isdir('foo'): os.unlink('foo') if not os.path.exists('foo'): os.mkdir('foo') try: os.lstat('foo/bar') except OSError: pass else: if os.path.isdir('foo/bar'): shutil.rmtree('foo/bar') else: os.unlink('foo/bar') if (os.path.exists('foo/bar') and open('foo/bar', 'r').read() != 'asdf'): open('foo/bar', 'w').write('asdf'): current = os.stat('foo/bar').st_mode if stat.S_IMODE(current) != 0755: os.chmod('foo', 0755) Wednesday, 3.July 13
  20. 20. SIMPLE File('foo/bar', content='asdf', mode=0755, leading=True) Wednesday, 3.July 13
  21. 21. class File(Component): namevar = 'path' def configure(self): self += Presence( self.path, leading=self.leading) self += Mode(self.path, self.mode) self += Content(self.path, self.content) File('foo/bar', content='asdf', mode=0755, leading=True) Wednesday, 3.July 13
  22. 22. class File(Component): namevar = 'path' def configure(self): self += Presence( self.path, leading=self.leading) self += Mode(self.path, self.mode) self += Content(self.path, self.content) File('foo/bar', content='asdf', mode=0755, leading=True) compute target state (no touching!) Wednesday, 3.July 13
  23. 23. class File(Component): namevar = 'path' def configure(self): self += Presence( self.path, leading=self.leading) self += Mode(self.path, self.mode) self += Content(self.path, self.content) File('foo/bar', content='asdf', mode=0755, leading=True) compute target state (no touching!) composition operator Wednesday, 3.July 13
  24. 24. class File(Component): namevar = 'path' def configure(self): self += Presence( self.path, leading=self.leading) self += Mode(self.path, self.mode) self += Content(self.path, self.content) File('foo/bar', content='asdf', mode=0755, leading=True) compute target state (no touching!) composition operator order matters Wednesday, 3.July 13
  25. 25. class Presence(Component): namevar = 'path' leading = False def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading) def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass Wednesday, 3.July 13
  26. 26. class Presence(Component): namevar = 'path' leading = False def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading) def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass run "anywhere" Wednesday, 3.July 13
  27. 27. class Presence(Component): namevar = 'path' leading = False def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading) def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass run on target run "anywhere" Wednesday, 3.July 13
  28. 28. class Presence(Component): namevar = 'path' leading = False def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading) def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass run on target run "anywhere" after all sub- components Wednesday, 3.July 13
  29. 29. class Presence(Component): namevar = 'path' leading = False def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading) def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass run on target only if needed run "anywhere" after all sub- components Wednesday, 3.July 13
  30. 30. class Presence(Component): namevar = 'path' leading = False def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading) def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass run on target only if needed run "anywhere" after all sub- components keep delegating! Wednesday, 3.July 13
  31. 31. class Directory(Component): namevar = 'path' leading = False def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path) Wednesday, 3.July 13
  32. 32. class Directory(Component): namevar = 'path' leading = False def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path) could be done with recursive composition Wednesday, 3.July 13
  33. 33. class Directory(Component): namevar = 'path' leading = False def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path) could be done with recursive composition refactor with sub-components if too complex Wednesday, 3.July 13
  34. 34. class Directory(Component): namevar = 'path' leading = False def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path) all methods optional: no configure() could be done with recursive composition refactor with sub-components if too complex Wednesday, 3.July 13
  35. 35. class Directory(Component): namevar = 'path' leading = False def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded() def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path) all methods optional: no configure() could be done with recursive composition pattern: just wipe out what's wrong refactor with sub-components if too complex Wednesday, 3.July 13
  36. 36. CONVERGENCE resume where needed handle many system states transparently avoid unnecessary updates Wednesday, 3.July 13
  37. 37. COMPONENTS composition of simple components no magic bullet, just a lot easier to factor your code configure - verify - update Wednesday, 3.July 13
  38. 38. Wednesday, 3.July 13
  39. 39. SINGLE- COMMAND Wednesday, 3.July 13
  40. 40. REPEATABLE RELIABLE Wednesday, 3.July 13
  41. 41. SIMPLE Wednesday, 3.July 13
  42. 42. ENTROPY Wednesday, 3.July 13
  43. 43. EXPRESSIVENESS READABILITY Wednesday, 3.July 13
  44. 44. REUSABLE Wednesday, 3.July 13
  45. 45. PLATFORM INDEPENDENCE Wednesday, 3.July 13
  46. 46. DOMAIN AGNOSTIC Wednesday, 3.July 13
  47. 47. NO ADDITIONAL RUNTIME DEPENDENCIES Wednesday, 3.July 13
  48. 48. CONTINUITY Wednesday, 3.July 13
  49. 49. MINIMAL DOWNTIMES Wednesday, 3.July 13
  50. 50. Wednesday, 3.July 13
  51. 51. PRACTICAL USAGE Wednesday, 3.July 13
  52. 52. REQUIREMENTS Python 2.7 SSH virtualenv Mercurial Wednesday, 3.July 13
  53. 53. ENVIRONMENTS [environment] service_user = myservice host_domain = flyingcircus.io branch = production [hosts] multikarl00 = nginx, haproxy multikarl01 = postgres, redis, memcached, crontab multikarl12 = supervisor, logrotate, doctotext, myapp multikarl13 = supervisor, logrotate, doctotext, myapp Wednesday, 3.July 13
  54. 54. LOCAL $ bin/batou-local dev localhost Updating Hello > File(hello) > Presence(hello) Updating Hello > File(hello) > Content(hello) $ bin/batou-local dev localhost $ Wednesday, 3.July 13
  55. 55. REMOTE $ bin/batou-remote prod test02.gocept.net: connecting test01.gocept.net: connecting test01.gocept.net: bootstrapping test02.gocept.net: bootstrapping OK OK Deploying test01.gocept.net/hello Updating Hello > File(hello) > Presence(hello) Updating Hello > File(hello) > Content(hello) OK Deploying test02.gocept.net/hello Updating Hello > File(hello) > Presence(hello) Updating Hello > File(hello) > Content(hello) OK Wednesday, 3.July 13
  56. 56. OVERRIDES class Hello(Component): hostname = "foo" [environment] ... [component:hello] hostname = bar Wednesday, 3.July 13
  57. 57. SECRETS class Hello(Component): db_password = none secrets/production.cfg [hello] db_password = reallysecretstuff Wednesday, 3.July 13
  58. 58. SECRETS class Hello(Component): db_password = none secrets/production.cfg [hello] db_password = reallysecretstuff SciFi but close Wednesday, 3.July 13
  59. 59. PROVIDE/REQUIRE class MyApp(Component): def configure(self): self.provide('appserver', self.host.fqdn) class HAProxy(Component): def configure(self): self.backends = self.require('appserver') Wednesday, 3.July 13
  60. 60. PLATFORMS class HAProxy(Component): ... @platform('flyingcircus.io', HAProxy) class SystemWideHAProxy(Component): def configure(self): self += File('/etc/haproxy', ensure='symlink', link_to=self.parent.haproxy_cfg.path) Wednesday, 3.July 13
  61. 61. VFS MAPPING ./ ... ./work ./work/_/etc/haproxy.cfg class HAProxy(Component): def configure(self): self += File('/etc/haproxy') [environment] ... [vfs] sandbox = Developer Wednesday, 3.July 13
  62. 62. FEATURES class MyApp(Component): features = ['instance', 'jobrunner'] def configure(self): if 'instance' in self.features: ... [hosts] hosta = myapp:instance hostb = myapp:jobrunner hostc = myapp:instance, myapp:jobrunner hostd = myapp Wednesday, 3.July 13
  63. 63. Wednesday, 3.July 13
  64. 64. CONVERGENCE COMPOSITION DETAILS Wednesday, 3.July 13
  65. 65. QUESTIONS? Wednesday, 3.July 13
  66. 66. batou.readthedocs.org pypi.python.org/pypi/batou bitbucket.org/gocept/batou Wednesday, 3.July 13

×