Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Doit apac-2010-1.0

2,088 views

Published on

doit presentation
pycon - asia pacific 2010 singapore
(http://python-doit.sourceforge.net/)

Published in: Technology
  • Be the first to comment

Doit apac-2010-1.0

  1. 1. doit – automation tool Bringing the power of build-tools to execute any kind of task Eduardo Schettino doit - automation tool 1
  2. 2. About me  Brazil  Beijing – China  Exoweb http://www.exoweb.net doit - automation tool 2
  3. 3. Talk overview  build-tools (make)  Who needs a build-tool?  doit  Questions & Answers doit - automation tool 3
  4. 4. build-tools Anything worth repeating is worth automating  manage repetitive tasks and their dependencies  speed-up development (faster turn-around)  1977: Make (C & other compiled languages) doit - automation tool 4
  5. 5. make model Rules Prerequisite 1 Prerequisite 2 Commands Targets ... Prerequisite N doit - automation tool 5
  6. 6. C project foo.c compile foo.o link foobar defs.h compile bar.o bar.c doit - automation tool 6
  7. 7. 1 - compile foo foo.c compile foo.o link foobar defs.h compile bar.o bar.c doit - automation tool 7
  8. 8. 2 - compile bar foo.c compile foo.o link foobar defs.h compile bar.o bar.c doit - automation tool 8
  9. 9. 3 - foobar foo.c compile foo.o link foobar defs.h compile bar.o bar.c doit - automation tool 9
  10. 10. initial build foo.c compile foo.o link foobar defs.h compile bar.o bar.c All 3 operations executed doit - automation tool 10
  11. 11. no-op rebuild foo.c compile foo.o link foobar defs.h compile bar.o bar.c No changes. No operation executed doit - automation tool 11
  12. 12. incremental rebuild foo.c compile foo.o link foobar defs.h compile bar.o bar.c foo.c changed. 2 operations executed doit - automation tool 12
  13. 13. how it works?  file time-stamp comparison  if any of the prerequisite files were modified after last modification on target => re-execute  if target is up-to-date => skip execution doit - automation tool 13
  14. 14. who needs a build-tool?  dynamic language => no compilation => static checkers + unit-tests  functional tests (DB + web) are slow  maintenance tasks doit - automation tool 14
  15. 15. python project foo.py unit-test main.py unit-test bar.py unit-test doit - automation tool 15
  16. 16. first run foo.py unit-test main.py unit-test bar.py unit-test doit - automation tool 16
  17. 17. no-op run??? foo.py unit-test main.py unit-test bar.py unit-test  no modifications on source code  all tests are executed again!  no targets to compare time-stamps... doit - automation tool 17
  18. 18. make problems  ad-hoc language (hard to use, debug, ...)  time-stamp based (fragile)  restricted to operations that create files  not good on dynamic creation of rules doit - automation tool 18
  19. 19. doit goals  more flexible model than traditional build-tools  “real” language => python  get out of your way doit - automation tool 19
  20. 20. doit model  based on “Tasks”, focus on “actions” Tasks dependency 1 dependency 2 Actions Targets ... dependency N doit - automation tool 20
  21. 21. how it works?  same principle but...  use a “db” file to save info from successful execution  tasks are defined on a “dodo” file, a plain python module  i.e. does not require targets to check if task is up-to-date doit - automation tool 21
  22. 22. Hello World def task_hello():     return {'actions':         ['echo Hello World > hello.txt']} doit - automation tool 22
  23. 23. Hello World A task generator ... def task_hello():     return {'actions':          ['echo Hello World > hello.txt']} doit - automation tool 23
  24. 24. Hello World A task generator ... def task_hello(): ... returns a dictionary with task meta-data     return {'actions':          ['echo Hello World > hello.txt']} doit - automation tool 24
  25. 25. Hello World A task generator ... def task_hello(): ... returns a dictionary with task meta-data     return {'actions':          ['echo Hello World > hello.txt']} strings are shell commands doit - automation tool 25
  26. 26. python actions a plain python function def pyhello():     with open('hello.txt', 'w') as hello_file:         hello_file.write("Hello Pythonn") def task_hello():     return {'actions': [(pyhello,)]} doit - automation tool 26
  27. 27. python actions a plain python function def pyhello():     with open('hello.txt', 'w') as hello_file:         hello_file.write("Hello Pythonn") tuple (callable, args, kwargs) def task_hello():     return {'actions': [(pyhello,)]} doit - automation tool 27
  28. 28. compile def task_compile():     return {'actions': ["cc ­c main.c"],             'file_dep': ["main.c","defs.h"],             'targets': ["main.o"]             } dependencies targets doit - automation tool 28
  29. 29. sub-tasks define task meta-data def task_create_file():     for i in range(3):         filename = "file%d.txt" % i         yield {'name': filename,                'actions':                   ["touch %s" % filename]} doit - automation tool 29
  30. 30. sub-tasks define task meta-data def task_create_file():     for i in range(3):         filename = "file%d.txt" % i         yield {'name': filename,                'actions':                   ["touch %s" % filename]} “yield” to create multiple tasks doit - automation tool 30
  31. 31. sub-tasks define task meta-data def task_create_file():     for i in range(3):         filename = "file%d.txt" % i required for sub-tasks         yield {'name': filename,                'actions':                   ["touch %s" % filename]} “yield” to create multiple tasks doit - automation tool 31
  32. 32. sub-tasks define task meta-data def task_create_file():     for i in range(3):         filename = "file%d.txt" % i required for sub-tasks         yield {'name': filename,                'actions':                   ["touch %s" % filename]} “yield” to create multiple tasks doit - automation tool 32
  33. 33. task dependency def task_foo():     return {'actions': ["echo foo"]} def task_bar():     return {'actions': ["echo bar"]} def task_mygroup():     return {'actions': None,             'task_dep': ['foo', 'bar']} tasks that must be executed before this task doit - automation tool 33
  34. 34. result dependency def task_which_version(): return {'actions':   ['bzr version­info ­­custom –template="{revno}n"']} def task_do_something(): return {'actions': ['echo "TODO: send an email"'],         'result_dep': ['which_version']} check result from another task instead of a file content doit - automation tool 34
  35. 35. result dependency def task_which_version(): return {'actions':   ['bzr version­info ­­custom –template="{revno}n"']} task “result” is string returned by action def task_do_something(): return {'actions': ['echo "TODO: send an email"'],         'result_dep': ['which_version']} check result from another task instead of a file content doit - automation tool 35
  36. 36. execution DOIT_CONFIG = {'default_tasks': ['t3']}  by default all tasks are executed  controlled by DOIT_CONFIG default_tasks $ doit .  task3 $ doit task2 task1:foo .  task2 .  task1:foo doit - automation tool 36
  37. 37. up-to-date ?  no file dependencies were modified  no result dependencies were modified  all targets exist  compares => timestamp, size, checksum doit - automation tool 37
  38. 38. up-to-date output $ doit . (dot) => executed .  compile $ doit -- (dashes) => skipped ­­ compile $ rm main.o $ doit .  compile doit - automation tool 38
  39. 39. environment setup def task_start_server():     for name in ('serverX', 'serverY'):         yield {'name': name,                'actions': [(start, (name,))],                'teardown': [(stop, (name,))],                } def task_test_A():     return {'actions':['echo fun_test_a'],             'setup': ['start_server:serverX'],            } test_A requires serverX to be running start serverX only if test_A not up-to-date doit - automation tool 39
  40. 40. environment setup def task_start_server():     for name in ('serverX', 'serverY'):         yield {'name': name,                'actions': [(start, (name,))],                'teardown': [(stop, (name,))],                } stop serverX after all tasks finish running def task_test_A():     return {'actions':['echo fun_test_a'],             'setup': ['start_server:serverX'],            } test_A requires serverX to be running start serverX only if test_A not up-to-date doit - automation tool 40
  41. 41. calculated dependency def task_mod_deps():     return {'actions': [(print_deps,)],             'calc_dep': ["get_dep"],            } dependencies are calculated on another task def get_dep(mod):     return {'file_dep': [‘a’, ‘b’, ‘c’]} def task_get_dep():     return {'actions':[(get_dep,)]} doit - automation tool 41
  42. 42. calculated dependency def task_mod_deps():     return {'actions': [(print_deps,)],             'calc_dep': ["get_dep"],            } dependencies are calculated on another task def get_dep(mod):     return {'file_dep': [‘a’, ‘b’, ‘c’]} def task_get_dep():     return {'actions':[(get_dep,)]} returns dictionary with same keys as task-dict (file_dep, task_dep, result_dep, calc_dep) doit - automation tool 42
  43. 43. other task features  clean => clean actions ($doit clean <tasks>)  doc => ($doit list)  params => get parameters from command line  getargs => get values computed in different tasks doit - automation tool 43
  44. 44. other runner features  verbosity => capture/display stdout/stderr  title => controls output (task name)  custom_reporters => complete output control  use wildcard to select tasks doit - automation tool 44
  45. 45. parallel execution  parallel execution of tasks in multiple processes  uses multiprocessing lib  subject to same limitations... doit - automation tool 45
  46. 46. auto execution  long running process  watches for file modifications and automatically re-execute outdated tasks  works on linux and mac  TDD, never leave the editor screen :) doit - automation tool 46
  47. 47. thanks  Questions ?  website: http://python-doit.sourceforge.net doit - automation tool 47

×