Doit apac-2010-1.0

1,721 views

Published on

doit presentation
pycon - asia pacific 2010 singapore
(http://python-doit.sourceforge.net/)

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,721
On SlideShare
0
From Embeds
0
Number of Embeds
1,098
Actions
Shares
0
Downloads
5
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Doit apac-2010-1.0

  1. 1. doit – automation tool Bringing the power of build-tools to execute any kind of task Eduardo Schettino doit - automation tool 1
  2. 2. About me  Brazil  Beijing – China  Exoweb http://www.exoweb.net doit - automation tool 2
  3. 3. Talk overview  build-tools (make)  Who needs a build-tool?  doit  Questions & Answers doit - automation tool 3
  4. 4. build-tools Anything worth repeating is worth automating  manage repetitive tasks and their dependencies  speed-up development (faster turn-around)  1977: Make (C & other compiled languages) doit - automation tool 4
  5. 5. make model Rules Prerequisite 1 Prerequisite 2 Commands Targets ... Prerequisite N doit - automation tool 5
  6. 6. C project foo.c compile foo.o link foobar defs.h compile bar.o bar.c doit - automation tool 6
  7. 7. 1 - compile foo foo.c compile foo.o link foobar defs.h compile bar.o bar.c doit - automation tool 7
  8. 8. 2 - compile bar foo.c compile foo.o link foobar defs.h compile bar.o bar.c doit - automation tool 8
  9. 9. 3 - foobar foo.c compile foo.o link foobar defs.h compile bar.o bar.c doit - automation tool 9
  10. 10. initial build foo.c compile foo.o link foobar defs.h compile bar.o bar.c All 3 operations executed doit - automation tool 10
  11. 11. no-op rebuild foo.c compile foo.o link foobar defs.h compile bar.o bar.c No changes. No operation executed doit - automation tool 11
  12. 12. incremental rebuild foo.c compile foo.o link foobar defs.h compile bar.o bar.c foo.c changed. 2 operations executed doit - automation tool 12
  13. 13. how it works?  file time-stamp comparison  if any of the prerequisite files were modified after last modification on target => re-execute  if target is up-to-date => skip execution doit - automation tool 13
  14. 14. who needs a build-tool?  dynamic language => no compilation => static checkers + unit-tests  functional tests (DB + web) are slow  maintenance tasks doit - automation tool 14
  15. 15. python project foo.py unit-test main.py unit-test bar.py unit-test doit - automation tool 15
  16. 16. first run foo.py unit-test main.py unit-test bar.py unit-test doit - automation tool 16
  17. 17. no-op run??? foo.py unit-test main.py unit-test bar.py unit-test  no modifications on source code  all tests are executed again!  no targets to compare time-stamps... doit - automation tool 17
  18. 18. make problems  ad-hoc language (hard to use, debug, ...)  time-stamp based (fragile)  restricted to operations that create files  not good on dynamic creation of rules doit - automation tool 18
  19. 19. doit goals  more flexible model than traditional build-tools  “real” language => python  get out of your way doit - automation tool 19
  20. 20. doit model  based on “Tasks”, focus on “actions” Tasks dependency 1 dependency 2 Actions Targets ... dependency N doit - automation tool 20
  21. 21. how it works?  same principle but...  use a “db” file to save info from successful execution  tasks are defined on a “dodo” file, a plain python module  i.e. does not require targets to check if task is up-to-date doit - automation tool 21
  22. 22. Hello World def task_hello():     return {'actions':         ['echo Hello World > hello.txt']} doit - automation tool 22
  23. 23. Hello World A task generator ... def task_hello():     return {'actions':          ['echo Hello World > hello.txt']} doit - automation tool 23
  24. 24. Hello World A task generator ... def task_hello(): ... returns a dictionary with task meta-data     return {'actions':          ['echo Hello World > hello.txt']} doit - automation tool 24
  25. 25. Hello World A task generator ... def task_hello(): ... returns a dictionary with task meta-data     return {'actions':          ['echo Hello World > hello.txt']} strings are shell commands doit - automation tool 25
  26. 26. python actions a plain python function def pyhello():     with open('hello.txt', 'w') as hello_file:         hello_file.write("Hello Pythonn") def task_hello():     return {'actions': [(pyhello,)]} doit - automation tool 26
  27. 27. python actions a plain python function def pyhello():     with open('hello.txt', 'w') as hello_file:         hello_file.write("Hello Pythonn") tuple (callable, args, kwargs) def task_hello():     return {'actions': [(pyhello,)]} doit - automation tool 27
  28. 28. compile def task_compile():     return {'actions': ["cc ­c main.c"],             'file_dep': ["main.c","defs.h"],             'targets': ["main.o"]             } dependencies targets doit - automation tool 28
  29. 29. sub-tasks define task meta-data def task_create_file():     for i in range(3):         filename = "file%d.txt" % i         yield {'name': filename,                'actions':                   ["touch %s" % filename]} doit - automation tool 29
  30. 30. sub-tasks define task meta-data def task_create_file():     for i in range(3):         filename = "file%d.txt" % i         yield {'name': filename,                'actions':                   ["touch %s" % filename]} “yield” to create multiple tasks doit - automation tool 30
  31. 31. sub-tasks define task meta-data def task_create_file():     for i in range(3):         filename = "file%d.txt" % i required for sub-tasks         yield {'name': filename,                'actions':                   ["touch %s" % filename]} “yield” to create multiple tasks doit - automation tool 31
  32. 32. sub-tasks define task meta-data def task_create_file():     for i in range(3):         filename = "file%d.txt" % i required for sub-tasks         yield {'name': filename,                'actions':                   ["touch %s" % filename]} “yield” to create multiple tasks doit - automation tool 32
  33. 33. task dependency def task_foo():     return {'actions': ["echo foo"]} def task_bar():     return {'actions': ["echo bar"]} def task_mygroup():     return {'actions': None,             'task_dep': ['foo', 'bar']} tasks that must be executed before this task doit - automation tool 33
  34. 34. result dependency def task_which_version(): return {'actions':   ['bzr version­info ­­custom –template="{revno}n"']} def task_do_something(): return {'actions': ['echo "TODO: send an email"'],         'result_dep': ['which_version']} check result from another task instead of a file content doit - automation tool 34
  35. 35. result dependency def task_which_version(): return {'actions':   ['bzr version­info ­­custom –template="{revno}n"']} task “result” is string returned by action def task_do_something(): return {'actions': ['echo "TODO: send an email"'],         'result_dep': ['which_version']} check result from another task instead of a file content doit - automation tool 35
  36. 36. execution DOIT_CONFIG = {'default_tasks': ['t3']}  by default all tasks are executed  controlled by DOIT_CONFIG default_tasks $ doit .  task3 $ doit task2 task1:foo .  task2 .  task1:foo doit - automation tool 36
  37. 37. up-to-date ?  no file dependencies were modified  no result dependencies were modified  all targets exist  compares => timestamp, size, checksum doit - automation tool 37
  38. 38. up-to-date output $ doit . (dot) => executed .  compile $ doit -- (dashes) => skipped ­­ compile $ rm main.o $ doit .  compile doit - automation tool 38
  39. 39. environment setup def task_start_server():     for name in ('serverX', 'serverY'):         yield {'name': name,                'actions': [(start, (name,))],                'teardown': [(stop, (name,))],                } def task_test_A():     return {'actions':['echo fun_test_a'],             'setup': ['start_server:serverX'],            } test_A requires serverX to be running start serverX only if test_A not up-to-date doit - automation tool 39
  40. 40. environment setup def task_start_server():     for name in ('serverX', 'serverY'):         yield {'name': name,                'actions': [(start, (name,))],                'teardown': [(stop, (name,))],                } stop serverX after all tasks finish running def task_test_A():     return {'actions':['echo fun_test_a'],             'setup': ['start_server:serverX'],            } test_A requires serverX to be running start serverX only if test_A not up-to-date doit - automation tool 40
  41. 41. calculated dependency def task_mod_deps():     return {'actions': [(print_deps,)],             'calc_dep': ["get_dep"],            } dependencies are calculated on another task def get_dep(mod):     return {'file_dep': [‘a’, ‘b’, ‘c’]} def task_get_dep():     return {'actions':[(get_dep,)]} doit - automation tool 41
  42. 42. calculated dependency def task_mod_deps():     return {'actions': [(print_deps,)],             'calc_dep': ["get_dep"],            } dependencies are calculated on another task def get_dep(mod):     return {'file_dep': [‘a’, ‘b’, ‘c’]} def task_get_dep():     return {'actions':[(get_dep,)]} returns dictionary with same keys as task-dict (file_dep, task_dep, result_dep, calc_dep) doit - automation tool 42
  43. 43. other task features  clean => clean actions ($doit clean <tasks>)  doc => ($doit list)  params => get parameters from command line  getargs => get values computed in different tasks doit - automation tool 43
  44. 44. other runner features  verbosity => capture/display stdout/stderr  title => controls output (task name)  custom_reporters => complete output control  use wildcard to select tasks doit - automation tool 44
  45. 45. parallel execution  parallel execution of tasks in multiple processes  uses multiprocessing lib  subject to same limitations... doit - automation tool 45
  46. 46. auto execution  long running process  watches for file modifications and automatically re-execute outdated tasks  works on linux and mac  TDD, never leave the editor screen :) doit - automation tool 46
  47. 47. thanks  Questions ?  website: http://python-doit.sourceforge.net doit - automation tool 47

×