paexec – distributes tasks over network or CPUs                 Aleksey Cheusov                  vle@gmx.net              ...
Problem     Huge amount of data to process     Typical desktop machines have more than one CPU/Core     Unlimited resource...
Solution                         Usage   paexec [OPTIONS]      -n ’machines of CPUs’      -t ’transport program’      -c ’...
Example 1: toupper   Our task is to convert strings to upper case                            ˜/bin/toupper   #!/usr/bin/aw...
Example 1: toupper                    paexec invocation   $ paexec -t ssh -c ~/bin/toupper       -n ’syn-proc7 syn-proc5’ ...
Example 1: toupper                   paexec -lr invocation   $ paexec -lr -t ssh -c ~/bin/toupper       -n ’syn-proc7 syn-...
Example 2: parallel banner(1)                     what is banner(1)?   $ banner   -f @ NetBSD   @     @                   ...
Example 2: parallel banner(1)                      ˜/bin/pbanner   #!/bin/sh   while read task; do       banner -f M "$tas...
Example 2: parallel banner(1)                                   Sliced result   $ cat results   2   2   2   M     M MMMMMM...
Example 2: parallel banner(1)                             Ordered result   $ paexec_reorder -S results    MMMMM       MM  ...
Example 3: dependency graph of tasks   paexec(1) is able to build tasks taking into account their   “dependencies”        ...
Example 3: dependency graph of tasks                        ˜/tmp/packages to build   audio/cd-discid audio/abcde   textpr...
Example 3: dependency graph of tasks                             ˜/bin/pkg builder   #!/usr/bin/awk -f   {       print "bu...
Example 3: dependency graph of tasks                     paexec -g invocation (no failures)   $ paexec -g -l -c ~/bin/pkg_...
Example 3: dependency graph of tasks                            ˜/bin/pkg builder   #!/usr/bin/awk -f   {       print "bui...
Example 3: dependency graph of tasks                    paexec -g invocation (with failures)   $ paexec -gl -c ~/bin/pkg_b...
Example 4: Resistance to network failures                            ˜/bin/pkg builder   #!/usr/bin/awk -f   {       "host...
Example 4: Resistance to network failures                    paexec -Z300 invocation (with failure)   $ paexec -gl -Z300 -...
The End
Upcoming SlideShare
Loading in...5
×

Paexec -- distributed tasks over network or cpus

314

Published on

Published in: Technology, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
314
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Paexec -- distributed tasks over network or cpus

  1. 1. paexec – distributes tasks over network or CPUs Aleksey Cheusov vle@gmx.net Minsk, Belarus, 2011
  2. 2. Problem Huge amount of data to process Typical desktop machines have more than one CPU/Core Unlimited resources are available through Internet Heterogeneous environment (*BSD, Linux, Windows...)
  3. 3. Solution Usage paexec [OPTIONS] -n ’machines of CPUs’ -t ’transport program’ -c ’calculator’ < tasks example ls *.wav | paexec -n +4 -c ~/bin/wav2flac example paexec -n ’host1 host2 host3’ -t /usr/bin/ssh -c ~/bin/toupper < tasks
  4. 4. Example 1: toupper Our task is to convert strings to upper case ˜/bin/toupper #!/usr/bin/awk -f { print " ", toupper($0) print "" # end-of-task marker! fflush() # We must flush stdout! } ˜/tmp/tasks apple bananas orange
  5. 5. Example 1: toupper paexec invocation $ paexec -t ssh -c ~/bin/toupper -n ’syn-proc7 syn-proc5’ < ~/tmp/tasks > results $ cat results BANANAS ORANGE APPLE $
  6. 6. Example 1: toupper paexec -lr invocation $ paexec -lr -t ssh -c ~/bin/toupper -n ’syn-proc7 syn-proc5’ < ~/tmp/tasks > results $ cat results syn-proc5 2 BANANAS syn-proc5 3 ORANGE syn-proc7 1 APPLE $
  7. 7. Example 2: parallel banner(1) what is banner(1)? $ banner -f @ NetBSD @ @ @@@@@@ @@@@@ @@@@@@ @@ @ @@@@@@ @@@@@ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @@@@@ @ @@@@@@ @@@@@ @ @ @ @ @ @ @ @ @ @ @ @ @ @@ @ @ @ @ @ @ @ @ @ @ @@@@@@ @ @@@@@@ @@@@@ @@@@@@ $
  8. 8. Example 2: parallel banner(1) ˜/bin/pbanner #!/bin/sh while read task; do banner -f M "$task" | sed ’s/ˆ/ /’ echo ’’ # end-of-task-marker done ˜/bin/tasks2 pae xec paexec invocation $ paexec -l -c ~/bin/pbanner -n +2 < ~/tmp/tasks2 > results $
  9. 9. Example 2: parallel banner(1) Sliced result $ cat results 2 2 2 M M MMMMMM MMMM 2 M M M M M 1 1 1 MMMMM MM MMMMMM 1 M M M M M 1 M M M M MMMMM 1 MMMMM MMMMMM M 2 MM MMMMM M 2 MM M M 2 M M M M M 2 M M MMMMMM MMMM 2 1 M M M M 1 M M M MMMMMM 1 $
  10. 10. Example 2: parallel banner(1) Ordered result $ paexec_reorder -S results MMMMM MM MMMMMM M M M M M M M M M MMMMM MMMMM MMMMMM M M M M M M M M MMMMMM M M MMMMMM MMMM M M M M M MM MMMMM M MM M M M M M M M M M MMMMMM MMMM $
  11. 11. Example 3: dependency graph of tasks paexec(1) is able to build tasks taking into account their “dependencies” lang/f2c devel/libtool-base devel/m4 devel/gmake audio/id3lib devel/bison audio/cd-discid textproc/gsed audio/cdparanoia misc/mkcue audio/id3v2 audio/id3 shells/bash audio/abcde
  12. 12. Example 3: dependency graph of tasks ˜/tmp/packages to build audio/cd-discid audio/abcde textproc/gsed audio/abcde audio/cdparanoia audio/abcde audio/id3v2 audio/abcde audio/id3 audio/abcde misc/mkcue audio/abcde shells/bash audio/abcde devel/libtool-base audio/cdparanoia devel/gmake audio/cdparanoia devel/libtool-base audio/id3lib devel/gmake audio/id3v2 audio/id3lib audio/id3v2 devel/m4 devel/bison lang/f2c devel/libtool-base devel/gmake misc/mkcue devel/bison shells/bash
  13. 13. Example 3: dependency graph of tasks ˜/bin/pkg builder #!/usr/bin/awk -f { print "build " $0 print ”success” # build succeeded! print "" # end-of-task marker fflush() # we must flush stdout }
  14. 14. Example 3: dependency graph of tasks paexec -g invocation (no failures) $ paexec -g -l -c ~/bin/pkg_builder -n ’syn-proc5 syn-proc7’ -t ssh < ~/tmp/packages_to_build | paexec_reorder > result $ cat result build textproc/gsed success build devel/gmake success build misc/mkcue success build devel/m4 success build devel/bison success ... build audio/id3v2 success build audio/abcde success $
  15. 15. Example 3: dependency graph of tasks ˜/bin/pkg builder #!/usr/bin/awk -f { print "build " $0 if ($0 == "devel/gmake") print ”failure” # Oh no... else print "success" # build succeeded! print "" # end-of-task marker fflush() # we must flush stdout }
  16. 16. Example 3: dependency graph of tasks paexec -g invocation (with failures) $ paexec -gl -c ~/bin/pkg_builder -n ’syn-proc5 syn-proc7’ -t ssh < ~/tmp/packages_to_build | paexec_reorder > result $ cat result build audio/cd-discid success build audio/id3 success build devel/gmake failure devel/gmake audio/cdparanoia audio/abcde audio/id3v2 misc/mkcue build devel/m4 success build textproc/gsed success ... $
  17. 17. Example 4: Resistance to network failures ˜/bin/pkg builder #!/usr/bin/awk -f { "hostname -s" | getline hostname print "build " $0 " on " hostname if (hostname == "syn-proc7" && $0 == "textproc/gsed") exit 0 # Damn it, I’m dying... else print "success" # Yes! :-) print "" # end-of-task marker fflush() # we must flush stdout }
  18. 18. Example 4: Resistance to network failures paexec -Z300 invocation (with failure) $ paexec -gl -Z300 -t ssh -c ~/bin/pkg_builder -n ’syn-proc5 syn-proc7’ < ~/tmp/packages_to_build | paexec_reorder > result $ cat result build audio/cd-discid on syn-proc5 success build textproc/gsed on syn-proc7 fatal build textproc/gsed on syn-proc5 success build audio/id3 on syn-proc5 success ... $
  19. 19. The End
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×