Introducing  Pipe2Py : Converting Yahoo Pipes to Python Code Original code:  Greg Gaughan Additional development:  Tuukka ...
pipes.yahoo.com
But what happens if Yahoo Pipes dies?
Pipe2Py github.com/ggaughan/pipe2py
Yahoo pipelines are translated into pipelines of Python generators* to give a close match to the original data flow. * bas...
Each Yahoo module is coded as a separate Python module.
So you can use   Yahoo Pipes   as a  graphical rapid prototyping application , and then  generate a Python code equivalent...
download code http://github.com/ggaughan/pipe2py to  dev8d/pipes/pipe2py set path export PYTHONPATH=dev8d/pipes installation
simplejson* sudo easy_install simplejson dependencies * only needed for Python pre 2.6
test directory python testbasics.py unit tests
python compile.py -p  pipelineid compilation - direct from Yahoo Pipes generates pipe_ pipelineid .py
python compile.py  pipelinefile.json compilation - from a file generates pipelinefile .py
python  pipe_ pipelineid .py command line execution runs pipe_ pipelineid .py
from pipe2py import Context from pipe2py.modules import * def pipe_404411a8d22104920f3fc1f428f33642(context, _INPUT, conf=...
Each call to the final generator will ripple through the pipeline issuing  .next()  calls onto the previous generator unti...
Each item is typically passed through the whole pipeline one at a time, so: memory usage is kept to a minimum no module is...
from pipe2py import Context import pipe_9dc8014dcfd34c834a960321afde68d9 as p C=Context() r = p.pipe_9dc8014dcfd34c834a960...
from pipe2py.compile import parse_and_build_pipe from pipe2py import Context pipe_def = """json representat...
context = Context(describe_input=True) p = pipe_ac45e9eb9b0174a4e53f23c4c9903c3f(context, None) need_inputs = p print need...
C = Context(inputs={'username':'greg', 'statustitle':'logo'},                      console=False) p = pipe_ac45e9eb9b0174a...
Yahoo Pipes modules: Pipe2Py implementation progress
Yahoo Pipes modules: Pipe2Py implementation progress
Yahoo Pipes modules: Pipe2Py implementation progress
;-) One more thing...
pipes-engine.appspot.com pipe2py  hosting on Google App Engine
- generate test pipes  that work  of increasing complexity -  generate test pipes that  don't work - commit  pipe2py  patc...
- simplify installation (easy_install?) - identify a good convention for integrating  pipe2py  compiled pipes in arbitrary...
the next step: produce an open source front end visual editor? wireit? pypes? Anything else?
generate a ready-to-run instance of a Google App Engine configuration based around a compiled pipe? Anything more else?
Pipe2Py github.com/ggaughan/pipe2py
Upcoming SlideShare
Loading in...5
×

Dev8d 2011-pipe2 py

931
-1

Published on

An introduction to pipe2py a Yahoo Pipes to Python compiler.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
931
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Dev8d 2011-pipe2 py

  1. 1. Introducing Pipe2Py : Converting Yahoo Pipes to Python Code Original code: Greg Gaughan Additional development:  Tuukka Hastrup Based on an original idea by:   Tony Hirst , Dept of Communication and Systems, The Open University
  2. 2. pipes.yahoo.com
  3. 3. But what happens if Yahoo Pipes dies?
  4. 4. Pipe2Py github.com/ggaughan/pipe2py
  5. 5. Yahoo pipelines are translated into pipelines of Python generators* to give a close match to the original data flow. * based on ideas by David Beazley http://www.dabeaz.com/generators-uk
  6. 6. Each Yahoo module is coded as a separate Python module.
  7. 7. So you can use Yahoo Pipes as a graphical rapid prototyping application , and then generate a Python code equivalent you can host yourself So what?
  8. 8. download code http://github.com/ggaughan/pipe2py to dev8d/pipes/pipe2py set path export PYTHONPATH=dev8d/pipes installation
  9. 9. simplejson* sudo easy_install simplejson dependencies * only needed for Python pre 2.6
  10. 10. test directory python testbasics.py unit tests
  11. 11. python compile.py -p pipelineid compilation - direct from Yahoo Pipes generates pipe_ pipelineid .py
  12. 12. python compile.py  pipelinefile.json compilation - from a file generates pipelinefile .py
  13. 13. python  pipe_ pipelineid .py command line execution runs pipe_ pipelineid .py
  14. 14. from pipe2py import Context from pipe2py.modules import * def pipe_404411a8d22104920f3fc1f428f33642(context, _INPUT, conf=None, **kwargs):     "Pipeline"     if conf is None:         conf = {}     forever = pipeforever.pipe_forever(context, None, conf=None)     sw_502 = pipefetch.pipe_fetch(context, forever, conf={u'URL': {u'type': u'url', u'value': u'http://blog.ouseful.info/feed'}})     _OUTPUT = pipeoutput.pipe_output(context, sw_502, conf={})     return _OUTPUT compiled code of the form...
  15. 15. Each call to the final generator will ripple through the pipeline issuing .next() calls onto the previous generator until the source is exhausted.
  16. 16. Each item is typically passed through the whole pipeline one at a time, so: memory usage is kept to a minimum no module is waiting on an earlier module to finish processing the whole data set by adding queues between the modules they could easily be made to run in parallel, each on a different CPU, to give great scalability
  17. 17. from pipe2py import Context import pipe_9dc8014dcfd34c834a960321afde68d9 as p C=Context() r = p.pipe_9dc8014dcfd34c834a960321afde68d9(C,None) for i in r:    print i    print i['title'] usage - compiled pipe
  18. 18. from pipe2py.compile import parse_and_build_pipe from pipe2py import Context pipe_def = """json representation of the pipe""" p = parse_and_build_pipe(Context(), pipe_def) for i in p:     print i usage - interpreted pipe
  19. 19. context = Context(describe_input=True) p = pipe_ac45e9eb9b0174a4e53f23c4c9903c3f(context, None) need_inputs = p print need_inputs >>> [(u'0', u'username', u'Twitter username', u'text', u''), ...    (u'1', u'statustitle',  u'Status title [string] or [logo] means twitter icon', u'text', u'logo')] ''' That is, tuples of the form     (position, name, prompt, type, default) ''' usage - user inputs #1              Identifying console prompts
  20. 20. C = Context(inputs={'username':'greg', 'statustitle':'logo'},                      console=False) p = pipe_ac45e9eb9b0174a4e53f23c4c9903c3f(C, None) for i in p:     print i usage - user inputs #2              avoiding console prompts
  21. 21. Yahoo Pipes modules: Pipe2Py implementation progress
  22. 22. Yahoo Pipes modules: Pipe2Py implementation progress
  23. 23. Yahoo Pipes modules: Pipe2Py implementation progress
  24. 24. ;-) One more thing...
  25. 25. pipes-engine.appspot.com pipe2py hosting on Google App Engine
  26. 26. - generate test pipes that work of increasing complexity - generate test pipes that don't work - commit pipe2py patches for test pipes that don't work How can you help?
  27. 27. - simplify installation (easy_install?) - identify a good convention for integrating pipe2py compiled pipes in arbitrary code - - identify a good convention for inserting arbitrary python functions into, or in-between, compiled pipe2py pipelines How else can you help?
  28. 28. the next step: produce an open source front end visual editor? wireit? pypes? Anything else?
  29. 29. generate a ready-to-run instance of a Google App Engine configuration based around a compiled pipe? Anything more else?
  30. 30. Pipe2Py github.com/ggaughan/pipe2py
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×