Your SlideShare is downloading. ×
0
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Dev8d 2011-pipe2 py
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Dev8d 2011-pipe2 py

838

Published on

An introduction to pipe2py a Yahoo Pipes to Python compiler.

An introduction to pipe2py a Yahoo Pipes to Python compiler.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
838
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Introducing Pipe2Py : Converting Yahoo Pipes to Python Code Original code: Greg Gaughan Additional development:  Tuukka Hastrup Based on an original idea by:   Tony Hirst , Dept of Communication and Systems, The Open University
  • 2. pipes.yahoo.com
  • 3. But what happens if Yahoo Pipes dies?
  • 4. Pipe2Py github.com/ggaughan/pipe2py
  • 5. Yahoo pipelines are translated into pipelines of Python generators* to give a close match to the original data flow. * based on ideas by David Beazley http://www.dabeaz.com/generators-uk
  • 6. Each Yahoo module is coded as a separate Python module.
  • 7. So you can use Yahoo Pipes as a graphical rapid prototyping application , and then generate a Python code equivalent you can host yourself So what?
  • 8. download code http://github.com/ggaughan/pipe2py to dev8d/pipes/pipe2py set path export PYTHONPATH=dev8d/pipes installation
  • 9. simplejson* sudo easy_install simplejson dependencies * only needed for Python pre 2.6
  • 10. test directory python testbasics.py unit tests
  • 11. python compile.py -p pipelineid compilation - direct from Yahoo Pipes generates pipe_ pipelineid .py
  • 12. python compile.py  pipelinefile.json compilation - from a file generates pipelinefile .py
  • 13. python  pipe_ pipelineid .py command line execution runs pipe_ pipelineid .py
  • 14. from pipe2py import Context from pipe2py.modules import * def pipe_404411a8d22104920f3fc1f428f33642(context, _INPUT, conf=None, **kwargs):     "Pipeline"     if conf is None:         conf = {}     forever = pipeforever.pipe_forever(context, None, conf=None)     sw_502 = pipefetch.pipe_fetch(context, forever, conf={u'URL': {u'type': u'url', u'value': u'http://blog.ouseful.info/feed'}})     _OUTPUT = pipeoutput.pipe_output(context, sw_502, conf={})     return _OUTPUT compiled code of the form...
  • 15. Each call to the final generator will ripple through the pipeline issuing .next() calls onto the previous generator until the source is exhausted.
  • 16. Each item is typically passed through the whole pipeline one at a time, so: memory usage is kept to a minimum no module is waiting on an earlier module to finish processing the whole data set by adding queues between the modules they could easily be made to run in parallel, each on a different CPU, to give great scalability
  • 17. from pipe2py import Context import pipe_9dc8014dcfd34c834a960321afde68d9 as p C=Context() r = p.pipe_9dc8014dcfd34c834a960321afde68d9(C,None) for i in r:    print i    print i['title'] usage - compiled pipe
  • 18. from pipe2py.compile import parse_and_build_pipe from pipe2py import Context pipe_def = """json representation of the pipe""" p = parse_and_build_pipe(Context(), pipe_def) for i in p:     print i usage - interpreted pipe
  • 19. context = Context(describe_input=True) p = pipe_ac45e9eb9b0174a4e53f23c4c9903c3f(context, None) need_inputs = p print need_inputs >>> [(u'0', u'username', u'Twitter username', u'text', u''), ...    (u'1', u'statustitle',  u'Status title [string] or [logo] means twitter icon', u'text', u'logo')] ''' That is, tuples of the form     (position, name, prompt, type, default) ''' usage - user inputs #1              Identifying console prompts
  • 20. C = Context(inputs={'username':'greg', 'statustitle':'logo'},                      console=False) p = pipe_ac45e9eb9b0174a4e53f23c4c9903c3f(C, None) for i in p:     print i usage - user inputs #2              avoiding console prompts
  • 21. Yahoo Pipes modules: Pipe2Py implementation progress
  • 22. Yahoo Pipes modules: Pipe2Py implementation progress
  • 23. Yahoo Pipes modules: Pipe2Py implementation progress
  • 24. ;-) One more thing...
  • 25. pipes-engine.appspot.com pipe2py hosting on Google App Engine
  • 26. - generate test pipes that work of increasing complexity - generate test pipes that don't work - commit pipe2py patches for test pipes that don't work How can you help?
  • 27. - simplify installation (easy_install?) - identify a good convention for integrating pipe2py compiled pipes in arbitrary code - - identify a good convention for inserting arbitrary python functions into, or in-between, compiled pipe2py pipelines How else can you help?
  • 28. the next step: produce an open source front end visual editor? wireit? pypes? Anything else?
  • 29. generate a ready-to-run instance of a Google App Engine configuration based around a compiled pipe? Anything more else?
  • 30. Pipe2Py github.com/ggaughan/pipe2py

×