Your SlideShare is downloading. ×
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Teasing talk for Flow-based programming made easy with PyF 2.0
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Teasing talk for Flow-based programming made easy with PyF 2.0

1,923

Published on

Published in: Design, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,923
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
35
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. EuroPython 2011 PyF – A dataflow processing framework Dataflow processing, mining, transforming and  reporting using Python. Presented by : Jonathan Schemoul Software Consulting and Services
  • 2. What is dataflow programming ?● In the 70s: John Paul Morrison in Canada ● Bank applications● We process data one by one ● Items pass from one block to another, creating a flow ● We transform (adapt) items in the flow like in a production  chain. Software Consulting and Services
  • 3. What is dataflow programming ?In the theory :● All the blocks (components) make a network● Components have input and output● Network in managed by a scheduler Checks network state, manage processes● Data sent as information packets Software Consulting and Services
  • 4. Dataflow programming with PythonWe have generators >>> def my_generator(size=100) : ... for i in range(size) : ... print « I continue », i ... yield dict(iteration=i, ... answer=42)Code execution gets  >>> iterator = my_generator(size=5) >>> print repr(iterator.next())resumed at next item I continue 0 {iteration: 0, answer: 42} >>> for value in iterator : ... print repr(value)What if we tried to use  I continue 1 {iteration: 1, answer: 42}that for dataflow  I continue 2 {iteration: 2, answer: 42}programming ? I continue 3 {iteration: 3, answer: 42} I continue 4 {iteration: 4, answer: 42} Software Consulting and Services
  • 5. Chaining generators >>> def my_adapter(flux) :To use generators for  ... for v in flux: ... print « I received », vdataflow programming... ... yield dict(value=v, ... double=v*2) >>> it1 = xrange(5) # 0 to 5 >>> it2 = my_adapter(it1)We define the blocks , with  >>> def my_consumer(flux_adapted) : ... for nv in flux_adapted:generators ... ... print nv[double] yield True # Its ok. >>> it3 = my_consumer(it2) >>> for value in it3: ... if value is True:● We pass the iterator to  ... print ok so far another generator, making a  ... ... else: print not ok pipe I received 0 0 ok so far I received 1● We « adapt » the flow with  2 ok so far each block [...] Software Consulting and Services
  • 6. Just a few words● Blocks (components) in a process make a network● Each component has input and output ports● The network is managed using generators (lazy)● Transferred data can be either standard python objects  or « Information Packets ». Software Consulting and Services
  • 7. Benefits using lazy flow programming with PyF● RAM usage is kept low even with millions of items● Easy to learn programmation paradigm (you take one item, you give one item – or  several, or none, you choose! :­))● All python objects can be used (not only packets)● No threads● Just standard plain python (no stackless)● Need to scale ? Just ask for multiprocessing ! Software Consulting and Services
  • 8. PyF in a few words● Development framework and environment for dataflow  programming● Based on Zflow (the low­level core is a « friendly fork »)● Much more than the basis : « batteries included ». Software Consulting and Services
  • 9. PyF ArchitectureSoftware Consulting and Services
  • 10. PyF in detail● Plugins system ● A lot of existing plugins – Extraction (website, rss, sql alchemy...),  – Adaptation (summarizing, attribute getters, ...), – Writing (csv, xml, fixed­length, xlsx, pdf with xhtml, odt or rml templating,  etc.)● Python (and soon javascript and ruby) code can be inserted  directly in tube source● A lot of other tools (visual designer, scheduling, event logs...) Software Consulting and Services
  • 11. The Web ServiceSoftware Consulting and Services
  • 12. Creating a tubeSoftware Consulting and Services
  • 13. Using CodeSoftware Consulting and Services
  • 14. Or pluginsSoftware Consulting and Services
  • 15. Link and arrange your nodes...Software Consulting and Services
  • 16. Software Consulting and Services
  • 17. Simple... Software Consulting and Services
  • 18. ● Thank you !● Training is tomorrow at 14:30 in Pizza Napoli ● For more information : http://www.pyfproject.org/ #pyf on irc.freenode.net http://groups.google.com/group/pyf­users● Business Cards available Software Consulting and Services
  • 19. PyF Sponsors http://www.xcg-consulting.fr http://www.jmsinfor.comSoftware Consulting and Services
  • 20. Training Info● Tomorrow at 14:30 (2:30 PM) at Pizza Napoli● Please have : ● Under linux : libxml2­dev libxslt­dev ● Installed virtualenv (« easy_install ­UZ virtualenv »)● Recommended : ● Predownload pyf in a new virtualenv: – « easy_install ­UZ pyf[fullstack] » Software Consulting and Services

×