Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Flow-based programming with Elixir

1,203 views

Published on

Flow-based programming with Elixir

Published in: Engineering
  • Be the first to comment

Flow-based programming with Elixir

  1. 1. Flow-based programming with Elixir 3rd Kyiv Elixir meetup, October 1, 2016
  2. 2. Hello! I am Anton Mishchuk Ruby developer at Matic Elixir fan ESpec’s author and maintainer Github: antonmi 2
  3. 3. Introduction ◈ Elixir new awesome feature - GenStage ◈ Elixir Flow-Based Programming (Peter C Marks) ◈ Flow-Based Programming, 2nd Edition (J. Paul Morrison) 3
  4. 4. What I’d tell you ◈ “Conventional programming” ◈ Flow-Based Programming ◈ “Telegram problem” example ◈ Elixir GenStage ◈ “Telegram problem” redesign ◈ Data-flow routing with GenStage 4
  5. 5. 1. Software design in conventional programming 5 Warning! Too general philosophy!
  6. 6. Micro level 6 ◈ Basic components: ○ Objects in OOP comprising both information and behaviour ○ Functions and data-structures in FP ◈ Hierarchical organization ◈ Sequential communication or evaluation
  7. 7. DSLs and Frameworks as “coordination languages” ◈ Define common abstractions ◈ Initialize your components ◈ Use components in right order 7
  8. 8. Visualization 8
  9. 9. Concurrency & parallelism 9
  10. 10. ◈ Hierarchic structure of program ◈ Code is procedural and sequential ◈ Visualisation is more about structure ◈ Concurrency is not native What is wrong with conventional programming? 10
  11. 11. Basic concepts 2. Flow-Based Programming 11
  12. 12. “FBP - a programming paradigm that defines applications as networks of "black box" processes, which exchange data across predefined connections by message passing, where the connections are specified externally to the processes. These black box processes can be reconnected endlessly to form different applications without having to be changed internally. https://en.wikipedia.org/wiki/Flow-based_programming 12
  13. 13. FBP diagram A B D C IN 1 IN 1 IN 1 IN 2 IN 1 IN 2 OUT 1 OUT 2 OUT 1 OUT 1 OUT 1 Processes, ports, connections 13
  14. 14. “... whereas the conventional approaches to programming start with process and view data as secondary, business applications are usually designed starting with data and viewing process as secondary – processes are just the way data is created, manipulated and destroyed. J. Paul Morrison, Flow-Based Programming, 2nd Edition 14
  15. 15. Express a problem in terms of transforms on streams of data 15
  16. 16. Soft Drink Bottling Factory ◈ Independent well-defined components ◈ Clean interfaces ◈ Simple to reconfigure ◈ Minimizes side-effects ◈ Designer can sit at one “station”, or can follow an item through system http://www.jpaulmorrison.com/fbp/FBPnew.ppt 16
  17. 17. FPB characteristics ◈ Asynchronous processes communicating via streams of data packets ◈ Data packets with a lifetime of their own ◈ Definition of connections external to components ◈ Consistent view from macro to micro http://www.jpaulmorrison.com/fbp/FBPnew.ppt 17
  18. 18. Native parallelism 18
  19. 19. FBP is about “coordination language” 19
  20. 20. Everything new is actually well-forgotten old ◈ 1971, IBM, basic concepts ◈ 1975, Bank of Montreal, on-line banking system, still works! ◈ 1994, Flow-Based Programming, 1st Ed ◈ 2010, Flow-Based Programming, 2st Ed ◈ 2012, NoFlo project by Henri Bergius, FBP in JavaScript ◈ JavaFBP, C#FBP, CppFBP (C++ and Lua) 20
  21. 21. What about Erlang/Elixir? 21
  22. 22. FBP example with Elixir 3. Telegram problem 22
  23. 23. Telegram problem A program which accepts lines of text and generates output lines of a different predefined length, without splitting any of the words in the text. The program accepts an input stream of lines of some length and produce an output stream of lines of another length. 23
  24. 24. Problem FBP design using GenServers ReadSeq Recompose WriteSeqDecompose Lines Words Lines ◈ Each component is a GenServer. ◈ Each component implements only one function - “run” which transforms input to output ◈ Communication is asynchronous ◈ Code is here: https://github.com/antonmi/kyiv_meetup_3 24
  25. 25. ReadSeq (naive implementation) 25
  26. 26. Decompose (naive implementation) 26
  27. 27. Recompose (naive implementation) 27
  28. 28. WriteSeq (naive implementation) 28
  29. 29. The main Telegram module (naive implementation) 29
  30. 30. What is cool in this simple implementation ◈ Each component do its own part of work ◈ The only API is “init” and “run” ◈ Components works in parallel ◈ Components are under supervisor 30
  31. 31. What is wrong with the implementation ◈ Components are not independent (each of them knows who is the next) ◈ Flow is controlled internally by each of the component ◈ There is no back-pressure mechanism (when some component lags it can be overflowed by data) 31
  32. 32. Experimental feature. 4. Elixir GenStage 32
  33. 33. GenStage abstractions ◈ GenStage is build on top of GenServer ◈ Each of the stage can be producer, consumer or both producer producer consumer consumer producer consumer 33
  34. 34. GenStage communication ◈ Consumer subscribes to producer ◈ Consumer asks for data (sends a demand) ◈ Producer sends data producer consumer Subscribe Ask Events 34
  35. 35. GenStage API ◈ init(state) which must return: ○ {:producer, state} ○ {:producer_consumer, state} ○ {:consumer, state} ◈ handle_demand(events, from, state) ○ {:noreply, events, new_state} ◈ handle_events(events, from, state) ○ {:noreply, events, new_state} 35
  36. 36. 5. Telegram problem redesign 36
  37. 37. ReadSeq GenStage 37
  38. 38. Decompose GenStage 38
  39. 39. Recompose GenStage 39
  40. 40. WriteSeq GenStage 40
  41. 41. Main Telegram module 41
  42. 42. It is awesome! ◈ Each component is independent! ◈ Data flow is coordinated externally! ◈ There is a back-pressure! 42
  43. 43. 5. Data-flow routing with GenStage 43
  44. 44. GenStage Dispatchers ◈ GenStage.DemandDispatcher ○ sends events to the highest demand ◈ GenStage.BroadcastDispatcher ○ accumulates demand from all consumers before broadcasting events to all of them ◈ GenStage.PartitionDispatcher ○ sends events according to partitions 44
  45. 45. Duplicate file example 45 ReadSeq WriteSeq Broadcast WriteSeq Code is here: https://github.com/antonmi/kyiv_meetup_3
  46. 46. The only new component is Broadcast 46
  47. 47. Split file example 47 Code is here: https://github.com/antonmi/kyiv_meetup_3 ReadSeq WriteSeq Split WriteSeq
  48. 48. The only new component is Split 48
  49. 49. 6. Conclusion 49
  50. 50. Erlang VM + Elixir GenStage = FBP paradigm 50
  51. 51. Thanks!

×