Downloaded 11 times
More Related Content
Similar to Flowex - Railway Flow-Based Programming with Elixir GenStage.
Flowex - Railway Flow-Based Programming with Elixir GenStage.
- 1. Flowex - Flow-Based Programmingwith Elixir GenStage Elixir Club 5, Kyiv, January 28, 2017
- 2. - Ruby developerat Matic Insurance - Elixir fan - Author and maintainer of ESpec BDD test framework github: antonmi Hello! I am Anton Mishchuk 2
- 3. The problem ◎ Thereare some problems in “conventional programing”: - code is procedural and sequential - parallelism is not native - hierarchical structure of program - visualisation is more about structure ◎ We need alternative approaches for building our programs to make software better! 3
- 4. My previous FBPtalk 4 http://www.slideshare.net/AntonMishchuk /flowbased-programming-with-elixir
- 5. The goal ofthe talk ◎ Discuss a special case of Flow-Based Programming: “Railway FBP” (R-FBP) ◎ Present Flowex library which helps to create R-FBP abstractions ◎ Show that Elixir GenStage is not only about data-processing but about software design 5
- 6. What will beabout ◎ Flow-Based Programming (FBP) ◎ Railway Oriented Programming (ROP) ◎ Railway FBP ◎ Flowex library Lots of pictures! 6
- 7.
- 8. “ FBP - aprogramming paradigm that defines applications as networks of "black box" processes, which exchange data across predefined connections by message passing J. Paul Morrison, Flow-Based Programming, 2nd Edition 8
- 9. FBP diagram A B D C IN 1 IN1 IN 1 IN 2 IN 1 IN 2 OUT 1 OUT 2 OUT 1 OUT 1 OUT 1 9
- 10. Express a problemin terms of transforms on streams of data http://www.jpaulmorrison.com/fbp/FBPnew.ppt 10
- 11. Design pros ◎ Independentand reusable components ◎ Clean interfaces ◎ Simple to reconfigure ◎ Minimizes side-effects ◎ Designer can sit at one “station”, or can follow an item through system http://www.jpaulmorrison.com/fbp/FBPnew.ppt 11
- 12.
- 13.
- 14. Express a problemin terms of sequence of functions calls request |> validate_request |> get_user |> update_db_from_request |> send_email |> return_http_message 14
- 15. Define common interfacefor successful and error cases ◎ For example use: {:ok, "data"} or {:error, "Failure reason"} ◎ Each function returns {:ok, "data"} or {:error, "Failure reason"} ◎ If function is called with {:error, "Failure reason"} it just bypasses it 15
- 16. Define interface tobypass errors http://fsharpforfunandprofit.com/rop/ 16
- 17. So the programis a railway from input to output http://fsharpforfunandprofit.com/rop/ 17
- 18. Elixir Plug isa good example! ◎ Each “plug” function receives and returns a “connection structure” %Plug.Conn{} ◎ %Plug.Conn{} contains: assigns, cookies, halted, host, method, params, resp_body, resp_cookies, status, and many other attributes 18
- 19. 3. Railway FBP ROP +FBP 19
- 20. Let’s transform eachpart of “Railway” into FBP component! 20
- 21. Place each functioninto separate process validate send_emailupdate_db That is the idea! 21
- 22.
- 23. GenStage ◎ Easiest wayto implement a chain of communicating processes ◎ Back-pressure mechanism producer consumer Subscribe Ask Events 23
- 24. Metaprogramming ◎ Simple wayfor sharing functionality ◎ Ability to create expressive DSL 24
- 25.
- 26.
- 27. Calculate (number +1) * 2 - 3 27
- 28. Note, the functionsare cool! ◎ They have the same interface ◎ They operate with predefined struct We can join them into pipeline: 28
- 29. Let’s add someFlowex magic! 29
- 30. … and renamethe module to FunPipeline 30
- 31. “use Flowex.Pipeline” ◎ Adds‘pipe’ macro which allows to mark functions “to be placed” into separate Genstage ◎ Defines ‘start’ and ‘stop’ functions for creating and destroying “Flowex pipelines” ◎ Defines ‘run’ function to perform calculations 31
- 32.
- 33. FunPipeline instance add_one minus_threemult_by_twoproducerconsumer Supervisor Elixir GenStages 33
- 34. %Flowex.Pipeline{} struct ◎ module- the name of the module ◎ in_name - unique name of 'producer' ◎ out_name - unique name of 'consumer' ◎ sup_pid - pid of the pipeline supervisor 34
- 35. Run calculations via“run” Using FunPipeline.run/2 function Note, ‘a’, ‘b’ and ‘c’ were set! 35
- 36. Run calculations usingFlowex.Client Using Flowex.Client module 36
- 37. How it worksin details add_one minus_three mult_by_two producer consumer Flowex.Client %FunPipeline{number: 2} %Flowex.IP{ struct: %FunPipeline{number: 2}, ...} number: 2 number: 2 number: 3 number: 6 number: 3 number: 3 self() %FunPipeline{number: 3} 37
- 38. What is syncand what is async? Client Asynchronous communicationself() Synchronous communication 38
- 39. Is there noparallelism? 39
- 40. There can bea lot of clients! Client Client Client 40
- 41. Bottlenecks. What if thereis very slow process in the pipeline? 41
- 42. Just small changesin the code 42
- 43. And you getthis add_one minus_three mult_by_twoproducer consumer mult_by_two minus_three mult_by_two 43
- 44. Reusable components with “modulepipelines” 44
- 45. Module must implementjust two functions Like Elixir Plug module 45
- 46.
- 47.
- 48. Why it iscool! 48
- 49. It is easyto understand ◎ Pipelines explicitly define a structure of data will be processed ◎ Pipelines explicitly define a way the data will come ◎ Pipelines explicitly define a parallel executors structure 49
- 50. It is easyto maintain and reuse ◎ There is a predefined set of working processes (components) in a program ◎ Each component is isolated ◎ Pipelines can (and should) reuse components 50
- 51. It is aboutcontrolled parallelism ◎ One can controll number of clients supplying data to a pipeline ◎ One can control the number of processes available for each component 51
- 52. Thank you! Questions? 52 Youcan star the project here: https://github.com/antonmi/flowex