Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Flowex: Flow-Based Programming with Elixir GenStage - Anton Mishchuk

647 views

Published on

Elixir Club 5

Published in: Software
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Flowex: Flow-Based Programming with Elixir GenStage - Anton Mishchuk

  1. 1. Flowex - Flow-Based Programming with Elixir GenStage Elixir Club 5, Kyiv, January 28, 2017
  2. 2. - Ruby developer at Matic Insurance - Elixir fan - Author and maintainer of ESpec BDD test framework github: antonmi Hello! I am Anton Mishchuk 2
  3. 3. The problem ◎ There are some problems in “conventional programing”: - code is procedural and sequential - parallelism is not native - hierarchical structure of program - visualisation is more about structure ◎ We need alternative approaches for building our programs to make software better! 3
  4. 4. My previous FBP talk 4 http://www.slideshare.net/AntonMishchuk /flowbased-programming-with-elixir
  5. 5. The goal of the talk ◎ Discuss a special case of Flow-Based Programming: “Railway FBP” (R-FBP) ◎ Present Flowex library which helps to create R-FBP abstractions ◎ Show that Elixir GenStage is not only about data-processing but about software design 5
  6. 6. What will be about ◎ Flow-Based Programming (FBP) ◎ Railway Oriented Programming (ROP) ◎ Railway FBP ◎ Flowex library Lots of pictures! 6
  7. 7. 1. Flow-Based Programming Basic concepts 7
  8. 8. “ FBP - a programming paradigm that defines applications as networks of "black box" processes, which exchange data across predefined connections by message passing J. Paul Morrison, Flow-Based Programming, 2nd Edition 8
  9. 9. FBP diagram A B D C IN 1 IN 1 IN 1 IN 2 IN 1 IN 2 OUT 1 OUT 2 OUT 1 OUT 1 OUT 1 9
  10. 10. Express a problem in terms of transforms on streams of data http://www.jpaulmorrison.com/fbp/FBPnew.ppt 10
  11. 11. Design pros ◎ Independent and reusable components ◎ Clean interfaces ◎ Simple to reconfigure ◎ Minimizes side-effects ◎ Designer can sit at one “station”, or can follow an item through system http://www.jpaulmorrison.com/fbp/FBPnew.ppt 11
  12. 12. “Native parallelism” 12
  13. 13. 2. Railway Oriented Programming Design pattern 13
  14. 14. Express a problem in terms of sequence of functions calls request |> validate_request |> get_user |> update_db_from_request |> send_email |> return_http_message 14
  15. 15. Define common interface for successful and error cases ◎ For example use: {:ok, "data"} or {:error, "Failure reason"} ◎ Each function returns {:ok, "data"} or {:error, "Failure reason"} ◎ If function is called with {:error, "Failure reason"} it just bypasses it 15
  16. 16. Define interface to bypass errors http://fsharpforfunandprofit.com/rop/ 16
  17. 17. So the program is a railway from input to output http://fsharpforfunandprofit.com/rop/ 17
  18. 18. Elixir Plug is a good example! ◎ Each “plug” function receives and returns a “connection structure” %Plug.Conn{} ◎ %Plug.Conn{} contains: assigns, cookies, halted, host, method, params, resp_body, resp_cookies, status, and many other attributes 18
  19. 19. 3. Railway FBP ROP + FBP 19
  20. 20. Let’s transform each part of “Railway” into FBP component! 20
  21. 21. Place each function into separate process validate send_emailupdate_db That is the idea! 21
  22. 22. Why Elixir? 22
  23. 23. GenStage ◎ Easiest way to implement a chain of communicating processes ◎ Back-pressure mechanism producer consumer Subscribe Ask Events 23
  24. 24. Metaprogramming ◎ Simple way for sharing functionality ◎ Ability to create expressive DSL 24
  25. 25. 4. Flowex Railway FBP 25
  26. 26. Consider an example! 26
  27. 27. Calculate (number + 1) * 2 - 3 27
  28. 28. Note, the functions are cool! ◎ They have the same interface ◎ They operate with predefined struct We can join them into pipeline: 28
  29. 29. Let’s add some Flowex magic! 29
  30. 30. … and rename the module to FunPipeline 30
  31. 31. “use Flowex.Pipeline” ◎ Adds ‘pipe’ macro which allows to mark functions “to be placed” into separate Genstage ◎ Defines ‘start’ and ‘stop’ functions for creating and destroying “Flowex pipelines” ◎ Defines ‘run’ function to perform calculations 31
  32. 32. Start FunPipeline! 32
  33. 33. FunPipeline instance add_one minus_threemult_by_twoproducer consumer Supervisor Elixir GenStages 33
  34. 34. %Flowex.Pipeline{} struct ◎ module - the name of the module ◎ in_name - unique name of 'producer' ◎ out_name - unique name of 'consumer' ◎ sup_pid - pid of the pipeline supervisor 34
  35. 35. Run calculations via “run” Using FunPipeline.run/2 function Note, ‘a’, ‘b’ and ‘c’ were set! 35
  36. 36. Run calculations using Flowex.Client Using Flowex.Client module 36
  37. 37. How it works in details add_one minus_three mult_by_two producer consumer Flowex.Client %FunPipeline{number: 2} %Flowex.IP{ struct: %FunPipeline{number: 2}, ...} number: 2 number: 2 number: 3 number: 6 number: 3 number: 3 self() %FunPipeline{number: 3} 37
  38. 38. What is sync and what is async? Client Asynchronous communicationself() Synchronous communication 38
  39. 39. Is there no parallelism? 39
  40. 40. There can be a lot of clients! Client Client Client 40
  41. 41. Bottlenecks. What if there is very slow process in the pipeline? 41
  42. 42. Just small changes in the code 42
  43. 43. And you get this add_one minus_three mult_by_twoproducer consumer mult_by_two minus_three mult_by_two 43
  44. 44. Reusable components with “module pipelines” 44
  45. 45. Module must implement just two functions Like Elixir Plug module 45
  46. 46. Pipeline module 46
  47. 47. Conclusion 47
  48. 48. Why it is cool! 48
  49. 49. It is easy to understand ◎ Pipelines explicitly define a structure of data will be processed ◎ Pipelines explicitly define a way the data will come ◎ Pipelines explicitly define a parallel executors structure 49
  50. 50. It is easy to maintain and reuse ◎ There is a predefined set of working processes (components) in a program ◎ Each component is isolated ◎ Pipelines can (and should) reuse components 50
  51. 51. It is about controlled parallelism ◎ One can controll number of clients supplying data to a pipeline ◎ One can control the number of processes available for each component 51
  52. 52. Thank you! Questions? 52 You can star the project here: https://github.com/antonmi/flowex

×