Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Multi-language
FBP with Flowex
Elixir Club Ternopil, March 25, 2017
Hello!
I am Anton Mishchuk
▷ Ruby developer at Matic Insurance Services
(we are hiring!)
▷ A big fan of Elixir programming...
Preface
Flow-based programming with Elixir
3
Preface
Flowex - FBP with Elixir GenStage
4
What’s next?
▷ Let Elixir do what it does best
▷ The rest can be implemented in
any other programming language
5
Agenda
▷ Quick intro to FBP
▷ Flowex library overview
▷ Tools for running external programs
▷ Multi-language components ex...
1.
Flow-Based Programming
Basic concepts
7
Flow-Based Programming is
a programming paradigm which
models application as a graph of
independent processes exchanging
d...
FBP diagram
A
B
D
C
IN 1
IN 1
IN 1
IN 2
IN 1
IN 2
OUT 1
OUT 2
OUT 1
OUT 1
OUT 1
9
FBP advantages
▷ Parallelism are natural
▷ No problems associated with shared
memory and locks
▷ Visualisation of data flo...
Two layers in FBP application
▷ “Bottom layer” - set of components
implementing parts of business
logic
▷ “Top layer” - “c...
“Bottom layer” - components
▷ Component is independent unit
with input and output
▷ The only thing it does - transform
inp...
“Top layer” - communication
▷ Need “coordination language”
▷ Actor model of Elixir/Erlang
▷ GenStage
13
GenStage
▷ behavior for exchanging events with
back-pressure between Elixir processes
▷ José Valim defined GenStage as “be...
2.
Flowex
Railway Flow-Based Programming
15
Flowex
▷ a set of abstractions built on top Elixir
GenStage which allows writing
program with FBP paradigm
▷ a mix of FBP ...
Railway oriented programming
17
Railway FBP
Let’s place each function inside
separate process
18
use Flowex.Pipeline
19
Functions’ definitions
20
Control and evaluation
21
Some new functions appear in the
FunPipeline module:
▷ “start” and “stop”
▷ “call” and “cast”
FunPipeline.start
22
“call” and “cast” (sync and async)
23
FunPipeline.call
24
add_one
minus_three
mult_by_two
producer
consumer
number: 2
number: 2
number: 3
number: 6
number: 3num...
“error_pipe”
If an error happens in some pipe, functions
in next pipes will not be called and IP will be
bypassed to the “...
Run pipeline via
Flowex.Client
▷ Use Clients for
parallel processing of
many IPs
▷ Allows effective using
of pipeline even...
Fight bottlenecks
27
“Controlled parallelism”
28
Reusable components with
module pipes
29
One should implement “init” and
“call” functions
(like in Elixir’s Plug library)
Advantages of Flowex approach
30
▷ Explicit definition of:
- structure of data will be processed
- way the data will come
...
3.
Running external programs
Tools
31
Why do we need this?
▷ Elixir/Erlang ecosystem is not so big as
Java, Ruby, Python, etc.
▷ Elixir/Erlang is not the fastes...
Erlang ports
▷ provide a byte-oriented interface to an
external program over STDIN and STDOUT
▷ are process-specific resou...
ErlPort project
▷ helps connect Erlang to other
programming languages
▷ currently supported external languages
are Python ...
Porcelain project
▷ helps launching and communicating with
external OS processes from Elixir
▷ launches external programs ...
4.
Multi-language components
Example project
36
https://github.com/antonmi/multi_flowex
MultiFlowex pipeline
▷ There will be Ruby, Python and shell “pipes”
▷ Each will say “Hello”
▷ The result will be:
“Hello f...
multi_flowex dependencies
38
Ruby component
Demonstrates how one can create
components using Ruby, Python or Shell
Read more on ….
39
Ruby code in main.rb file
40
Ruby component specs
41
Python component
Demonstrates how one can create
components using Ruby, Python or Shell
Read more on ….
42
Python code in main.py file
43
Shell component
44
The last, Elixir component
45
MultiPipeline
46
Pipeline starts external processes
47
▷ 3 Ruby processes
▷ 4 Python processes
MultiPipeline specs
48
Conclusion
49
▷ FBP paradigm is an advantageous
alternative to “conventional programming”
▷ Flowex makes it easy to conver...
Thanks!
Any questions?
50
Upcoming SlideShare
Loading in …5
×

Anton Mishchuk - Multi-language FBP with Flowex

114 views

Published on

Elixir Club Ternopil - March 25, 2017
Friendly Restoration "Mamont"

Published in: Technology
  • Be the first to comment

Anton Mishchuk - Multi-language FBP with Flowex

  1. 1. Multi-language FBP with Flowex Elixir Club Ternopil, March 25, 2017
  2. 2. Hello! I am Anton Mishchuk ▷ Ruby developer at Matic Insurance Services (we are hiring!) ▷ A big fan of Elixir programming language ▷ Author and maintainer of: ESpec and Flowex libraries github: antonmi 2
  3. 3. Preface Flow-based programming with Elixir 3
  4. 4. Preface Flowex - FBP with Elixir GenStage 4
  5. 5. What’s next? ▷ Let Elixir do what it does best ▷ The rest can be implemented in any other programming language 5
  6. 6. Agenda ▷ Quick intro to FBP ▷ Flowex library overview ▷ Tools for running external programs ▷ Multi-language components example: Ruby, Python, shell 6
  7. 7. 1. Flow-Based Programming Basic concepts 7
  8. 8. Flow-Based Programming is a programming paradigm which models application as a graph of independent processes exchanging data across predefined connections 8
  9. 9. FBP diagram A B D C IN 1 IN 1 IN 1 IN 2 IN 1 IN 2 OUT 1 OUT 2 OUT 1 OUT 1 OUT 1 9
  10. 10. FBP advantages ▷ Parallelism are natural ▷ No problems associated with shared memory and locks ▷ Visualisation of data flow networks is simple ▷ Components are independent and reusable 10
  11. 11. Two layers in FBP application ▷ “Bottom layer” - set of components implementing parts of business logic ▷ “Top layer” - “communication logic”  -  an organization of data flows from one component to another 11
  12. 12. “Bottom layer” - components ▷ Component is independent unit with input and output ▷ The only thing it does - transform input IP to output IP ▷ In general, can be implemented using any programming language 12
  13. 13. “Top layer” - communication ▷ Need “coordination language” ▷ Actor model of Elixir/Erlang ▷ GenStage 13
  14. 14. GenStage ▷ behavior for exchanging events with back-pressure between Elixir processes ▷ José Valim defined GenStage as “better abstractions for working with collections” ▷ but we can use it for “better interprocess communication” 14
  15. 15. 2. Flowex Railway Flow-Based Programming 15
  16. 16. Flowex ▷ a set of abstractions built on top Elixir GenStage which allows writing program with FBP paradigm ▷ a mix of FBP and Railway Oriented Programming (ROP) approach 16
  17. 17. Railway oriented programming 17
  18. 18. Railway FBP Let’s place each function inside separate process 18
  19. 19. use Flowex.Pipeline 19
  20. 20. Functions’ definitions 20
  21. 21. Control and evaluation 21 Some new functions appear in the FunPipeline module: ▷ “start” and “stop” ▷ “call” and “cast”
  22. 22. FunPipeline.start 22
  23. 23. “call” and “cast” (sync and async) 23
  24. 24. FunPipeline.call 24 add_one minus_three mult_by_two producer consumer number: 2 number: 2 number: 3 number: 6 number: 3number: 3 self()
  25. 25. “error_pipe” If an error happens in some pipe, functions in next pipes will not be called and IP will be bypassed to the “error_pipe” 25
  26. 26. Run pipeline via Flowex.Client ▷ Use Clients for parallel processing of many IPs ▷ Allows effective using of pipeline even with synchronous “call” 26
  27. 27. Fight bottlenecks 27
  28. 28. “Controlled parallelism” 28
  29. 29. Reusable components with module pipes 29 One should implement “init” and “call” functions (like in Elixir’s Plug library)
  30. 30. Advantages of Flowex approach 30 ▷ Explicit definition of: - structure of data will be processed - way the data will come ▷ Easy to maintain and reuse: - predefined set of working processes - components are isolated and reusable ▷ Controlled parallelism: - one can control number of clients supplying data - one can control the number of processes available for each component
  31. 31. 3. Running external programs Tools 31
  32. 32. Why do we need this? ▷ Elixir/Erlang ecosystem is not so big as Java, Ruby, Python, etc. ▷ Elixir/Erlang is not the fastest platform ▷ There are not so many Elixir/Erlang developers 32
  33. 33. Erlang ports ▷ provide a byte-oriented interface to an external program over STDIN and STDOUT ▷ are process-specific resources - only owner process can talk to external process ▷ are flexible, safe and allows error trapping 33
  34. 34. ErlPort project ▷ helps connect Erlang to other programming languages ▷ currently supported external languages are Python and Ruby ▷ has Elixir wrapper - export Read more on erlport.org 34
  35. 35. Porcelain project ▷ helps launching and communicating with external OS processes from Elixir ▷ launches external programs in a synchronous or asynchronous manner ▷ implements multiple ways of passing input to the program and getting back its output Read more on alco/porcelain 35
  36. 36. 4. Multi-language components Example project 36 https://github.com/antonmi/multi_flowex
  37. 37. MultiFlowex pipeline ▷ There will be Ruby, Python and shell “pipes” ▷ Each will say “Hello” ▷ The result will be: “Hello from Ruby, Hello from Python, Hello from shell” 37
  38. 38. multi_flowex dependencies 38
  39. 39. Ruby component Demonstrates how one can create components using Ruby, Python or Shell Read more on …. 39
  40. 40. Ruby code in main.rb file 40
  41. 41. Ruby component specs 41
  42. 42. Python component Demonstrates how one can create components using Ruby, Python or Shell Read more on …. 42
  43. 43. Python code in main.py file 43
  44. 44. Shell component 44
  45. 45. The last, Elixir component 45
  46. 46. MultiPipeline 46
  47. 47. Pipeline starts external processes 47 ▷ 3 Ruby processes ▷ 4 Python processes
  48. 48. MultiPipeline specs 48
  49. 49. Conclusion 49 ▷ FBP paradigm is an advantageous alternative to “conventional programming” ▷ Flowex makes it easy to convert sequential code into a pipeline of independent components ▷ Erlang ports open new horizons, so you can leverage the advantages of other programming languages
  50. 50. Thanks! Any questions? 50

×