The talk will be about Flow-Based programming (FBP) using Flowex library built on top of Elixir GenStage feature. You will know about Flowex Railway-FBP design approach, about its abstractions, and the way to easily create pipelines of independent components. I’ll also demonstrate how one can use Ruby, Python, and, in general, any other programming language inside Flowex pipes and therefore create reusable components with language-specific functionality.

1. Multi-language
FBP with Flowex
#pivorak 21, Elixir edition, March 24, 2017

2. Hello!
I am Anton Mishchuk
▷ Ruby developer at Matic Insurance Services
(we are hiring!)
▷ A big fan of Elixir programming language
▷ Author and maintainer of:
ESpec and Flowex libraries
github: antonmi
2

3. Preface
Flow-based programming with Elixir
3

4. Preface
Flowex - FBP with Elixir GenStage
4

5. What’s next?
▷ Let Elixir do what it does best
▷ The rest can be implemented in
any other programming language
5

6. Agenda
▷ Quick intro to FBP
▷ Flowex library overview
▷ Tools for running external programs
▷ Multi-language components example:
Ruby, Python, shell
6

7. 1.
Flow-Based Programming
Basic concepts
7

8. Flow-Based Programming is
a programming paradigm which
models application as a graph of
independent processes exchanging
data across predefined connections
8

9. FBP diagram
A
B
D
C
IN 1
IN 1
IN 1
IN 2
IN 1
IN 2
OUT 1
OUT 2
OUT 1
OUT 1
OUT 1
9

10. FBP advantages
▷ Parallelism are natural
▷ No problems associated with shared
memory and locks
▷ Visualisation of data flow networks is simple
▷ Components are independent and reusable
10

11. Two layers in FBP application
▷ “Bottom layer” - set of components
implementing parts of business
logic
▷ “Top layer” - “communication logic”
- an organization of data flows
from one component to another
11

12. “Bottom layer” - components
▷ Component is independent unit
with input and output
▷ The only thing it does - transform
input IP to output IP
▷ In general, can be implemented
using any programming language
12

13. “Top layer” - communication
▷ Need “coordination language”
▷ Actor model of Elixir/Erlang
▷ GenStage
13

14. GenStage
▷ behavior for exchanging events with
back-pressure between Elixir processes
▷ José Valim defined GenStage as “better
abstractions for working with collections”
▷ but we can use it for “better interprocess
communication”
14

15. 2.
Flowex
Railway Flow-Based Programming
15

16. Flowex
▷ a set of abstractions built on top Elixir
GenStage which allows writing
program with FBP paradigm
▷ a mix of FBP and Railway Oriented
Programming (ROP) approach
16

17. Railway oriented programming
17

18. Railway FBP
Let’s place each function inside
separate process
18

19. use Flowex.Pipeline
19

20. Functions’ definitions
20

21. Control and evaluation
21
Some new functions appear in the
FunPipeline module:
▷ “start” and “stop”
▷ “call” and “cast”

22. FunPipeline.start
22

23. “call” and “cast” (sync and async)
23

24. FunPipeline.call
24
add_one
minus_three
mult_by_two
producer
consumer
number: 2
number: 2
number: 3
number: 6
number: 3number: 3
self()

25. “error_pipe”
If an error happens in some pipe, functions
in next pipes will not be called and IP will be
bypassed to the “error_pipe”
25

26. Run pipeline via
Flowex.Client
▷ Use Clients for
parallel processing of
many IPs
▷ Allows effective using
of pipeline even with
synchronous “call”
26

27. Fight bottlenecks
27

28. “Controlled parallelism”
28

29. Reusable components with
module pipes
29
One should implement “init” and
“call” functions
(like in Elixir’s Plug library)

30. Advantages of Flowex approach
30
▷ Explicit definition of:
- structure of data will be processed
- way the data will come
▷ Easy to maintain and reuse:
- predefined set of working processes
- components are isolated and reusable
▷ Controlled parallelism:
- one can control number of clients
supplying data
- one can control the number of processes
available for each component

31. 3.
Running external programs
Tools
31

32. Why do we need this?
▷ Elixir/Erlang ecosystem is not so big as
Java, Ruby, Python, etc.
▷ Elixir/Erlang is not the fastest platform
▷ There are not so many Elixir/Erlang
developers
32

33. Erlang ports
▷ provide a byte-oriented interface to an
external program over STDIN and STDOUT
▷ are process-specific resources - only owner
process can talk to external process
▷ are flexible, safe and allows error trapping
33

34. ErlPort project
▷ helps connect Erlang to other
programming languages
▷ currently supported external languages
are Python and Ruby
▷ has Elixir wrapper - export
Read more on erlport.org
34

35. Porcelain project
▷ helps launching and communicating with
external OS processes from Elixir
▷ launches external programs in a
synchronous or asynchronous manner
▷ implements multiple ways of passing input
to the program and getting back its output
Read more on alco/porcelain 35

36. 4.
Multi-language components
Example project
36
https://github.com/antonmi/multi_flowex

37. MultiFlowex pipeline
▷ There will be Ruby, Python and shell “pipes”
▷ Each will say “Hello”
▷ The result will be:
“Hello from Ruby, Hello from Python, Hello
from shell”
37

38. multi_flowex dependencies
38

39. Ruby component
Demonstrates how one can create
components using Ruby, Python or Shell
Read more on ….
39

40. Ruby code in main.rb file
40

41. Ruby component specs
41

42. Python component
Demonstrates how one can create
components using Ruby, Python or Shell
Read more on ….
42

43. Python code in main.py file
43

44. Shell component
44

45. The last, Elixir component
45

46. MultiPipeline
46

47. Pipeline starts external processes
47
▷ 3 Ruby processes
▷ 4 Python processes

48. MultiPipeline specs
48

49. Conclusion
49
▷ FBP paradigm is an advantageous
alternative to “conventional programming”
▷ Flowex makes it easy to convert sequential
code into a pipeline of independent
components
▷ Erlang ports open new horizons,
so you can leverage the advantages of
other programming languages

50. Thanks!
Any questions?
50