A light introduction to Elixir Flow, event streaming and streaming processing

1. Elixir Flow

2. Renato
@renatorfr
Software developer @ Mirego

3. Elixir Flow
"Computational flows with stages"

4. Map Reduce
COLLECTION PROCESSING RESULT

5. Enum
File.stream!("path/to/some/file")
|> Enum.flat_map(&String.split(&1, " "))
|> Enum.reduce(%{}, fn word, acc ->
Map.update(acc, word, 1, & &1
+ 1)
end)
|> Enum.to_list()

6. Stream
File.stream!("path/to/some/file")
|> Stream.flat_map(&String.split(&1, " "))
|> Enum.reduce(%{}, fn word, acc ->
Map.update(acc, word, 1, & &1
+ 1)
end)
|> Enum.to_list()

7. Flow

8. GENSTAGE
https://github.com/elixir-lang/gen_stage

9. BOUNDED DATA

10. UNBOUNDED DATA

11. EVENT STREAMING

12. "Realtime" monitoring

13. Stream
File.stream!("path/to/some/file")
|> Stream.flat_map(&String.split(&1, " "))
|> Enum.reduce(%{}, fn word, acc ->
Map.update(acc, word, 1, & &1
+ 1)
end)
|> Enum.to_list()

14. WINDOWS

16. Flow.Window
● Global window
● Count window
● Session window
● Fixed window
● Periodic window

17. Event time vs Processing time

18. CODE TIME

19. Flow
Flow.from_enumerable(...)
|> Flow.filter(...)
|> Flow.partition(...)
|> Flow.reduce(...)
|> Flow.on_trigger(...)
|> Flow.run()

20. Conclusions
● Tool in our toolbelt
● Flow is in memory
● Keep your flows small
● Results speed vs Results precision

22. References
Elixir Conf Keynote 2016
https://youtu.be/srtMWzyqdp8
Announcing Genstage
https://elixir-lang.org/blog/2016/07/14/announcing-genstage/
Flow docs
https://hexdocs.pm/flow/Flow.html
Demo code
https://github.com/renatorfr/elixir-flow