1. Practical Fault Tolerance
in Elixir
Alexei Sholik
Elixir Club Kyiv, 17 Mar 2018

2. About me
Backend engineer at Contractbook.co.
Co-host at BeamEaters podcast.
Contributor to Elixir.
github.com/alco

3. What is fault tolerance?

5. Why it’s important

6. Why do only
Erlang/Elixir communities
seem to care about it?

7. A practical example
(demo)

8. “Let it crash”
is not the full story

9. Fail fast → restart → try again

11. Building blocks of fault tolerance

12. Process

15. Error

16. Link

18. Monitor

20. def call(process, request, timeout) do
monitor = Process.monitor(process)xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
send(process, {:"$gen_call", {self(), monitor}, request})
receive
{^monitor, reply} ->
Process.demonitor(monitor, [:flush])
{:ok, reply}
{:DOWN, ^monitor, _, _, reason} ->⁣xxxxxxxxxxxxxxxxxxxxxxxxxxxx
exit(reason)
after timeout ->
Process.demonitor(monitor, [:flush])
exit(:timeout)
end
end

21. Task

22. Supervisor

23. Bohrbug

24. Heisenbug

25. Improving our example
(demo)

26. ...
DB
SuperDocs.Web.Endpoint
<request process>
<db_connection process>
Ecto’s connection pool
supervisor

27. DB
SuperDocs.Web.Endpoint
<request process>
<db_connection process>
SuperDocs.TaskSup
User.send_confirmation_email
...

28. Durable message queue
hex.pm/packages/amqp

29. DB
SuperDocs.Web.Endpoint
<request process>
<db_connection process>
RabbitMQ
SuperDocs.TaskSup
ImportService.import_documents_for
AMQPClient
AMQPWorker AMQPWorker
...

30. Other tools and techniques

31. Remote error tracking
Sentry, Rollbar, log aggregators

32. Exponential backoff
hex.pm/packages/db_connection
hex.pm/packages/gen_retry

33. Alternative supervisor
hex.pm/packages/director

34. And so on...

35. Recap
● Trust in OTP…
● ...but don’t dismiss other tools
● Anticipate failures
● Isolate failures
● Fail fast, restart, try again

36. ● Why do computers fail and what can be done about it?
● Making reliable distributed systems in the presence of software
errors
● It's About the Guarantees
● On Erlang, State and Crashes
● Error Kernels
Reading material

37. Thank you!
Questions?