1.
Practical Fault Tolerance
in Elixir
Alexei Sholik
Elixir Club Kyiv, 17 Mar 2018

2.
About me
Backend engineer at Contractbook.co.
Co-host at BeamEaters podcast.
Contributor to Elixir.
github.com/alco

3.
What is fault tolerance?

4.
Why it’s important

5.
Why do only
Erlang/Elixir communities
seem to care about it?

6.
A practical example
(demo)

7.
“Let it crash”
is not the full story

8.
Fail fast → restart → try again

9.
Building blocks of fault tolerance

10.
Process

11.
Error

12.
Link

13.
Monitor

14.
def call(process, request, timeout) do
monitor = Process.monitor(process)xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
send(process, {:"$gen_call", {self(), monitor}, request})
receive
{^monitor, reply} ->
Process.demonitor(monitor, [:flush])
{:ok, reply}
{:DOWN, ^monitor, _, _, reason} ->⁣xxxxxxxxxxxxxxxxxxxxxxxxxxxx
exit(reason)
after timeout ->
Process.demonitor(monitor, [:flush])
exit(:timeout)
end
end

15.
Task

16.
Supervisor

17.
Bohrbug

18.
Heisenbug

19.
Improving our example
(demo)

20.
...
DB
SuperDocs.Web.Endpoint
<request process>
<db_connection process>
Ecto’s connection pool
supervisor

21.
DB
SuperDocs.Web.Endpoint
<request process>
<db_connection process>
SuperDocs.TaskSup
User.send_confirmation_email
...

22.
Durable message queue
hex.pm/packages/amqp

23.
DB
SuperDocs.Web.Endpoint
<request process>
<db_connection process>
RabbitMQ
SuperDocs.TaskSup
ImportService.import_documents_for
AMQPClient
AMQPWorker AMQPWorker
...

24.
Other tools and techniques

25.
Remote error tracking
Sentry, Rollbar, log aggregators

26.
Exponential backoff
hex.pm/packages/db_connection
hex.pm/packages/gen_retry

27.
Alternative supervisor
hex.pm/packages/director

28.
And so on...

29.
Recap
● Trust in OTP…
● ...but don’t dismiss other tools
● Anticipate failures
● Isolate failures
● Fail fast, restart, try again

30.
● Why do computers fail and what can be done about it?
● Making reliable distributed systems in the presence of software
errors
● It's About the Guarantees
● On Erlang, State and Crashes
● Error Kernels
Reading material

31.
Thank you!
Questions?