Introduction to Snabbkaffe

Introduction to Snabbkae
@k32
April 25, 2021
1 / 46
©2021 EMQ Technologies Co., Ltd.

Outline
Background and motivation
Instrumenting the code
Testing
Basic functions for working with traces
Event correlations
Nemesis
Using snabbkae as a benchmark library
Conclusion
2 / 46

Topic
Testing
Event correlations
Nemesis
Conclusion
3 / 46

What is snabbkae
I Snabbkae is a library that allows to test concurrent and distributed
systems
I It does so by moving focus from states to eects
I Developers nd bugs by looking at the logs, snabbkae does the same and
automates the process
I Advanced modes of testing: fault and scheduling injection
I Eciency: run test scenario once, verify multiple properties
4 / 46

Name
I Snabbkaffe means instant coee in Swedish
I There is no hidden meaning, the name was chosen randomly
5 / 46

Nondeterminism
Denitions
I Deterministic functions always produce the same output for the same input
I Nondeterministic functions can produce dierent outputs for the same input
Sources of nondeterminism
I Process schedulings
I Fault-tolerance (failovers)
I Network (packet loss and reordering)
I Load balancing, worker pools
Nondeterminism is an inherent property of concurrent systems
6 / 46

Why change the paradigm of testing?
Diculty levels
1. Easy: Pure functions
2. Normal: Stateful systems
3. Hard: Nondeterministic systems
4. Nightmare: Nondeterministic fault-tolerant distributed systems
Traditional approach
I Traditional approach is sucient for testing pure functions. There are a
variety of excellent tools for this, like PropER
I It can do pretty well with stateful systems (stateful PropER)
I Where it starts to crack: typical Erlang application assembled together
7 / 46

Challenges of testing Erlang applications in situ
Erlang partitions the state of the system between processes
I It eliminates some concurrency bugs and isolates failures
I But it makes the system harder to analyze
I More often than not it's impossible to inspect state of the process
(sys:get_state)
I Often it's useless
I Tests that rely on the state usually need to wait for the system to stabilize,
it can be slow, and they can't analyze the system in motion
I Asynchronous data processing patterns (e.g. job queues) are incredibly
annoying to test using traditional methods
8 / 46

Possible solution to this problem?
Move away from states and embrace events and their temporal relationships
9 / 46

Topic
Testing
Event correlations
Nemesis
Conclusion
10 / 46

Tracing macros
1. Replace your regular logs with structured logs (the industry is moving
towards structured logs anyway)
2. Include snabbkaffe/include/trace.hrl header to your module
3. Use ?tp or ?tp_span macros for logging (tp stands for Trace Point)
I In the release build these macros will become regular logger messages
I In the test build these macros will emit trace events
11 / 46

Examples
%% Disappears in the release build:
?tp(rlog_replica_import_trans,
#{ seqno = SeqNo
, txid = TXID
, transaction = Transaction
})
%% Becomes a notice message
?tp(notice, Remote process died,
#{ reason = Reason
, my_state = State
})
%% Emits two events:
?tp_span(trans_write, #{record = Record, txid = get_txid()},
mnesia:write(Record)).
12 / 46

Distributed tracing
Forward traces from the remote node:
%% on the ct_master:
snabbkaffe:forward_trace(Node)
All features (incl. fault and scheduling injections) will work
13 / 46

Why not using erlang tracing?
An often asked question: why not using dbg:tracer or the like?
It was a conscious design choice:
I Traces obtained this way are tightly bound to the structure of the code.
Refactoring the code or adding a new function argument would break the
tests
I Debug traces are excessive, it would be hard to see the forest through the
trees
I Log messages are typically placed in interesting places
I Interesting for humans = interesting for tests
14 / 46

Topic
Testing
Event correlations
Nemesis
Conclusion
15 / 46

Stages of the test
Every testcase is split in two stages:
Run stage
where the program runs and emits the event trace
Check stage
where the trace is collected as a list of Erlang maps and validated against the
spec(s)
16 / 46

check_trace macro
my_test() -
?check_trace(begin
%% Run stage:
launch_system(),
do_stuff(),
wait_for_completion(),
get_result()
end,
fun(Result, Trace) -
%% Check stage:
?assertMatch(ok, Result),
check_history(Trace),
check_history2(Trace),
...
end).
17 / 46

Waiting for events
It is possible to block run stage until a certain event occurs:
?block_until(#{ ?snk_kind := message_acked
, message_id := Id
} when Id 42,
Timeout, BackInTime)
This macro will return immediately as soon as the event happens, so it's more
ecient and less prone to akiness than a sleep.
18 / 46

Starting an asynchronous action
?block_until is ne, but it requires timeout tuning. What if the same event
happened in the past?
?wait_async_action( send_async_request(Req)
, #{ ?snk_kind := request_handled
, request := Req
}
[, Timeout]
)
This macro doesn't have to look into the past.
19 / 46

Integrating with PropER
It's not a problem to combine stateless PropER tests with snabbkae.
?check_trace macro can be used inside proper's ?FORALL macro.
Snabbkae provides some convenience macros that simplify
trace-property-based testing:
trace_prop_test(Config) -
Prop = ?forall_trace(
X, list(),
begin
%% Run stage:
do_stuff(X)
end,
fun(Result, Trace) -
%% Check stage
true
end),
?run_prop(Config, Prop).
20 / 46

Misc. macros
There are a few macros that are not directly related to trace-based testing, but
often needed:
?panic
?panic(Unexpected result, #{result = A})
Throw an error
?retry
?retry(_RetryPeriod = 100, _MaxRetries = 10,
some_action_that_can_fail())
Run some code periodically until it succeeds (use as the last resort, when
?block_until won't work)
?give_or_take
?give_or_take(_Expected = 20, _Deviation = 2, Value)
Check that an integer value is within range
21 / 46

Topic
Testing
Event correlations
Nemesis
Conclusion
22 / 46

Structure of the trace event
[#{ ?snk_kind = foo
, ?snk_meta = #{ domain = [foo, bar]
, node = 'foo@localhost'
, time = monotonic time
, pid = 0.343.0
, group_leader = 0.67.0
, custom_field1 = baz
, custom_field2 = foobar
}
, field1 = 42
, field2 = foo
},
...
]
23 / 46

Structure of the span trace
[#{ ?snk_kind = foo
, ?snk_meta = #{ ... }
, ?snk_span = start
, field1 = 42
, field2 = foo
},
#{ ?snk_kind = foo
, ?snk_meta = #{ ... }
, ?snk_span = {complete, ReturnValue}
, field1 = 42
, field2 = foo
}
]
24 / 46

Filtering the trace
?of_kind macro
FooEvents = ?of_kind(foo, Trace)
FooOrBarEvents = ?of_kind([foo, bar], Trace)
?of_domain macro
EMQXEvents = ?of_domain([emqx|_], Trace)
?of_node macro
NodeFooEvents = ?of_node('foo@localhost', Trace)
More complex ltering
Of course, lists:filter and list comprehensions work too
25 / 46

Extracting elds
[1, 2] = ?projection(id, ?of_kind(handle_message, Trace))
[{1, hello}, {2, world}] =
?projection( [id, message]
, ?of_kind(handle_message, Trace)
)
26 / 46

Topic
Testing
Event correlations
Nemesis
Conclusion
27 / 46

?strict_causality macro
?strict_causality( #{?snk_kind := msg_received, id := _Id}
, #{?snk_kind := msg_processed, id := _Id}
, Trace
)
Return value:
I true if some matching events were found
I false if no events were found
I exception if causality is violated
28 / 46

?strict_causality with guard
Suppose we're testing a base64 server:
?strict_causality( #{req := _Req}
, #{resp := _Resp}
, _Resp =:= base64:encode(_Req)
, Trace
)
29 / 46

?causality macro
Overall, it's the same as ?strict_causality, except it doesn't require each
cause to have an eect
?causality( #{?snk_kind := msg_received, id := _Id}
, #{?snk_kind := msg_processed, id := _Id}
[, Guard]
, Trace
)
30 / 46

Pitfalls
There is a potential problem with causality macros:
1. Format of the event is changed in the code
2. Match expressions in causality stop matching the events
3. Tests still pass, even though they didn't nd any events
Solution:
Always wrap ?causality and ?strict_causality in ?assert, unless there
are legit situations when no events can be produced in the test
31 / 46

?nd_pairs macro
?find_pairs(Strict, MatchCause, MatchEffect [, Guard] , Trace)
1. It returns a list of {pair, Cause, Effect} or {singleton, Cause}
2. When Strict is true this macro also checks that eects don't occur
before causes (much like ?causality)
32 / 46

Splitting traces
Often it is useful to split traces to parts before and after some event (for
example, restart)
There are a variety of macros for this:
split_at
?split_trace_at(bar, [foo, bar, baz, bar]).
{[foo], [bar, baz, bar]}.
splitr
?splitr_trace(foo, [1, 2, foo, 1, 2, foo])
[[1, 2], [foo, 1, 2], [foo]].
splitl
?splitl_trace(foo, [1, 2, foo, 1, 2, foo]).
[[1, 2, foo], [1, 2, foo]].
33 / 46

Topic
Testing
Event correlations
Nemesis
Conclusion
34 / 46

Are your supervisors tested?
I Taking fault-tolerance seriously is one of the selling points of Erlang.
Organizing processes in supervisor trees is used widely, but not often tested
I Tuning supervisor trees is an art
I Snabbkae wants to turn it into a chore
I It does so by injecting deliberate faults into the system
35 / 46

Fault injection
Any tracepoint can be used to inject errors into the system
?inject_crash( #{?snk_meta := #{domain := [ekka, rlog|_]}}
, snabbkaffe_nemesis:random_crash(0.1)
)
I First argument: event matching expression
I Second argument: fault scenario
36 / 46

Fault scenarios
Always crash:
?inject_crash(..., snabbkaffe_nemesis:always_crash())
Crash N times:
?inject_crash(..., snabbkaffe_nemesis:recover_after(10))
Crash randomly with probability P:
?inject_crash(..., snabbkaffe_nemesis:random_crash(0.1))
Crash periodically:
snabbkaffe_nemesis:periodic_crash(
_Period = 10, _DutyCycle = 0.5, _Phase = math:pi())
37 / 46

Manipulating process schedulings
I Sometimes it is necessary to test a certain process scheduling
I Imagine working on a bugx using TDD-style
I For example, normally eect foo occurs after eect bar, and everything
works. But in rare cases the opposite happens, and the bug occurs
I Snabbkae can manipulate process schedulings to some extend:
%% run stage...
?force_ordering(#{?snk_kind := bar}, #{?snk_kind := foo})
(It also supports guards)
38 / 46

Topic
Testing
Event correlations
Nemesis
Conclusion
39 / 46

Disclaimer
I This usecase is a secondary, and quite rudimentary. Don't expect much
I It lacks many features of proper benchmarking libraries, such as
I Warmup
I Advanced statistical analysis, e.g. outlier detection
40 / 46

Reporting metrics
%% Report a single scalar datapoint:
snabbkaffe:push_stat(scalar_metric_name, Value),
%% Report a single datapoint with X coord:
snabbkaffe:push_stat(metric_name, X, Value),
%% Report multiple scalar datapoints:
snabbkaffe:push_stats(metric_name, [Val1, Val2, ...]),
%% Report multiple datapoints with X coord:
snabbkaffe:push_stats(metric_name, X, [Val1, Val2, ...])
41 / 46

Using spans to analyze performance
push_stats function also works together with ?find_pairs macro:
Pairs = ?find_pairs( #{ ?snk_span := start
, ?snk_kind := foo
, pid := _Pid
}
, #{ ?snk_span := {complete, _}
, ?snk_kind := foo
, pid := _Pid
}
),
snabbkaffe:push_stats(metric_name, [X,] Pairs)
42 / 46

Analyzing the results
Use the following function in the end of check stage:
snabbkaffe:analyze_statistics()
It will print the results in the console:
Mean scalar_metric_name: 10.0
Statisitics of metric_name
100.479087 ^ *
| *
| *
|
| *
| *
0 +----------------------------------------
0 1100
43 / 46

Topic
Testing
Event correlations
Nemesis
Conclusion
44 / 46

Useful links
https://github.com/kafka4beam/snabbkaffe
The library is being actively developed.
Suggestions, bug reports and patches are welcome!
Snabbkae in action
Feel free to use as an example (clickable links):
I brod
I kow
I ekka
I EMQ X
45 / 46

Questions?
46 / 46

Introduction to Snabbkaffe

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Introduction to Snabbkaffe

Similar to Introduction to Snabbkaffe (20)

More from EMQ

More from EMQ (6)

Recently uploaded

Recently uploaded (20)

Introduction to Snabbkaffe