Akka Streams provides a powerful framework for building scalable and resilient reactive applications at Zalando. Some initial challenges included understanding the graph DSL and handling errors and failures. Lessons were learned around Akka HTTP clients timing out requests and internal buffer sizes. Monitoring streams required building custom stages. Overall, Akka Streams enables easy tuning, scaling, reliability and testability once the reactive paradigm is understood. While it takes time to master, Akka Streams is a potent tool that can manage everything from simple to complex event processing at large scale.

1. H O P E S, R E G RETS A N D
“ B E ST P R ACTICES”
RISKING
EVERYT HING
W IT H
AKKA ST REAMS
08-12-20 1 6
JOACHIM HOFER
(@JOHOF ER )

2. 2
0. What We Do
1. First Impressions
2. Lessons Learnt
3. Awesomeness
4. Ops
5. Verdict
T ABLE OF
CONT ENTS

3. 3
WHAT WE DO @ ZALANDO

4. 4
net sales 2015: 3 billion euros
several 1000 updates / second
latency: ideally seconds
WHAT WE DO: SOME NUMBERS

5. 5
“ SOME” RISK INVOLVED!

6. 6
Akka Streams vs. Futures, RxScala, Actors
Tech Blog “Which shoe fits you?”
https://github.com/zalando/
scala-concurrency-playground
Akka Streams fits us best!
Image: Nevit Dilmen (CC BY-SA 3.0)
FIRST IMPRESSIONS — PREPARATION

7. 7
H U H?
FIRST IMPRESSIONS — GRAPH DSL
import GraphDSL.Implicits._
val bcSqsIn = b add Broadcast[StreamCreationEvent](2)
val rules = b add ruleStore.stage.async
val eval = b add evaluator.stage.async
val publish = b add nakadi.stage.async
val ack = b add ackStage.stage.async
bcSqsIn ~> rules ~> eval.in0
bcSqsIn ~> eval.in1; eval.out ~> publish ~> ack
FlowShape(bcSqsIn.in, ack.out)

8. 8
A H !
FIRST IMPRESSIONS — PLAIN SOURCES
products
.mapAsync(parallelism = 5)(ruleEvaluatorEvent(flowId))
.groupedWithin(3, 100 millis)
.filter(_.nonEmpty)
.mapAsync(parallelism = 50)(sqsGateway.send)
.runForeach(result => log.info(result.getFailed.size))

9. 9 Images: Jerry Daykin (CC BY 2.0, left), The Present Group (CC BY 3.0 US, right)
FIRST IMPRESSIONS — OTHER BIG SCARIES
MAT ERIALISATION B A C K P R E S SU R E

10. 1 0
T AST Y DOCUMENTATION
Image: Wikivisual (CC BY-NC-SA 3.0)

11. 1 1
MISTAKES W ERE MADE
Image: Hapesoft (public domain)

12. 1 2
Caused by onError
Completes the stream
Recover using recover / recoverWithRetries
FAILURES (VS ERRORS)

13. 1 3
Caused by exceptions
Get escalated to failures by default
Recover using a Supervision Strategy
Can be ignored easily (“Resume” strategy)
ERRORS (VS FAILURES)

14. 1 4
R E S UMING
LESSONS LEARNT
RETRIEVE
EVENTS
EVENTS
RETRIEVE
RULES
EVALUAT E1

15. 1 5
R E S UMING
LESSONS LEARNT
RETRIEVE
EVENTS
EVENTS
RETRIEVE
RULES
EVALUAT E
1
1

16. 1 6
R E S UMING
LESSONS LEARNT
RETRIEVE
EVENTS
EVENTS
RETRIEVE
RULES
EVALUAT E
1

17. 1 7
LESSONS LEARNT
RETRIEVE
EVENTS
EVENTS
RETRIEVE
RULES
EVALUAT E
1
2

18. 1 8
LESSONS LEARNT
RETRIEVE
EVENTS
EVENTS
RETRIEVE
RULES
EVALUAT E
1
2
2

19. 1 9
LESSONS LEARNT
RETRIEVE
EVENTS
EVENTS
RETRIEVE
RULES
EVALUAT E
1
2
2

20. 2 0
G O IN G O U T - OF-SYNC
LESSONS LEARNT
RETRIEVE
EVENTS
EVENTS
RETRIEVE
RULES
EVALUAT E
12
2

21. 2 1
Unconsumed response entities
Solution: discardEntityBytes
LESSONS LEARNT — AKKA-HT TP CLIENTS (1)

22. 2 2
No response from the server
“Currently Akka HTTP doesn’t implement client-side request timeout
checking itself as this functionality can be regarded as a more general
purpose streaming infrastructure feature.” (akka http docs)
Solution: Use an explicit xxxTimeout stage
LESSONS LEARNT — AKKA-HT TP CLIENTS (2)

23. 2 3
Default: only up to 16?!
16 (buffer size) x 50 (parallel flows) x 10 (events per batch) = 8000 events
Solution: keep low, use explicit buffer stages
LESSONS LEARNT — INTERNAL BUFFERS

24. 2 4
C O N FIGURATION A S A S O U RCE?
Image: Alpha (CC BY-SA 2.0)
LESSONS LEARNT
RETRIEVE
EVENTS
EVENTS
RETRIEVE
RULES
EVALUAT E
CONFI GU R A TI ON

25. 2 5
REACTIVE
MANIFESTO

26. 2 6
Backpressure
Automatically asynchronous
AWESOMENESS — REACTIVE

27. 2 7
AWESOMENESS — TESTABILITY
EVALUAT ETEST EVENTS OUTPUT S TO CHECK

28. 2 8
groupedWithin
throttle
mapConcat
recoverWithRetries
…
AWESOMENESS — BUILT -I N STAGES
Flow[RuleEvaluationEvent]
.groupedWithin(batchSize, batchTimeout)
.mapAsync(parallelism = 1)(provider.deleteFromStreamCreation)
.mapConcat(identity)
.via(logAndMonitor(Checkpoint.Acknowledged, "acknowledged msg"))

29. 2 9
no out-of-the-box solution
built-in stage monitor not very helpful
no access to internal buffers
OPS — MONITORI NG: THE BAD

30. 3 0
Trace events along streams
Create your own monitoring stage
OPS — MONITORI NG: THE GOOD

31. 3 1
E X A MPLE P A S S -THROUGH S T AG E L O G IC
OPS — MONITORI NG
…
new GraphStageLogic(shape) {
setHandlers(in, out, new InHandler with OutHandler {
override def onPush(): Unit = {
push(out, grab(in))
}
override def onPull(): Unit = {
pull(in)
}
})
}
…

32. 3 2
E X A MPLE P A S S -THROUGH S T AG E L O G IC
OPS — MONITORI NG
…
new GraphStageLogic(shape) {
setHandlers(in, out, new InHandler with OutHandler {
override def onPush(): Unit = {
stats.countPush()
push(out, grab(in))
}
override def onPull(): Unit = {
stats.countPull()
pull(in)
}
})
}
…

33. 3 3
easy to tune
very efficient
easy to scale
see also: Gearpump Materializer
Image: Duncan Rawlinson (CC BY 2.0)
OPS — TUNING AND SCALING

34. 3 4
everything under control
just keeps on running
OPS — RELIABILITY

35. 3 6
Takes time to understand
Very potent
Image: Twice25 (CC BY-SA 2.5)
VERDICT: IT’S MAGIC!

36. 3 7
R I S K ➟ REW ARD

37. joachim.hofer@zalando.de
@johofer
J O A CHIM H O F ER
Availability Engineering
Backend Engineer
08-12-20 1 6