Reactive streaming is becoming the best approach to handle data flows across asynchronous boundaries. Here, we present the implementation of a real-world application based on Akka Streams. After reviewing the basics, we will discuss the development of a data processing pipeline that collects real-time sensor data and sends it to a Kinesis stream. There are various possible point of failures in this architecture. What should happen when Kinesis is unavailable? If the data flow is not handled in the correct way, some information may get lost. Akka Streams are the tools that enabled us to build a reliable processing logic for the pipeline that avoids data losses and maximizes the robustness of the entire system.

1. A dive into Akka Streams
From the basics to a real-world scenario
Gioia Ballin, Simone Fabiano - SW developers

2. Who are these guys?
Gioia Ballin
Twitter / @GioiaBallin
Simone Fabiano
Twitter / @mone_tpatpc
R&D Data Analysis Microservices Infrastructure WebBlog

3. Agenda
The Problem
Akka Streams Basics
The Collector Service
Recap and Learnings

4. Data processing pipeline
COLLECTOR SVC AMAZON KINESIS
WI-FI/3GSENSORS

5. The Collector Service: two months ago...
AMAZON
KINESIS
INBOX
ACTOR
TCP
Kinesis Client
INBOX
ACTOR Kinesis Client
INBOX
ACTOR Kinesis Client
TCP
TCP
chunks
chunks
chunks
events
events
events
put
put
put

7. Data may be lost
COLLECTOR SVC AMAZON KINESIS
WI-FI/3GSENSORS
CONNECTION
ISSUES
MESSAGE
OVERLOAD

8. Not REALLY reactive: why?
Lack of...
RECOVERY MECHANISMS
BACKPRESSURE

9. Recovery mechanisms
Upload failure? Retry
Kinesis unavailable? Buffer

10. Ingestion speed under control
Take advantage of sensors buffer
Sensors switch from busier to less busy services

11. Welcome Akka Streams!
Backpressure on the network
+
Easy fit for stream paradigm
Typed flow & Increased Readability

12. Akka Streams Basics - 3 2 1 Go!

13. SOURCE FLOW SINK

14. val source = Source(1 to 42)
val flow = Flow[Int].map(_ + 1)
val sink = Sink.foreach(println)
val graph = source.via(flow).to(sink)
graph.run()

15. Flows are collections (almost)
OK map, filter, fold, …
No flatMap! (there’s mapConcat though)
Extras mapAsync, mapAsyncUnordered, buffer..

16. How backpressure works
Source Flow Sink
pull
pull
push
push

17. Collector with Akka Streams

18. path("data") {
post {
extractRequest { request =>
val source = request.entity.dataBytes
val flow = processing()
val sink = Sink.ignore
source.via(flow).runWith(sink)
… } }

19. Flow[ByteString]
.via(chunkBuffer)

20. Transformation tools: Framing.delimiter
val chunkBuffer = Framing.delimiter(
ByteString("n"),
maxPresenceEventBytes,
false
)
.map(_.dropRight(1))
.map(_.utf8String)

21. Flow[ByteString]
.via(chunkBuffer)
.via(presenceEventFactory.asStream)

22. Flow[ByteString]
.via(chunkBuffer)
.via(presenceEventFactory.asStream)
.via(persistentBuffer)

23. class PersistentBuffer[A] (...)
extends GraphStage[FlowShape[A, A]] {
val in =
Inlet[A]("PersistentBuffer.in")
val out =
Outlet[A]("PersistentBuffer.out")
override val shape = FlowShape.of(in,out)
When the available stages are not enough, write your own

24. Custom stages: State
override def createLogic(
attr: Attributes
): GraphStageLogic =
new GraphStageLogic(shape) {
var state: StageState[A] = initialState[A]

25. val cb = getAsyncCallback[Try[A]] {
case Success(elements:A) =>
state = state.copy(...)
...
}
queue.getCurrentQueue
.onComplete(cb.invoke)

26. Custom Stages: ports
setHandler(in, new InHandler {
override def onPush() = {
val element = grab(in)
pull(in)
...
}
...
})
setHandler(out, new OutHandler {
override def onPull(): Unit = {
push(out, something)
}
...
})

27. Custom Stages: Start and stop
override def postStop(): Unit = {
...
}
override def preStart(): Unit = {
...
}

28. Flow[ByteString]
.via(chunkBuffer)
.via(presenceEventFactory.asStream)
.via(persistentBuffer)
.via(kinesisPublisher)

29. Delegating work to actors
Flow[KinesisEvent]
.mapAsync(1) { event =>
(publisher ? Publish(event))
.mapTo[PublishCompleted]
}

30. Flow[KinesisEvent]
.mapAsync(1) { event =>
(publisher ? Publish(event))
.mapTo[Future[PublishCompleted]]
.flatten
}

31. Flow[ByteString]
.via(chunkBuffer)
.via(presenceEventFactory.asStream)
.via(persistentBuffer)
.via(kinesisPublisher)
.alsoTo(countEvents)

32. Extra side-effects
val countEvents = Sink.foreach[Event] { _ =>
metricEventsPerTimeUnitOfSensor.mark()
metricEventsPerTimeUnit.mark()
}

33. Automatic Fusion and Async Stages
A stream is handled by 1 thread
NO CROSS-THREAD COMMUNICATIONS
NO PARALLELISM
FASTER CODE!
SLOWER CODE!
A boundary will split your stream on different threads
source
.via(doSomething).async
.via(doSomething).async
.runWith(Sink.foreach(println))

34. Materialized Values
val sink: Sink[Any, Future[Done]] = Sink.ignore
.runWith(sink) is a shortcut for .toMat(sink)(Keep.Right).run
STREAM RUN
Obtain a materialized
value per each stage

35. Materialized Values Example: TestKit
val myFlow = Flow[String].map { v => v.take(1) }
val (pub, sub) = TestSource.probe[String]
.via(myFlow)
.toMat(TestSink.probe[Any])(Keep.both)
.run()
sub.request(1)
pub.sendNext("Gathering")
sub.expectNext("G")

36. Stream Ending: Supervisioning
val flow = otherFlow
.withAttributes
ActorAttributes.supervisionStrategy {
case _ => Supervision.resume
}
)

37. Stream Ending: Supervisioning
RESUME
RESET
STOP
> drop failing elem
> keep going
> drop failing elem
> reset stage state
> keep going
fail all the things!

38. Other shapes
FAN OUT FAN IN WHATEVER

39. Finally Got Reactive!

40. In two months to production...
Memory Leaks
Data Losses
Backpressure on the network
Recovery mechanisms

41. Key learnings
Actors aren’t sufficient to be reactive
Backpressure as a building block
Akka Streams API is rich

42. THANK YOU!
Sleep more worry less