Intro to Akka Streams

Agenda
• Reactive Streams
• Why Akka Streams?
• API Overview

public interface Publisher<T> {
public void subscribe(Subscriber<? super T> s);
}
public interface Subscriber<T> {
public void onSubscribe(Subscription s);
public void onNext(T t);
public void onError(Throwable t);
public void onComplete();
}
public interface Processor<T, R> extends Subscriber<T>, Publisher<R> {
}
public interface Subscription {
public void request(long n);
public void cancel();
}
Reactive Streams

A standardised spec/contract to achieve
asynchronous
back-pressured stream processing.

Standardised ?
Gives us consistent interop between libraries and
platforms that implement this spec.

everything is async & back-pressured

Reactive Streams
Stream API Stream API Stream API

Reactive Streams
Users use this API

Reactive Streams
Users use this API
Library authors use this API

• We know async IO from last week
• But there are other types of async operations, that
cross over different async boundaries
• between applications
• between threads
• and over the network as we saw

Think abstractly about these lines.
“async boundary”
This can be the network, or threads on the same CPU.
Publisher[T] Subscriber[T]

What problem are we trying
to solve?
Discrepancy in the rate of processing
• Fast Publisher / Slow Subscriber
• Slow Publisher / Fast Subscriber

100 messages /
1 second
1 message /
1second
Fast Slow

drop overflowed
require resending

has to keep track
of messages to resend
not safe & complicated

stop!

stop!
sh#t!

publisher didn’t receive NACK in time
so we lost that last message
not safe

100 messages /
1 second
1 message /
1second
FastSlow

gimme!

• Spam!
• Redundant messaging -> flooding the connection
• No buffer/batch support

We have to take into account the following scenarios:
• Fast Pub / Slow Sub
• Slow Pub / Fast Sub
Which can happen dynamically

Data
Demand(n)

Data
Demand(n)
Dynamic Push/Pull
bounded buffers with no overflow
demand can be accumulated
batch processing -> performance

• Cool let’s implement this using Actors!
• We can, it’s possible … but should it be done ?

The problem(s) with Akka Actors

Composition
In FP this makes us warm and fuzzy
val f: A => B
val g: B => C
val h: A => C = f andThen g

• Using Actors?
• An Actor is aware of who sent it messages and where it
must forward/reply them.
• No compositionality without thinking about it explicitly.

Data Flow
• What are streams ? Flows of data.
• Imagine a 10 stage data pipeline you want to model
• Now imagine writing that in Actors.

• Following the flow of data in Actors requires
jumping around all over the code base
• Low level, error prone and hard to reason about

Akka Streams API
building blocks

Design Philosophy
• Everything we will cover now are blueprints that
describe the actions/effects they perform.
• Reusability
• Compositionality

• “Design your program with a pure functional core,
push side-effects to the end of the world and
detonate to execute.
- some guy on stackoverflow

• Publisher of data
• Exactly one output
Image from boldradius.com

val singleSrc = Source.single(1)
val iteratorSrc = Source.fromIterator(() => Iterator from 0)
val futureSrc = Source.fromFuture(Future("abc"))
val collectionSrc = Source(List(1,2,3))
val tickSrc = Source.tick(
initialDelay = 1 second,
interval = 1 second,
tick = "tick-tock")
val requestSource = req.entity.dataBytes

• Subscriber (consumer) of data
• Describes where the data in our stream will go.
• Exactly one input

Sink.head
Sink.reduce[Int]((a, b) => a + b)
Sink.fold[Int, Int](0)(_ + _)
Sink.foreach[String](println)
FileIO.toPath(Paths.get("file.txt"))

val fold: Sink[Int, Future[Int]] = Sink.fold[Int, Int](0)(_ + _)

Input type

Materialized type

Materialized type
Available when the stream ‘completes’

val futureRes: Future[Int] = Source(1 to 10).runWith(fold)
futureRes.foreach(println)
// 55

So I can get data from somewhere
and I can put data somewhere else.
But I want to do something with it.

• A processor of data
• Has one input and one output

val double: Flow[Int, Int, NotUsed] = Flow[Int].map(_ * 2)

val src = Source(1 to 10)
val double = Flow[Int].map(_ * 2)
val negate = Flow[Int].map(_ * -1)
val print = Sink.foreach[Int](println)
val graph = src via double via negate to print
graph.run()
-2
-4
-6
-8
-10
-12
-14
-16
-18
-20

• Flow is immutable, thread-safe, and thus
freely shareable

• Are Linear flows enough ?
• No, we want to be able to describe arbitrarilly
complex steps in our pipelines

• We define multiple linear flows and then use the
Graph DSL to connect them.
• We can combine multiple streams - fan in
• Split a stream into substreams - fan out

Some sort of video uploading service
- Stream in video
- Process it
- Store it

bcast
ByteString
Convert to
Array[Byte]
flow
bcast
Process High
Res flow
Process Low
Res flow
Process Med
Res flow
sink
sink
sink

Sink.fromGraph(GraphDSL.create(highRes, mediumRes, lowRes)((_, _, _){ implicit b =>
(highSink, mediumSink, lowSink) => {
import GraphDSL.Implicits._
val bcastInput = b.add(Broadcast[ByteString](1))
val bcastRawBytes = b.add(Broadcast[Array[Byte]](3))
val processHigh: Flow[Array[Byte], ByteString, NotUsed]
val processMedium: Flow[Array[Byte], ByteString, NotUsed]
val processLow: Flow[Array[Byte], ByteString, NotUsed]
bcastInput.out(0) ~> byteAcc ~> bcastRawBytes ~> processHigh ~> highSink
bcastRawBytes ~> processMedium ~> mediumSink
bcastRawBytes ~> processLow ~> lowSink
SinkShape(bcastInput.in)
}
})
Our custom Sink

}
})
Has one input of type ByteString

Takes 3 Sinks, which can be Files, DBs, etc.
}
})

Describes 3 processing stages
That are Flows of Array[Byte] => ByteString
}
})

Describes 3 processing stages
That are Flows of Array[Byte] => ByteString
}
})
Emits result to the 3 Sinks

Has a type of:
Sink[ByteString, (Future[IOResult], Future[IOResult], Future[IOResult])]
}
})

Sink[ByteString, (Future[IOResult], Future[IOResult], Future[IOResult])]
Materialized values
}
})

Things we didn’t have time
for
• Integrating with Actors
• Buffering and throttling streams
• Defining custom Graph shapes and stages

Intro to Akka Streams

More Related Content

What's hot

Similar to Intro to Akka Streams

Recently uploaded

Intro to Akka Streams