Intro to Akka Streams

Agenda
• Reactive Streams
• Why Akka Streams?
• API Overview

public interface Publisher<T> {
public void subscribe(Subscriber<? super T> s);
}
public interface Subscriber<T> {
public void onSubscribe(Subscription s);
public void onNext(T t);
public void onError(Throwable t);
public void onComplete();
}
public interface Processor<T, R> extends Subscriber<T>, Publisher<R> {
}
public interface Subscription {
public void request(long n);
public void cancel();
}
Reactive Streams

A standardised spec/contract to achieve
asynchronous
back-pressured stream processing.

Standardised ?
Gives us consistent interop between libraries and
platforms that implement this spec.

everything is async & back-pressured

Reactive Streams
Stream API Stream API Stream API

Reactive Streams
Users use this API

Reactive Streams
Users use this API
Library authors use this API

• We know async IO from last week
• But there are other types of async operations, that
cross over different async boundaries
• between applications
• between threads
• and over the network as we saw

Think abstractly about these lines.
“async boundary”
This can be the network, or threads on the same CPU.
Publisher[T] Subscriber[T]

What problem are we trying
to solve?
Discrepancy in the rate of processing
• Fast Publisher / Slow Subscriber
• Slow Publisher / Fast Subscriber

100 messages /
1 second
1 message /
1second
Fast Slow

drop overflowed
require resending

has to keep track
of messages to resend
not safe & complicated

stop!

stop!
sh#t!

publisher didn’t receive NACK in time
so we lost that last message
not safe

100 messages /
1 second
1 message /
1second
FastSlow

gimme!

• Spam!
• Redundant messaging -> flooding the connection
• No buffer/batch support

We have to take into account the following scenarios:
• Fast Pub / Slow Sub
• Slow Pub / Fast Sub
Which can happen dynamically

Data
Demand(n)

Data
Demand(n)
Dynamic Push/Pull
bounded buffers with no overflow
demand can be accumulated
batch processing -> performance

• Cool let’s implement this using Actors!
• We can, it’s possible … but should it be done ?

The problem(s) with Akka Actors

Composition
In FP this makes us warm and fuzzy
val f: A => B
val g: B => C
val h: A => C = f andThen g

• Using Actors?
• An Actor is aware of who sent it messages and where it
must forward/reply them.
• No compositionality without thinking about it explicitly.

Data Flow
• What are streams ? Flows of data.
• Imagine a 10 stage data pipeline you want to model
• Now imagine writing that in Actors.

• Following the flow of data in Actors requires
jumping around all over the code base
• Low level, error prone and hard to reason about

Akka Streams API
building blocks

Design Philosophy
• Everything we will cover now are blueprints that
describe the actions/effects they perform.
• Reusability
• Compositionality

• “Design your program with a pure functional core,
push side-effects to the end of the world and
detonate to execute.
- some guy on stackoverflow

• Publisher of data
• Exactly one output
Image from boldradius.com

val singleSrc = Source.single(1)
val iteratorSrc = Source.fromIterator(() => Iterator from 0)
val futureSrc = Source.fromFuture(Future("abc"))
val collectionSrc = Source(List(1,2,3))
val tickSrc = Source.tick(
initialDelay = 1 second,
interval = 1 second,
tick = "tick-tock")
val requestSource = req.entity.dataBytes

• Subscriber (consumer) of data
• Describes where the data in our stream will go.
• Exactly one input

Sink.head
Sink.reduce[Int]((a, b) => a + b)
Sink.fold[Int, Int](0)(_ + _)
Sink.foreach[String](println)
FileIO.toPath(Paths.get("file.txt"))

val fold: Sink[Int, Future[Int]] = Sink.fold[Int, Int](0)(_ + _)

Input type

Materialized type

Materialized type
Available when the stream ‘completes’

val futureRes: Future[Int] = Source(1 to 10).runWith(fold)
futureRes.foreach(println)
// 55

So I can get data from somewhere
and I can put data somewhere else.
But I want to do something with it.

• A processor of data
• Has one input and one output

val double: Flow[Int, Int, NotUsed] = Flow[Int].map(_ * 2)

val src = Source(1 to 10)
val double = Flow[Int].map(_ * 2)
val negate = Flow[Int].map(_ * -1)
val print = Sink.foreach[Int](println)
val graph = src via double via negate to print
graph.run()
-2
-4
-6
-8
-10
-12
-14
-16
-18
-20

• Flow is immutable, thread-safe, and thus
freely shareable

• Are Linear flows enough ?
• No, we want to be able to describe arbitrarilly
complex steps in our pipelines

• We define multiple linear flows and then use the
Graph DSL to connect them.
• We can combine multiple streams - fan in
• Split a stream into substreams - fan out

Some sort of video uploading service
- Stream in video
- Process it
- Store it

bcast
ByteString
Convert to
Array[Byte]
flow
bcast
Process High
Res flow
Process Low
Res flow
Process Med
Res flow
sink
sink
sink

Sink.fromGraph(GraphDSL.create(highRes, mediumRes, lowRes)((_, _, _){ implicit b =>
(highSink, mediumSink, lowSink) => {
import GraphDSL.Implicits._
val bcastInput = b.add(Broadcast[ByteString](1))
val bcastRawBytes = b.add(Broadcast[Array[Byte]](3))
val processHigh: Flow[Array[Byte], ByteString, NotUsed]
val processMedium: Flow[Array[Byte], ByteString, NotUsed]
val processLow: Flow[Array[Byte], ByteString, NotUsed]
bcastInput.out(0) ~> byteAcc ~> bcastRawBytes ~> processHigh ~> highSink
bcastRawBytes ~> processMedium ~> mediumSink
bcastRawBytes ~> processLow ~> lowSink
SinkShape(bcastInput.in)
}
})
Our custom Sink

}
})
Has one input of type ByteString

Takes 3 Sinks, which can be Files, DBs, etc.
}
})

Describes 3 processing stages
That are Flows of Array[Byte] => ByteString
}
})

Describes 3 processing stages
That are Flows of Array[Byte] => ByteString
}
})
Emits result to the 3 Sinks

Has a type of:
Sink[ByteString, (Future[IOResult], Future[IOResult], Future[IOResult])]
}
})

Sink[ByteString, (Future[IOResult], Future[IOResult], Future[IOResult])]
Materialized values
}
})

Things we didn’t have time
for
• Integrating with Actors
• Buffering and throttling streams
• Defining custom Graph shapes and stages

Intro to Akka Streams

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Intro to Akka Streams

Similar to Intro to Akka Streams (20)

Recently uploaded

Recently uploaded (20)

Intro to Akka Streams