akka streams
Reactive Integrations with
that just work™
Johan Andrén
Scala Usergroup Stockholm 2016-10-18
Johan Andrén
Akka Team
Stockholm Scala User Group
Make building powerful concurrent &
distributed applications simple.
Akka is a toolkit and runtime
for building highly concurrent,
distributed, and resilient
message-driven applications
on the JVM
Actors – simple & high performance concurrency
Cluster / Remoting – location transparency, resilience
Cluster tools – and more prepackaged patterns
Streams – back-pressured stream processing
Persistence – Event Sourcing
HTTP – complete, fully async and reactive HTTP Server
Official Kafka, Cassandra, DynamoDB integrations, tons
more in the community
Complete Java & Scala APIs for all features
What’s in the toolkit?
“Stream”has many meanings…
akka streams
Asynchronous back pressured stream processing
Source Sink
Flow
akka streams
Asynchronous back pressured stream processing
Source Sink
(possible)
asynchronous
boundaries
Flow
akka streams
Asynchronous back pressured stream processing
Source Sink
10 msg/s 1 msg/s
OutOfMemoryError!!
Flow
akka streams
Asynchronous back pressured stream processing
Source Sink
10 msg/s 1 msg/s
hand me 3 morehand me 3 more
1 msg/s Flow
akka streams
Not only linear streams
Source
SinkFlow
Source
Sink
Flow
Flow
Reactive Streams
Reactive Streams is an initiative to provide a standard for
asynchronous stream processing with non-blocking back
pressure. This encompasses efforts aimed at runtime
environments (JVM and JavaScript) as well as network
protocols
http://www.reactive-streams.org
Part of JDK 9
java.util.concurrent.Flow
Reactive Streams
RS Library A RS library B
async
boundary
Reactive Streams
RS Library A RS library B
async
boundary
Make building powerful concurrent &
distributed applications simple.
The API
Akka Streams
Complete and awesome
Java and Scala APIs
(Just like everything in Akka)
Akka Streams in 20 seconds:
val source: Source[Int, NotUsed] =

Source(0 to 200000)



val flow: Flow[Int, String, NotUsed] =

Flow[Int].map(_.toString)



val sink: Sink[String, Future[Done]] =

Sink.foreach(println)



val runnableGraph =
source.via(flow).to(sink)



runnableGraph.run()
Akka Streams in 20 seconds:
source.via(flow).to(sink)

Source[Int, NotUsed]
Flow[Int, String, NotUsed]
Sink[String, Future[Done]]
Materialization
Gears from GeeCON.org,(it’s an awesome conf)
What is “materialization” really?
What is “materialization” really?
What is “materialization” really?
What is “materialization” really?
AlpakkaA community for Streams connectors
http://blog.akka.io/integrations/2016/08/23/intro-alpakka
Alpakka – a community for Stream connectors
Threading & Concurrency in Akka Streams Explained (part I)
Mastering GraphStages (part I, Introduction)
Akka Streams Integration, codename Alpakka
A gentle introduction to building Sinks and Sources using GraphStage APIs
(Mastering GraphStages, Part II)
Writing Akka Streams Connectors for existing APIs
Flow control at the boundary of Akka Streams and a data provider
Akka Streams Kafka 0.11
Alpakka – a community for Stream connectors
Existing examples:
MQTT
AMQP
Streaming HTTP
Streaming TCP
Streaming FileIO
Cassandra Queries
“Reactive Kafka” (akka-stream-kafka)
S3, SQS & other Amazon APIs
Streaming JSON
Streaming XML
…
Alpakka – a community for Stream connectors
Demo
KafkaSink
FileTailSource
Kafka
Bytes =>
Lines
Alpakka – a community for Stream connectors
Demo
Kafka
parse line
window
last 10
Find
hacker
KafkaSource
Sound the
alarm!
Alpakka – a community for Stream connectors
Kafka Stream Stream
Stream
Stream
cluster
Akka Streams & HTTP
streams
& HTTP
Akka Streams / HTTP
Quiz time!
Akka Streams / HTTP
recv buffer
send buffer
🚚
🚚
🚚
🚚
🚚
🚚
🚚
Akka Streams / HTTP
recv buffer
send buffer
🚚
🚚
🚚
🚚
🚚
🚚
🚚
🚚
🚚
🚚
🚑
Streaming in Akka HTTP
http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala
“Framed entity streaming”
http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html
HttpServer as a:
Flow[HttpRequest, HttpResponse]
Streaming in Akka HTTP
HttpServer as a:
Flow[HttpRequest, HttpResponse]
HTTP Entity as a:
Source[ByteString, _]
http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala
“Framed entity streaming”
http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html
Streaming in Akka HTTP
HttpServer as a:
Flow[HttpRequest, HttpResponse]
HTTP Entity as a:
Source[ByteString, _]
Websocket connection as a:
Flow[ws.Message, ws.Message]
http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala
“Framed entity streaming”
http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html
Streaming from Akka HTTP
send buffer
🚚
🚚
OSApp
EntrySource
Json =>
Bytes
Entry =>
Json
TcpSink
Streaming from Akka HTTP
send buffer
🚚
🚚
🚚
OSApp
EntrySource
Json =>
Bytes
Entry =>
Json
TcpSink
🚚
🚚
Streaming from Akka HTTP
send buffer
🚚
🚚
🚚
OSApp
EntrySource
Json =>
Bytes
Entry =>
Json
TcpSink
🚚
🚚
No demand from TCP
=
No demand upstream
=
Source won’t read log
=>
Bounded memory
stream processing!
Demo
Akka ❤ contributions
Easy to contribute tickets:
https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy-to-contribute
https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3A%22nice-to-have+%28low-prio%29%22
Akka Stream Contrib
https://github.com/akka/akka-stream-contrib
Mailing list:
https://groups.google.com/group/akka-user
Public chat rooms:
http://gitter.im/akka/dev developing Akka
http://gitter.im/akka/akka using Akka
Thanks for listening!
@apnylle
johan.andren@lightbend.com
Sample sources
https://github.com/johanandren/akka-stream-samples/tree/scala-stockholm-2016-10

Scala usergroup stockholm - reactive integrations with akka streams

  • 1.
    akka streams Reactive Integrationswith that just work™ Johan Andrén Scala Usergroup Stockholm 2016-10-18
  • 2.
  • 3.
    Make building powerfulconcurrent & distributed applications simple. Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM
  • 4.
    Actors – simple& high performance concurrency Cluster / Remoting – location transparency, resilience Cluster tools – and more prepackaged patterns Streams – back-pressured stream processing Persistence – Event Sourcing HTTP – complete, fully async and reactive HTTP Server Official Kafka, Cassandra, DynamoDB integrations, tons more in the community Complete Java & Scala APIs for all features What’s in the toolkit?
  • 5.
  • 6.
    akka streams Asynchronous backpressured stream processing Source Sink Flow
  • 7.
    akka streams Asynchronous backpressured stream processing Source Sink (possible) asynchronous boundaries Flow
  • 8.
    akka streams Asynchronous backpressured stream processing Source Sink 10 msg/s 1 msg/s OutOfMemoryError!! Flow
  • 9.
    akka streams Asynchronous backpressured stream processing Source Sink 10 msg/s 1 msg/s hand me 3 morehand me 3 more 1 msg/s Flow
  • 10.
    akka streams Not onlylinear streams Source SinkFlow Source Sink Flow Flow
  • 11.
    Reactive Streams Reactive Streamsis an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments (JVM and JavaScript) as well as network protocols http://www.reactive-streams.org
  • 12.
    Part of JDK9 java.util.concurrent.Flow
  • 13.
    Reactive Streams RS LibraryA RS library B async boundary
  • 14.
    Reactive Streams RS LibraryA RS library B async boundary Make building powerful concurrent & distributed applications simple.
  • 15.
    The API Akka Streams Completeand awesome Java and Scala APIs (Just like everything in Akka)
  • 16.
    Akka Streams in20 seconds: val source: Source[Int, NotUsed] =
 Source(0 to 200000)
 
 val flow: Flow[Int, String, NotUsed] =
 Flow[Int].map(_.toString)
 
 val sink: Sink[String, Future[Done]] =
 Sink.foreach(println)
 
 val runnableGraph = source.via(flow).to(sink)
 
 runnableGraph.run()
  • 17.
    Akka Streams in20 seconds: source.via(flow).to(sink)
 Source[Int, NotUsed] Flow[Int, String, NotUsed] Sink[String, Future[Done]]
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
    AlpakkaA community forStreams connectors http://blog.akka.io/integrations/2016/08/23/intro-alpakka
  • 24.
    Alpakka – acommunity for Stream connectors Threading & Concurrency in Akka Streams Explained (part I) Mastering GraphStages (part I, Introduction) Akka Streams Integration, codename Alpakka A gentle introduction to building Sinks and Sources using GraphStage APIs (Mastering GraphStages, Part II) Writing Akka Streams Connectors for existing APIs Flow control at the boundary of Akka Streams and a data provider Akka Streams Kafka 0.11
  • 25.
    Alpakka – acommunity for Stream connectors Existing examples: MQTT AMQP Streaming HTTP Streaming TCP Streaming FileIO Cassandra Queries “Reactive Kafka” (akka-stream-kafka) S3, SQS & other Amazon APIs Streaming JSON Streaming XML …
  • 26.
    Alpakka – acommunity for Stream connectors Demo KafkaSink FileTailSource Kafka Bytes => Lines
  • 27.
    Alpakka – acommunity for Stream connectors Demo Kafka parse line window last 10 Find hacker KafkaSource Sound the alarm!
  • 28.
    Alpakka – acommunity for Stream connectors Kafka Stream Stream Stream Stream cluster
  • 29.
    Akka Streams &HTTP streams & HTTP
  • 30.
    Akka Streams /HTTP Quiz time!
  • 31.
    Akka Streams /HTTP recv buffer send buffer 🚚 🚚 🚚 🚚 🚚 🚚 🚚
  • 32.
    Akka Streams /HTTP recv buffer send buffer 🚚 🚚 🚚 🚚 🚚 🚚 🚚 🚚 🚚 🚚 🚑
  • 33.
    Streaming in AkkaHTTP http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming” http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html HttpServer as a: Flow[HttpRequest, HttpResponse]
  • 34.
    Streaming in AkkaHTTP HttpServer as a: Flow[HttpRequest, HttpResponse] HTTP Entity as a: Source[ByteString, _] http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming” http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html
  • 35.
    Streaming in AkkaHTTP HttpServer as a: Flow[HttpRequest, HttpResponse] HTTP Entity as a: Source[ByteString, _] Websocket connection as a: Flow[ws.Message, ws.Message] http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming” http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html
  • 36.
    Streaming from AkkaHTTP send buffer 🚚 🚚 OSApp EntrySource Json => Bytes Entry => Json TcpSink
  • 37.
    Streaming from AkkaHTTP send buffer 🚚 🚚 🚚 OSApp EntrySource Json => Bytes Entry => Json TcpSink 🚚 🚚
  • 38.
    Streaming from AkkaHTTP send buffer 🚚 🚚 🚚 OSApp EntrySource Json => Bytes Entry => Json TcpSink 🚚 🚚 No demand from TCP = No demand upstream = Source won’t read log => Bounded memory stream processing! Demo
  • 39.
    Akka ❤ contributions Easyto contribute tickets: https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy-to-contribute https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3A%22nice-to-have+%28low-prio%29%22 Akka Stream Contrib https://github.com/akka/akka-stream-contrib Mailing list: https://groups.google.com/group/akka-user Public chat rooms: http://gitter.im/akka/dev developing Akka http://gitter.im/akka/akka using Akka
  • 40.
    Thanks for listening! @apnylle johan.andren@lightbend.com Samplesources https://github.com/johanandren/akka-stream-samples/tree/scala-stockholm-2016-10