Reactive Streams: Handling Data-Flow the Reactive Way

Roland Kuhn
Roland KuhnCTO at Actyx AG
Dr. Roland Kuhn
Akka Tech Lead
@rolandkuhn
Reactive Streams
Handling Data-Flows the Reactive Way
Introduction: Streams
What is a Stream?
• ephemeral flow of data
• focused on describing transformation
• possibly unbounded in size
3
Common uses of Streams
• bulk data transfer
• real-time data sources
• batch processing of large data sets
• monitoring and analytics
4
What is special about Reactive Streams?
Reactive	
  Applications
The Four Reactive Traits
6
http://reactivemanifesto.org/
Needed: Asynchrony
• Resilience demands it:
• encapsulation
• isolation
• Scalability demands it:
• distribution across nodes
• distribution across cores
7
Many Kinds of Async Boundaries
8
Many Kinds of Async Boundaries
• between different applications
8
Many Kinds of Async Boundaries
• between different applications
• between network nodes
8
Many Kinds of Async Boundaries
• between different applications
• between network nodes
• between CPUs
8
Many Kinds of Async Boundaries
• between different applications
• between network nodes
• between CPUs
• between threads
8
Many Kinds of Async Boundaries
• between different applications
• between network nodes
• between CPUs
• between threads
• between actors
8
The Problem:
!
Getting Data across an Async Boundary
Possible Solutions
10
Possible Solutions
• the Traditional way: blocking calls
10
Possible Solutions
10
Possible Solutions
!
• the Push way: buffering and/or dropping
11
Possible Solutions
11
Possible Solutions
!
!
• the Reactive way:

non-blocking & non-dropping & bounded
12
How do we achieve that?
Supply and Demand
• data items flow downstream
• demand flows upstream
• data items flow only when there is demand
• recipient is in control of incoming data rate
• data in flight is bounded by signaled demand
14
Publisher Subscriber
data
demand
Dynamic Push–Pull
• “push” behavior when consumer is faster
• “pull” behavior when producer is faster
• switches automatically between these
• batching demand allows batching data
15
Publisher Subscriber
data
demand
Explicit Demand: Tailored Flow Control
16
demand
data
splitting the data means merging the demand
Explicit Demand: Tailored Flow Control
17
merging the data means splitting the demand
Reactive Streams
• asynchronous non-blocking data flow
• asynchronous non-blocking demand flow
• minimal coordination and contention
• message passing allows for distribution
• across applications
• across nodes
• across CPUs
• across threads
• across actors
18
Are Streams Collections?
What is a Collection?
20
What is a Collection?
• Oxford Dictionary:
• “a group of things or people”
20
What is a Collection?
• Oxford Dictionary:
• “a group of things or people”
• wikipedia:
• “a grouping of some variable number of data items”
20
What is a Collection?
• Oxford Dictionary:
• “a group of things or people”
• wikipedia:
• “a grouping of some variable number of data items”
• backbone.js:
• “collections are simply an ordered set of models”
20
What is a Collection?
• Oxford Dictionary:
• “a group of things or people”
• wikipedia:
• “a grouping of some variable number of data items”
• backbone.js:
• “collections are simply an ordered set of models”
• java.util.Collection:
• definite size, provides an iterator, query membership
20
User Expectations
• an Iterator is expected to visit all elements

(especially with immutable collections)
• x.head + x.tail == x
• the contents does not depend on who is
processing the collection
• the contents does not depend on when the
processing happens

(especially with immutable collections)
21
Streams have Unexpected Properties
• the observed sequence depends on
• … when the observer subscribed to the stream
• … whether the observer can process fast enough
• … whether the streams flows fast enough
22
Streams are not Collections!
• java.util.stream:

Stream is not derived from Collection

“Streams differ from Coll’s in several ways”
• no storage
• functional in nature
• laziness seeking
• possibly unbounded
• consumable
23
Streams are not Collections!
• a collection can be streamed
• a stream observer can create a collection
• … but saying that a Stream is just a lazy
Collection evokes the wrong associations
24
So, Reactive Streams:
why not just java.util.stream.Stream?
Java 8 Stream
26
import java.util.stream.*;
!
// get some stream
final Stream<Integer> s = Stream.of(1, 2, 3);
// describe transformation
final Stream<String> s2 = s.map(i -> "a" + i);
// make a pull collection
s2.iterator();
// or alternatively push it somewhere
s2.forEach(i -> System.out.println(i));
// (need to pick one, Stream is consumable)
Java 8 Stream
• provides a DSL for describing transformation
• introduces staged computation

(but does not allow reuse)
• prescribes an eager model of execution
• offers either push or pull, chosen statically
27
What about RxJava?
RxJava
29
import rx.Observable;
import rx.Observable.*;
!
// get some stream source
final Observable<Integer> obs = range(1, 3);
// describe transformation
final Observable<String> obs2 =
obs.map(i -> "b" + i);
// and use it twice
obs2.subscribe(i -> System.out.println(i));
obs2.filter(i -> i.equals("b2"))
.subscribe(i -> System.out.println(i));
RxJava
• implements pure “push” model
• includes extensive DSL for transformations
• only allows blocking for back pressure
• currently uses unbounded buffering for
crossing an async boundary
• work on distributed Observables sparked
participation in Reactive Streams
30
The Reactive Streams Project
Participants
• Engineers from
• Netflix
• Oracle
• Pivotal
• Red Hat
• Twitter
• Typesafe
• Individuals like Doug Lea and Todd Montgomery
32
The Motivation
• all participants had the same basic problem
• all are building tools for their community
• a common solution benefits everybody
• interoperability to make best use of efforts
• e.g. use Reactor data store driver with Akka
transformation pipeline and Rx monitoring to drive a
vert.x REST API (purely made up, at this point)
33
see also Jon Brisbin’s post on “Tribalism as a Force for Good”
Recipe for Success
• minimal interfaces
• rigorous specification of semantics
• full TCK for verification of implementation
• complete freedom for many idiomatic APIs
34
The Meat
35
trait Publisher[T] {
def subscribe(sub: Subscriber[T]): Unit
}
trait Subscription {
def requestMore(n: Int): Unit
def cancel(): Unit
}
trait Subscriber[T] {
def onSubscribe(s: Subscription): Unit
def onNext(elem: T): Unit
def onError(thr: Throwable): Unit
def onComplete(): Unit
}
The Sauce
• all calls on Subscriber must dispatch async
• all calls on Subscription must not block
• Publisher is just there to create Subscriptions
36
How does it Connect?
37
SubscriberPublisher
subscribe
onSubscribeSubscription
requestMore
onNext
Akka Streams
Akka Streams
• powered by Akka Actors
• execution
• distribution
• resilience
• type-safe streaming through Actors with
bounded buffering
39
Basic Akka Example
40
implicit val system = ActorSystem("Sys")
val mat = FlowMaterializer(...)
!
Flow(text.split("s").toVector).
map(word => word.toUpperCase).
foreach(tranformed => println(tranformed)).
onComplete(mat) {
case Success(_) => system.shutdown()
case Failure(e) =>
println("Failure: " + e.getMessage)
system.shutdown()
}
More Advanced Akka Example
41
val s = Source.fromFile("log.txt", "utf-8")
Flow(s.getLines()).
groupBy {
case LogLevelPattern(level) => level
case other => "OTHER"
}.
foreach { case (level, producer) =>
val out = new PrintWriter(level+".txt")
Flow(producer).
foreach(line => out.println(line)).
onComplete(mat)(_ => Try(out.close()))
}.
onComplete(mat)(_ => Try(s.close()))
Java 8 Example
42
final ActorSystem system = ActorSystem.create("Sys");
final MaterializerSettings settings =
MaterializerSettings.create();
final FlowMaterializer materializer =
FlowMaterializer.create(settings, system);
!
final String[] lookup = { "a", "b", "c", "d", "e", "f" };
final Iterable<Integer> input = Arrays.asList(0, 1, 2, 3, 4, 5);
!
Flow.create(input).drop(2).take(3). // leave 2, 3, 4
map(elem -> lookup[elem]). // translate to "c","d","e"
filter(elem -> !elem.equals("c")). // filter out the "c"
grouped(2). // make into a list
mapConcat(list -> list). // flatten the list
fold("", (acc, elem) -> acc + elem). // accumulate into "de"
foreach(elem -> System.out.println(elem)). // print it
consume(materializer);
Akka TCP Echo Server
43
val future = IO(StreamTcp) ? Bind(addr, ...)
future.onComplete {
case Success(server: TcpServerBinding) =>
println("listen at "+server.localAddress)
!
Flow(server.connectionStream).foreach{ c =>
println("client from "+c.remoteAddress)
c.inputStream.produceTo(c.outputStream)
}.consume(mat)
!
case Failure(ex) =>
println("cannot bind: "+ex)
system.shutdown()
}
Akka HTTP Sketch (NOT REAL CODE)
44
val future = IO(Http) ? RequestChannelSetup(...)
future.onSuccess {
case ch: HttpRequestChannel =>
val p = ch.processor[Int]
val body = Flow(new File(...)).toProducer(mat)
Flow(HttpRequest(method = POST,
uri = "http://ex.amp.le/42",
entity = body) -> 42)
.produceTo(mat, p)
Flow(p).map { case (resp, token) =>
val out = FileSink(new File(...))
Flow(resp.entity.data).produceTo(mat, out)
}.consume(mat)
}
Closing Remarks
Current State
• Early Preview is available:

"org.reactivestreams" % "reactive-streams-spi" % "0.2"

"com.typesafe.akka" %% "akka-stream-experimental" % "0.3"
• check out the Activator template

"Akka Streams with Scala!"

(https://github.com/typesafehub/activator-akka-stream-scala)
46
Next Steps
• we work towards inclusion in future JDK
• try it out and give feedback!
• http://reactive-streams.org/
• https://github.com/reactive-streams
47
©Typesafe 2014 – All Rights Reserved
1 of 60

Recommended

Physical Plans in Spark SQL by
Physical Plans in Spark SQLPhysical Plans in Spark SQL
Physical Plans in Spark SQLDatabricks
7K views126 slides
Native Support of Prometheus Monitoring in Apache Spark 3.0 by
Native Support of Prometheus Monitoring in Apache Spark 3.0Native Support of Prometheus Monitoring in Apache Spark 3.0
Native Support of Prometheus Monitoring in Apache Spark 3.0Databricks
2.4K views41 slides
Best Practice of Compression/Decompression Codes in Apache Spark with Sophia... by
 Best Practice of Compression/Decompression Codes in Apache Spark with Sophia... Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...
Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...Databricks
6.1K views19 slides
Hardening Kafka Replication by
Hardening Kafka Replication Hardening Kafka Replication
Hardening Kafka Replication confluent
5.2K views232 slides
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in... by
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...InfluxData
3.6K views55 slides
Druid by
DruidDruid
DruidDori Waldman
1.6K views45 slides

More Related Content

What's hot

Memory access tracing [poug17] by
Memory access tracing [poug17]Memory access tracing [poug17]
Memory access tracing [poug17]Mahmoud Hatem
3.9K views52 slides
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai by
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin HuaiA Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin HuaiDatabricks
10.9K views41 slides
Let’s Make Your CFO Happy; A Practical Guide for Kafka Cost Reduction with El... by
Let’s Make Your CFO Happy; A Practical Guide for Kafka Cost Reduction with El...Let’s Make Your CFO Happy; A Practical Guide for Kafka Cost Reduction with El...
Let’s Make Your CFO Happy; A Practical Guide for Kafka Cost Reduction with El...HostedbyConfluent
637 views62 slides
Transactional writes to cloud storage with Eric Liang by
Transactional writes to cloud storage with Eric LiangTransactional writes to cloud storage with Eric Liang
Transactional writes to cloud storage with Eric LiangDatabricks
2.4K views46 slides
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent by
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, ConfluentKafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, ConfluentHostedbyConfluent
1.9K views30 slides
Percona Live 2012PPT: MySQL Query optimization by
Percona Live 2012PPT: MySQL Query optimizationPercona Live 2012PPT: MySQL Query optimization
Percona Live 2012PPT: MySQL Query optimizationmysqlops
4.3K views139 slides

What's hot(20)

Memory access tracing [poug17] by Mahmoud Hatem
Memory access tracing [poug17]Memory access tracing [poug17]
Memory access tracing [poug17]
Mahmoud Hatem3.9K views
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai by Databricks
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin HuaiA Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
Databricks10.9K views
Let’s Make Your CFO Happy; A Practical Guide for Kafka Cost Reduction with El... by HostedbyConfluent
Let’s Make Your CFO Happy; A Practical Guide for Kafka Cost Reduction with El...Let’s Make Your CFO Happy; A Practical Guide for Kafka Cost Reduction with El...
Let’s Make Your CFO Happy; A Practical Guide for Kafka Cost Reduction with El...
HostedbyConfluent637 views
Transactional writes to cloud storage with Eric Liang by Databricks
Transactional writes to cloud storage with Eric LiangTransactional writes to cloud storage with Eric Liang
Transactional writes to cloud storage with Eric Liang
Databricks2.4K views
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent by HostedbyConfluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, ConfluentKafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
HostedbyConfluent1.9K views
Percona Live 2012PPT: MySQL Query optimization by mysqlops
Percona Live 2012PPT: MySQL Query optimizationPercona Live 2012PPT: MySQL Query optimization
Percona Live 2012PPT: MySQL Query optimization
mysqlops 4.3K views
Exactly-Once Financial Data Processing at Scale with Flink and Pinot by Flink Forward
Exactly-Once Financial Data Processing at Scale with Flink and PinotExactly-Once Financial Data Processing at Scale with Flink and Pinot
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Flink Forward699 views
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ... by Databricks
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
Databricks5K views
Magnet Shuffle Service: Push-based Shuffle at LinkedIn by Databricks
Magnet Shuffle Service: Push-based Shuffle at LinkedInMagnet Shuffle Service: Push-based Shuffle at LinkedIn
Magnet Shuffle Service: Push-based Shuffle at LinkedIn
Databricks487 views
ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl... by Altinity Ltd
ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...
ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...
Altinity Ltd1.2K views
On Improving Broadcast Joins in Apache Spark SQL by Databricks
On Improving Broadcast Joins in Apache Spark SQLOn Improving Broadcast Joins in Apache Spark SQL
On Improving Broadcast Joins in Apache Spark SQL
Databricks1.6K views
Maximum Overdrive: Tuning the Spark Cassandra Connector (Russell Spitzer, Dat... by DataStax
Maximum Overdrive: Tuning the Spark Cassandra Connector (Russell Spitzer, Dat...Maximum Overdrive: Tuning the Spark Cassandra Connector (Russell Spitzer, Dat...
Maximum Overdrive: Tuning the Spark Cassandra Connector (Russell Spitzer, Dat...
DataStax16.3K views
Low latency microservices in java QCon New York 2016 by Peter Lawrey
Low latency microservices in java   QCon New York 2016Low latency microservices in java   QCon New York 2016
Low latency microservices in java QCon New York 2016
Peter Lawrey7.3K views
SQL on everything, in memory by Julian Hyde
SQL on everything, in memorySQL on everything, in memory
SQL on everything, in memory
Julian Hyde7.4K views
Spark Summit EU talk by Ted Malaska by Spark Summit
Spark Summit EU talk by Ted MalaskaSpark Summit EU talk by Ted Malaska
Spark Summit EU talk by Ted Malaska
Spark Summit8.8K views
Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal-(Josh Ro... by Spark Summit
Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal-(Josh Ro...Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal-(Josh Ro...
Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal-(Josh Ro...
Spark Summit18.2K views
[2019] 바르게, 빠르게! Reactive를 품은 Spring Kafka by NHN FORWARD
[2019] 바르게, 빠르게! Reactive를 품은 Spring Kafka[2019] 바르게, 빠르게! Reactive를 품은 Spring Kafka
[2019] 바르게, 빠르게! Reactive를 품은 Spring Kafka
NHN FORWARD1K views
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes by DataWorks Summit
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on KubernetesApache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
DataWorks Summit3.3K views
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013 by mumrah
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
mumrah61.3K views
cLoki: Like Loki but for ClickHouse by Altinity Ltd
cLoki: Like Loki but for ClickHousecLoki: Like Loki but for ClickHouse
cLoki: Like Loki but for ClickHouse
Altinity Ltd1.1K views

Similar to Reactive Streams: Handling Data-Flow the Reactive Way

Reactive Streams 1.0.0 and Why You Should Care (webinar) by
Reactive Streams 1.0.0 and Why You Should Care (webinar)Reactive Streams 1.0.0 and Why You Should Care (webinar)
Reactive Streams 1.0.0 and Why You Should Care (webinar)Legacy Typesafe (now Lightbend)
35.6K views42 slides
Introduction to Akka - Atlanta Java Users Group by
Introduction to Akka - Atlanta Java Users GroupIntroduction to Akka - Atlanta Java Users Group
Introduction to Akka - Atlanta Java Users GroupRoy Russo
2.4K views87 slides
Streamlining with rx by
Streamlining with rxStreamlining with rx
Streamlining with rxAkhil Dad
128 views36 slides
Akka Streams and HTTP by
Akka Streams and HTTPAkka Streams and HTTP
Akka Streams and HTTPRoland Kuhn
41.6K views58 slides
Drinking from the Firehose - Real-time Metrics by
Drinking from the Firehose - Real-time MetricsDrinking from the Firehose - Real-time Metrics
Drinking from the Firehose - Real-time MetricsSamantha Quiñones
1.1K views88 slides
ELK stack introduction by
ELK stack introduction ELK stack introduction
ELK stack introduction abenyeung1
37 views60 slides

Similar to Reactive Streams: Handling Data-Flow the Reactive Way(20)

Introduction to Akka - Atlanta Java Users Group by Roy Russo
Introduction to Akka - Atlanta Java Users GroupIntroduction to Akka - Atlanta Java Users Group
Introduction to Akka - Atlanta Java Users Group
Roy Russo2.4K views
Streamlining with rx by Akhil Dad
Streamlining with rxStreamlining with rx
Streamlining with rx
Akhil Dad128 views
Akka Streams and HTTP by Roland Kuhn
Akka Streams and HTTPAkka Streams and HTTP
Akka Streams and HTTP
Roland Kuhn41.6K views
Drinking from the Firehose - Real-time Metrics by Samantha Quiñones
Drinking from the Firehose - Real-time MetricsDrinking from the Firehose - Real-time Metrics
Drinking from the Firehose - Real-time Metrics
Samantha Quiñones1.1K views
ELK stack introduction by abenyeung1
ELK stack introduction ELK stack introduction
ELK stack introduction
abenyeung137 views
CTS2 Development Framework In Action by cts2framework
CTS2 Development Framework In ActionCTS2 Development Framework In Action
CTS2 Development Framework In Action
cts2framework767 views
Meetup on Apache Zookeeper by Anshul Patel
Meetup on Apache ZookeeperMeetup on Apache Zookeeper
Meetup on Apache Zookeeper
Anshul Patel678 views
Introduction to SolrCloud by Varun Thacker
Introduction to SolrCloudIntroduction to SolrCloud
Introduction to SolrCloud
Varun Thacker2.9K views
"Data Provenance: Principles and Why it matters for BioMedical Applications" by Pinar Alper
"Data Provenance: Principles and Why it matters for BioMedical Applications""Data Provenance: Principles and Why it matters for BioMedical Applications"
"Data Provenance: Principles and Why it matters for BioMedical Applications"
Pinar Alper397 views
Introduction to Galaxy and RNA-Seq by Enis Afgan
Introduction to Galaxy and RNA-SeqIntroduction to Galaxy and RNA-Seq
Introduction to Galaxy and RNA-Seq
Enis Afgan3.3K views
Journey into Reactive Streams and Akka Streams by Kevin Webber
Journey into Reactive Streams and Akka StreamsJourney into Reactive Streams and Akka Streams
Journey into Reactive Streams and Akka Streams
Kevin Webber1.6K views
Cloud Architect Alliance #15: Openstack by Microsoft
Cloud Architect Alliance #15: OpenstackCloud Architect Alliance #15: Openstack
Cloud Architect Alliance #15: Openstack
Microsoft1.4K views
CERN IT Monitoring by Tim Bell
CERN IT Monitoring CERN IT Monitoring
CERN IT Monitoring
Tim Bell719 views
Apache Big Data EU 2016: Building Streaming Applications with Apache Apex by Apache Apex
Apache Big Data EU 2016: Building Streaming Applications with Apache ApexApache Big Data EU 2016: Building Streaming Applications with Apache Apex
Apache Big Data EU 2016: Building Streaming Applications with Apache Apex
Apache Apex686 views
Springone2gx 2014 Reactive Streams and Reactor by Stéphane Maldini
Springone2gx 2014 Reactive Streams and ReactorSpringone2gx 2014 Reactive Streams and Reactor
Springone2gx 2014 Reactive Streams and Reactor
Stéphane Maldini5.7K views

More from Roland Kuhn

Taming Distribution: Formal Protocols for Akka Typed by
Taming Distribution: Formal Protocols for Akka TypedTaming Distribution: Formal Protocols for Akka Typed
Taming Distribution: Formal Protocols for Akka TypedRoland Kuhn
1.2K views36 slides
Distributed systems vs compositionality by
Distributed systems vs compositionalityDistributed systems vs compositionality
Distributed systems vs compositionalityRoland Kuhn
2.9K views37 slides
Reactive Design Patterns — J on the Beach by
Reactive Design Patterns — J on the BeachReactive Design Patterns — J on the Beach
Reactive Design Patterns — J on the BeachRoland Kuhn
3.3K views36 slides
The Newest in Session Types by
The Newest in Session TypesThe Newest in Session Types
The Newest in Session TypesRoland Kuhn
3.9K views24 slides
Akka Typed — between Session Types and the Actor Model by
Akka Typed — between Session Types and the Actor ModelAkka Typed — between Session Types and the Actor Model
Akka Typed — between Session Types and the Actor ModelRoland Kuhn
6.7K views31 slides
Project Gålbma – Actors vs Types by
Project Gålbma – Actors vs TypesProject Gålbma – Actors vs Types
Project Gålbma – Actors vs TypesRoland Kuhn
2.7K views49 slides

More from Roland Kuhn(10)

Taming Distribution: Formal Protocols for Akka Typed by Roland Kuhn
Taming Distribution: Formal Protocols for Akka TypedTaming Distribution: Formal Protocols for Akka Typed
Taming Distribution: Formal Protocols for Akka Typed
Roland Kuhn1.2K views
Distributed systems vs compositionality by Roland Kuhn
Distributed systems vs compositionalityDistributed systems vs compositionality
Distributed systems vs compositionality
Roland Kuhn2.9K views
Reactive Design Patterns — J on the Beach by Roland Kuhn
Reactive Design Patterns — J on the BeachReactive Design Patterns — J on the Beach
Reactive Design Patterns — J on the Beach
Roland Kuhn3.3K views
The Newest in Session Types by Roland Kuhn
The Newest in Session TypesThe Newest in Session Types
The Newest in Session Types
Roland Kuhn3.9K views
Akka Typed — between Session Types and the Actor Model by Roland Kuhn
Akka Typed — between Session Types and the Actor ModelAkka Typed — between Session Types and the Actor Model
Akka Typed — between Session Types and the Actor Model
Roland Kuhn6.7K views
Project Gålbma – Actors vs Types by Roland Kuhn
Project Gålbma – Actors vs TypesProject Gålbma – Actors vs Types
Project Gålbma – Actors vs Types
Roland Kuhn2.7K views
Akka and AngularJS – Reactive Applications in Practice by Roland Kuhn
Akka and AngularJS – Reactive Applications in PracticeAkka and AngularJS – Reactive Applications in Practice
Akka and AngularJS – Reactive Applications in Practice
Roland Kuhn17.3K views
Go Reactive: Blueprint for Future Applications by Roland Kuhn
Go Reactive: Blueprint for Future ApplicationsGo Reactive: Blueprint for Future Applications
Go Reactive: Blueprint for Future Applications
Roland Kuhn3.1K views
Akka cluster overview at 010dev by Roland Kuhn
Akka cluster overview at 010devAkka cluster overview at 010dev
Akka cluster overview at 010dev
Roland Kuhn6.8K views
Akka typed-channels by Roland Kuhn
Akka typed-channelsAkka typed-channels
Akka typed-channels
Roland Kuhn2.9K views

Recently uploaded

.NET Deserialization Attacks by
.NET Deserialization Attacks.NET Deserialization Attacks
.NET Deserialization AttacksDharmalingam Ganesan
7 views50 slides
Introduction to Maven by
Introduction to MavenIntroduction to Maven
Introduction to MavenJohn Valentino
7 views10 slides
Supercharging your Python Development Environment with VS Code and Dev Contai... by
Supercharging your Python Development Environment with VS Code and Dev Contai...Supercharging your Python Development Environment with VS Code and Dev Contai...
Supercharging your Python Development Environment with VS Code and Dev Contai...Dawn Wages
5 views51 slides
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P... by
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...NimaTorabi2
17 views17 slides
Using Qt under LGPL-3.0 by
Using Qt under LGPL-3.0Using Qt under LGPL-3.0
Using Qt under LGPL-3.0Burkhard Stubert
14 views11 slides
Quality Engineer: A Day in the Life by
Quality Engineer: A Day in the LifeQuality Engineer: A Day in the Life
Quality Engineer: A Day in the LifeJohn Valentino
10 views18 slides

Recently uploaded(20)

Supercharging your Python Development Environment with VS Code and Dev Contai... by Dawn Wages
Supercharging your Python Development Environment with VS Code and Dev Contai...Supercharging your Python Development Environment with VS Code and Dev Contai...
Supercharging your Python Development Environment with VS Code and Dev Contai...
Dawn Wages5 views
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P... by NimaTorabi2
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...
NimaTorabi217 views
Quality Engineer: A Day in the Life by John Valentino
Quality Engineer: A Day in the LifeQuality Engineer: A Day in the Life
Quality Engineer: A Day in the Life
John Valentino10 views
Electronic AWB - Electronic Air Waybill by Freightoscope
Electronic AWB - Electronic Air Waybill Electronic AWB - Electronic Air Waybill
Electronic AWB - Electronic Air Waybill
Freightoscope 6 views
Top-5-production-devconMunich-2023-v2.pptx by Tier1 app
Top-5-production-devconMunich-2023-v2.pptxTop-5-production-devconMunich-2023-v2.pptx
Top-5-production-devconMunich-2023-v2.pptx
Tier1 app9 views
FOSSLight Community Day 2023-11-30 by Shane Coughlan
FOSSLight Community Day 2023-11-30FOSSLight Community Day 2023-11-30
FOSSLight Community Day 2023-11-30
Shane Coughlan8 views
ADDO_2022_CICID_Tom_Halpin.pdf by TomHalpin9
ADDO_2022_CICID_Tom_Halpin.pdfADDO_2022_CICID_Tom_Halpin.pdf
ADDO_2022_CICID_Tom_Halpin.pdf
TomHalpin96 views
How Workforce Management Software Empowers SMEs | TraQSuite by TraQSuite
How Workforce Management Software Empowers SMEs | TraQSuiteHow Workforce Management Software Empowers SMEs | TraQSuite
How Workforce Management Software Empowers SMEs | TraQSuite
TraQSuite7 views
Dapr Unleashed: Accelerating Microservice Development by Miroslav Janeski
Dapr Unleashed: Accelerating Microservice DevelopmentDapr Unleashed: Accelerating Microservice Development
Dapr Unleashed: Accelerating Microservice Development
Miroslav Janeski16 views
predicting-m3-devopsconMunich-2023.pptx by Tier1 app
predicting-m3-devopsconMunich-2023.pptxpredicting-m3-devopsconMunich-2023.pptx
predicting-m3-devopsconMunich-2023.pptx
Tier1 app10 views
Introduction to Git Source Control by John Valentino
Introduction to Git Source ControlIntroduction to Git Source Control
Introduction to Git Source Control
John Valentino8 views
predicting-m3-devopsconMunich-2023-v2.pptx by Tier1 app
predicting-m3-devopsconMunich-2023-v2.pptxpredicting-m3-devopsconMunich-2023-v2.pptx
predicting-m3-devopsconMunich-2023-v2.pptx
Tier1 app14 views
Streamlining Your Business Operations with Enterprise Application Integration... by Flexsin
Streamlining Your Business Operations with Enterprise Application Integration...Streamlining Your Business Operations with Enterprise Application Integration...
Streamlining Your Business Operations with Enterprise Application Integration...
Flexsin 5 views

Reactive Streams: Handling Data-Flow the Reactive Way

  • 1. Dr. Roland Kuhn Akka Tech Lead @rolandkuhn Reactive Streams Handling Data-Flows the Reactive Way
  • 3. What is a Stream? • ephemeral flow of data • focused on describing transformation • possibly unbounded in size 3
  • 4. Common uses of Streams • bulk data transfer • real-time data sources • batch processing of large data sets • monitoring and analytics 4
  • 5. What is special about Reactive Streams?
  • 6. Reactive  Applications The Four Reactive Traits 6 http://reactivemanifesto.org/
  • 7. Needed: Asynchrony • Resilience demands it: • encapsulation • isolation • Scalability demands it: • distribution across nodes • distribution across cores 7
  • 8. Many Kinds of Async Boundaries 8
  • 9. Many Kinds of Async Boundaries • between different applications 8
  • 10. Many Kinds of Async Boundaries • between different applications • between network nodes 8
  • 11. Many Kinds of Async Boundaries • between different applications • between network nodes • between CPUs 8
  • 12. Many Kinds of Async Boundaries • between different applications • between network nodes • between CPUs • between threads 8
  • 13. Many Kinds of Async Boundaries • between different applications • between network nodes • between CPUs • between threads • between actors 8
  • 14. The Problem: ! Getting Data across an Async Boundary
  • 16. Possible Solutions • the Traditional way: blocking calls 10
  • 18. Possible Solutions ! • the Push way: buffering and/or dropping 11
  • 20. Possible Solutions ! ! • the Reactive way:
 non-blocking & non-dropping & bounded 12
  • 21. How do we achieve that?
  • 22. Supply and Demand • data items flow downstream • demand flows upstream • data items flow only when there is demand • recipient is in control of incoming data rate • data in flight is bounded by signaled demand 14 Publisher Subscriber data demand
  • 23. Dynamic Push–Pull • “push” behavior when consumer is faster • “pull” behavior when producer is faster • switches automatically between these • batching demand allows batching data 15 Publisher Subscriber data demand
  • 24. Explicit Demand: Tailored Flow Control 16 demand data splitting the data means merging the demand
  • 25. Explicit Demand: Tailored Flow Control 17 merging the data means splitting the demand
  • 26. Reactive Streams • asynchronous non-blocking data flow • asynchronous non-blocking demand flow • minimal coordination and contention • message passing allows for distribution • across applications • across nodes • across CPUs • across threads • across actors 18
  • 28. What is a Collection? 20
  • 29. What is a Collection? • Oxford Dictionary: • “a group of things or people” 20
  • 30. What is a Collection? • Oxford Dictionary: • “a group of things or people” • wikipedia: • “a grouping of some variable number of data items” 20
  • 31. What is a Collection? • Oxford Dictionary: • “a group of things or people” • wikipedia: • “a grouping of some variable number of data items” • backbone.js: • “collections are simply an ordered set of models” 20
  • 32. What is a Collection? • Oxford Dictionary: • “a group of things or people” • wikipedia: • “a grouping of some variable number of data items” • backbone.js: • “collections are simply an ordered set of models” • java.util.Collection: • definite size, provides an iterator, query membership 20
  • 33. User Expectations • an Iterator is expected to visit all elements
 (especially with immutable collections) • x.head + x.tail == x • the contents does not depend on who is processing the collection • the contents does not depend on when the processing happens
 (especially with immutable collections) 21
  • 34. Streams have Unexpected Properties • the observed sequence depends on • … when the observer subscribed to the stream • … whether the observer can process fast enough • … whether the streams flows fast enough 22
  • 35. Streams are not Collections! • java.util.stream:
 Stream is not derived from Collection
 “Streams differ from Coll’s in several ways” • no storage • functional in nature • laziness seeking • possibly unbounded • consumable 23
  • 36. Streams are not Collections! • a collection can be streamed • a stream observer can create a collection • … but saying that a Stream is just a lazy Collection evokes the wrong associations 24
  • 37. So, Reactive Streams: why not just java.util.stream.Stream?
  • 38. Java 8 Stream 26 import java.util.stream.*; ! // get some stream final Stream<Integer> s = Stream.of(1, 2, 3); // describe transformation final Stream<String> s2 = s.map(i -> "a" + i); // make a pull collection s2.iterator(); // or alternatively push it somewhere s2.forEach(i -> System.out.println(i)); // (need to pick one, Stream is consumable)
  • 39. Java 8 Stream • provides a DSL for describing transformation • introduces staged computation
 (but does not allow reuse) • prescribes an eager model of execution • offers either push or pull, chosen statically 27
  • 41. RxJava 29 import rx.Observable; import rx.Observable.*; ! // get some stream source final Observable<Integer> obs = range(1, 3); // describe transformation final Observable<String> obs2 = obs.map(i -> "b" + i); // and use it twice obs2.subscribe(i -> System.out.println(i)); obs2.filter(i -> i.equals("b2")) .subscribe(i -> System.out.println(i));
  • 42. RxJava • implements pure “push” model • includes extensive DSL for transformations • only allows blocking for back pressure • currently uses unbounded buffering for crossing an async boundary • work on distributed Observables sparked participation in Reactive Streams 30
  • 44. Participants • Engineers from • Netflix • Oracle • Pivotal • Red Hat • Twitter • Typesafe • Individuals like Doug Lea and Todd Montgomery 32
  • 45. The Motivation • all participants had the same basic problem • all are building tools for their community • a common solution benefits everybody • interoperability to make best use of efforts • e.g. use Reactor data store driver with Akka transformation pipeline and Rx monitoring to drive a vert.x REST API (purely made up, at this point) 33 see also Jon Brisbin’s post on “Tribalism as a Force for Good”
  • 46. Recipe for Success • minimal interfaces • rigorous specification of semantics • full TCK for verification of implementation • complete freedom for many idiomatic APIs 34
  • 47. The Meat 35 trait Publisher[T] { def subscribe(sub: Subscriber[T]): Unit } trait Subscription { def requestMore(n: Int): Unit def cancel(): Unit } trait Subscriber[T] { def onSubscribe(s: Subscription): Unit def onNext(elem: T): Unit def onError(thr: Throwable): Unit def onComplete(): Unit }
  • 48. The Sauce • all calls on Subscriber must dispatch async • all calls on Subscription must not block • Publisher is just there to create Subscriptions 36
  • 49. How does it Connect? 37 SubscriberPublisher subscribe onSubscribeSubscription requestMore onNext
  • 51. Akka Streams • powered by Akka Actors • execution • distribution • resilience • type-safe streaming through Actors with bounded buffering 39
  • 52. Basic Akka Example 40 implicit val system = ActorSystem("Sys") val mat = FlowMaterializer(...) ! Flow(text.split("s").toVector). map(word => word.toUpperCase). foreach(tranformed => println(tranformed)). onComplete(mat) { case Success(_) => system.shutdown() case Failure(e) => println("Failure: " + e.getMessage) system.shutdown() }
  • 53. More Advanced Akka Example 41 val s = Source.fromFile("log.txt", "utf-8") Flow(s.getLines()). groupBy { case LogLevelPattern(level) => level case other => "OTHER" }. foreach { case (level, producer) => val out = new PrintWriter(level+".txt") Flow(producer). foreach(line => out.println(line)). onComplete(mat)(_ => Try(out.close())) }. onComplete(mat)(_ => Try(s.close()))
  • 54. Java 8 Example 42 final ActorSystem system = ActorSystem.create("Sys"); final MaterializerSettings settings = MaterializerSettings.create(); final FlowMaterializer materializer = FlowMaterializer.create(settings, system); ! final String[] lookup = { "a", "b", "c", "d", "e", "f" }; final Iterable<Integer> input = Arrays.asList(0, 1, 2, 3, 4, 5); ! Flow.create(input).drop(2).take(3). // leave 2, 3, 4 map(elem -> lookup[elem]). // translate to "c","d","e" filter(elem -> !elem.equals("c")). // filter out the "c" grouped(2). // make into a list mapConcat(list -> list). // flatten the list fold("", (acc, elem) -> acc + elem). // accumulate into "de" foreach(elem -> System.out.println(elem)). // print it consume(materializer);
  • 55. Akka TCP Echo Server 43 val future = IO(StreamTcp) ? Bind(addr, ...) future.onComplete { case Success(server: TcpServerBinding) => println("listen at "+server.localAddress) ! Flow(server.connectionStream).foreach{ c => println("client from "+c.remoteAddress) c.inputStream.produceTo(c.outputStream) }.consume(mat) ! case Failure(ex) => println("cannot bind: "+ex) system.shutdown() }
  • 56. Akka HTTP Sketch (NOT REAL CODE) 44 val future = IO(Http) ? RequestChannelSetup(...) future.onSuccess { case ch: HttpRequestChannel => val p = ch.processor[Int] val body = Flow(new File(...)).toProducer(mat) Flow(HttpRequest(method = POST, uri = "http://ex.amp.le/42", entity = body) -> 42) .produceTo(mat, p) Flow(p).map { case (resp, token) => val out = FileSink(new File(...)) Flow(resp.entity.data).produceTo(mat, out) }.consume(mat) }
  • 58. Current State • Early Preview is available:
 "org.reactivestreams" % "reactive-streams-spi" % "0.2"
 "com.typesafe.akka" %% "akka-stream-experimental" % "0.3" • check out the Activator template
 "Akka Streams with Scala!"
 (https://github.com/typesafehub/activator-akka-stream-scala) 46
  • 59. Next Steps • we work towards inclusion in future JDK • try it out and give feedback! • http://reactive-streams.org/ • https://github.com/reactive-streams 47
  • 60. ©Typesafe 2014 – All Rights Reserved