Being Functional on Reactive Streams with Spring Reactor

Being FunctionalBeing Functional
on
Reactive StreamsReactive Streams
with
Spring ReactorSpring Reactor

Source code is available on
https://github.com/maxxhuang/functional-reactive-stream-with-spring-reactor
GitHub

More Functional with Java 8More Functional with Java 8
Optional
Java 8 Stream
CompletableFuture

java.util.Optional<T>java.util.Optional<T>

Assume we are building a service
on the imaginary banking model
public class User {
private String id;
private String name;
...
}
public class Account {
private String accountNumber;
private String userId;
private double balance;
...
}
public class AccountInfo {
private String userId;
private String userName;
private String accountNumber;
private double balance;
...
}

and with Java 8 Optional, we have
the repositories
public class UserRepository {
public Optional<User> get(String userId) {
return Optional.ofNullable(FakeData.users.get(userId));
}
}
public class AccountRepository {
public Optional<Account> get(String accountNumber) {
return FakeData.accounts.get(accountNumber).stream().findFirst();
}
}

Now we are building a service for
querying account details combining
user information
public class AccountService {
private UserRepository userRepository;
private AccountRepository accountRepository;
...
public Optional<AccountInfo> getAccountInfo(String accountNumber) {
Optional<Account> optAccount = this.accountRepository.get(accountNumber);
if (!optAccount.isPresent()) {
return Optional.empty();
}
Account account = optAccount.get();
Optional<User> optUser = this.userRepository.get(account.getUserId());
if (!optUser.isPresent()) {
}
User user = optUser.get();
return Optional.of(AccountInfo.create(user, account));
}
}

We are treating Optional as a data structure
containing a possible existing value
That's nothing diﬀerent from using null as
an indication of "absence of value"
Optional<Account> optAccount =
this.accountRepository.get(accountNumber);
}
Optional<User> optUser =
this.userRepository.get(account.getUserId());
}
Account account =
if (account == null) {
return null;
}
User user =
if (user == null) {
return null;
}
return AccountInfo.create(user, account);

Can we combine 2 Optional values
in a more functional way?
Optional<Account> optAccount =
}
Optional<User> optUser =
}
public Optional flatMap(
Function<? super T, Optional> mapper) {
requireNonNull(mapper);
if (!isPresent())
return empty();
else {
return requireNonNull(mapper.apply(value));
}
}
public Optional map(
Function<? super T, ? extends U> mapper) {
if (!isPresent())
return empty();
else {
return Optional.ofNullable(mapper.apply(value));
}
}
Let's check out
Optional.ﬂatMap and Optional.map

public class Optional<T> {
public Optional map(Function<? super T, ? extends U> mapper) {
if (!isPresent())
return empty();
else {
return Optional.ofNullable(mapper.apply(value));
}
}
}
Optional.map takes a function of (T => U) to
transform the value of type "T" into something
else of type "U"
Optional.map creates an Optional out
of an Optional<T>

public class Optional<T> {
public Optional flatMap(Function<? super T, Optional> mapper) {
if (!isPresent())
return empty();
else {
return requireNonNull(mapper.apply(value));
}
}
}
Optional.ﬂatMap takes a mapper function of (T => Optional)

This mapper function, with the access to the actual value of
Optional<T>, creates another Optional

Optional.flatMap is said to be more powerful than
Optional.map
flatMap is able to compose other Optional objects and come
out with a new Optional object

flatMap has the ability of flow control. It can decide whether
or not to create a new Optional object or simply an empty
Optional object depending on the given access to previous
Optional value.
Something about flatMap

Attempt to compose Optional using map
Optional<Account> optAccount = accountRepository.get(accountNumber);
Optional<Optional<AccountInfo>> result = optAccount.map(account -> {
Optional<User> optUser = userRepository.get(account.getUserIdI());
Optional<AccountInfo> optAccountInfo =
optUser.map(user -> AccountInfo.create(user, account);
return optAccountInfo
});
Optional.map is unable to compose 2 Optional objects and peel
oﬀ or ﬂatten the outer Optional.

Optional.map just accumulates the layers of nested Optional.
It is not easy to access the value of multi-layer Optonal, e.g.
Optinoal<Optional<AccountInfo>>

Attempt to compose Optional using flatMap
Optional<AccountInfo> result = optAccount.flatMap(account -> {
optUser.flatMap(user -> Optional.of(AccountInfo.create(user, account));
});
With flatMap, we are capable of composing
Optional<Account> and Optional<User> to
obtain a "flattened" Optional<AccountInfo>

Use ﬂatMap for Flow Control
if (account.isCredential()) {
}
});

ﬂatMap and map work together
});
Don't bother to wrap AccountInfo with an Optional in the last ﬂatMap composition.
We can use "map" to achieve the same result
optUser.map(user -> AccountInfo.create(user, account);
});

Composing 2 or more Optionals
Optional<TransactionInfo> result = accountRepository.get(accountNumber).flatMap(account ->
userRepository.get(account.getUserId()).flatMap(user ->
transactionRepository.get(account.getAccountNumber()).map(transaction -> {
// use "account", "user", "transaction" to create TransactionInfo
...
return TransactionInfo.create(user, account, transaction);
})
)
);
Optional<SomeType> result = optional1.flatMap(v1 ->
optional2.flatMap(v2 ->
optional3.flatMap(v3 ->
...
optionaln.map(vn ->
// do something with v1, v2, v3,..., vn
//return some value of some type
return SomeType.create(...);
)
)
)
);

java.util.stream.Stream<T>java.util.stream.Stream<T>

Banking Repositories with Java 8 Stream
public List<User> get(String userId) {
return FakeData.users.containsKey(userId) ?
Collections.singletonList(FakeData.users.get(userId)) :
Collections.emptyList();
}
}
public List<Account> get(String accountNumber) {
return FakeData.accounts.get(accountNumber);
}
}

AccountService
...
public List<AccountInfo> getAccountInfo(String accountNumber) {
List<AccountInfo> infos = new ArrayList<>();
for (Account account : this.accountRepository.get(accountNumber)) {
for (User user : this.userRepository.get(account.getUserId())) {
infos.add(AccountInfo.create(user, account));
}
}
return infos;
}
}

AccountService being functional
public class FunctionalAccountService {
...
public List<AccountInfo> getAccountInfo(String accountNumber) {
return this.accountRepository.get(accountNumber).stream().flatMap(account ->
this.userRepository.get(account.getUserId()).stream().map(user ->
AccountInfo.create(user, account)))
.collect(Collectors.toList());
}
}

java.util.concurrent.CompletableFuture<T>java.util.concurrent.CompletableFuture<T>

Banking Repositories with CompletableFuture
public class UserRepository extends BaseRepository {
public UserRepository() {
super(defaultExecutorService());
}
public CompletableFuture<User> get(String userId) {
return CompletableFuture.supplyAsync(
() -> FakeData.users.get(userId),
this.executorService
);
}
}
public class AccountRepository extends BaseRepository {
public AccountRepository() {
super(defaultExecutorService());
}
public CompletableFuture<Account> get(String accountNumber) {
return CompletableFuture.supplyAsync(
() -> {
List<Account> accounts = FakeData.accounts.get(accountNumber);
return accounts.isEmpty() ? null : accounts.get(0);
},
this.executorService);
}
}

AccountService
...
public CompletableFuture<AccountInfo> getAccountInfo(String accountNumber) {
CompletableFuture<AccountInfo> result = new CompletableFuture<>();
try {
CompletableFuture<Account> accountFuture =
Account account = accountFuture.get();
if (account == null) result.complete(null);
CompletableFuture<User> userFuture = this.userRepository.get(account.getUserId());
User user = userFuture.get();
if (user == null) result.complete(null);
else result.complete(AccountInfo.create(user, account));
} catch (Exception e) {
result.completeExceptionally(e);
}
return result;
}
}

map and flatMap in disguise
CompletableFuture does not have map and flatMap

CompletableFuture defines thenApply and thenCompose
with similar sematics:
thenApply -> map
thenCompose -> flatMap
public CompletableFuture<T> implements Future<T>, CompletionStage<
// map
public CompletableFuture thenApply(
Function<? super T,? extends U> fn) {
...
}
// flatMap
public CompletableFuture thenCompose(
Function<? super T, ? extends CompletionStage> fn) {
...
}
}

Threading Control for
CompletableFuture.map/ﬂatMap
There are variant thenApply and thenCompose to control
the threading policy for the execution of mapping functions
public CompletableFuture<T> implements Future<T>, CompletionStage<T> {
public CompletableFuture thenApply(
Function<? super T,? extends U> fn) {...}
public CompletableFuture thenApplyAsync(
Function<? super T,? extends U> fn) {...}
public CompletableFuture thenApplyAsync(
Function<? super T,? extends U> fn, Executor executor) {...}
Function<? super T, ? extends CompletionStage> fn) {...}
public CompletableFuture thenComposeAsync(
Function<? super T, ? extends CompletionStage> fn) {...}
public CompletableFuture thenComposeAsync(
Function<? super T, ? extends CompletionStage> fn,
Executor executor) {...}
}

AccountService with brighter future
...
public CompletableFuture<AccountInfo> getAccountInfo(String accountNumber) {
return this.accountRepository.get(accountNumber).thenCompose(account ->
this.userRepository.get(account.getUserId()).thenApply(user ->
AccountInfo.create(user, account)));
}
}

Why are Optional, Stream, CompletableFuture
able to chain the computations?

What's the pattern?
public final class Optional<T> {
public Optional flatMap(Function<? super T, Optional> mapper)
public Optional map(Function<? super T, ? extends U> mapper)
}
public interface Stream<T> extends BaseStream<T, Stream<T>> {
public Stream flatMap(Function<? super T, ? extends Stream<? extends U>> mapper)
public Stream map(Function<? super T, ? extends U> mapper)
}
public CompletableFuture<T> implements Future<T>, CompletionStage<T> {
// flatMap
Function<? super T, ? extends CompletionStage> fn)
// map
public CompletableFuture thenApply(Function<? super T,? extends U> fn)
}

Monad
// This is psudo code illustrating Monad structure
M[T] {
flatMap(mapper: T => M[U]): M[U]
// unit wraps a value into monad context
//
// in Optional, it is Optional.of(...), Optional.ofNullable(...)
//
// in Stream, it is Stream.of(...)
//
// in CompletableFuture, it is CompletableFuture.supplyAsync(...)
// or {new CompletableFuture().complete(...)}
unit(value: T): M[T]
}
What about map?
map can always be implemented with ﬂatMap and unit. This
proves again ﬂatMap is more powerful than map.
map(mapper: T => U) {
return flatMap(v -> unit(mapper(v)))
}

Monad
A great article explaining Monad in plain English
http://blog.leichunfeng.com/blog/2015/11/08/functor-applicative-and-monad
A monad is a computational context for some value with a
"unit" method and a "ﬂatMap" method

Context value of M[T] is passes as parameter of (T => M[U]).
In the mapper function, you get to access the context value
and decide the new monad value to return.
accountRepository.get(accountNumber).flatMap(account -> {
if (account.isCredential()) {
}
});

Use flatMap to implement other combinators
JDK implementation for Optional.filter
public Optional<T> filter(Predicate<? super T> predicate) {
Objects.requireNonNull(predicate);
if (!isPresent())
return this;
else
return predicate.test(value) ? this : empty();
}
another implementation with flatMap
public Optional<T> filter(Predicate<? super T> predicate) {
return flatMap(v -> predicate.test(v) ? this : Optional.empty();
}
the emptiness test (isPresent()) has been done in flatMap

Monad Comparision
Type Context Value Composition Eﬀect (ﬂatMap)
Optional a value may or may not exist composition happens if Optional
contains value
composition stops if Optional is
empty and subsequent
compositions are ignored
CompletableFuture a value available in the future composition happens if
CompletableFuture obtains a
value without error ???
composition stops if error
occurred in CompletableFuture
and subsequent compositions
are ignored ???
Stream each value in the stream composition happens if the
stream is not empty
composition stops if Stream is
empty and subsequent
compositions are ignored
The resulting sub-streams are
merged to a joint stream

Reactive StreamsReactive Streams

Streams are a series of elements emitted over time. The
series potentially has no beginning and no end.
Kevin Webber, A Journey into Reactive Streams
Reactive Stream goes a step further by being able to
signal demand from downstream thus controlling the
overall speed of data elements ﬂowing through it
Walter Chang, Reactive Streams in Scala
What is Reactive Streams

Who deﬁnes Reactive Streams?
What is it?
How is it related to Java 9 Flow
https://en.wikipedia.org/wiki/Reactive_Streams

The scope of Reactive Streams is to find a minimal set of interfaces, methods and
protocols that will describe the necessary operations and entities to achieve the goal—
asynchronous streams of data with non-blocking back pressure.
www.reactive-streams.org
So there are only 4 interfaces and and 7 methods in total inside reactive-streams-1.0.2.jar
package org.reactivestreams;
public interface Publisher<T> {
public void subscribe(Subscriber<? super T> s);
}
public interface Subscriber<T> {
public void onSubscribe(Subscription s);
public void onNext(T t);
public void onError(Throwable t);
public void onComplete();
}
public interface Subscription {
public void request(long n);
public void cancel();
}
public interface Processor<T, R> extends Subscriber<T>, Publisher<R> {}
End-user DSLs or protocol binding APIs have purposefully been left out of the scope to encourage and
enable different implementations that potentially use different programming languages to stay as true as
possible to the idioms of their platform.
www.reactive-streams.org

Reactive Streams is a speciﬁcation for library developers.

Reactive-stream libraries complying with Reactive Steams speciﬁcation are
capable of
1. back-pressure control
2. interoperate with other libraries.

For example, a Reactor Processor can subscribe to RxJava Producer given that
Spring Reactor and RxJava are both Reactive Streams compliant.

Dynamic Push / Pull
request n elements
push at most n elements
Subscriber PublisherSubscription
Requests can be made asynchronously.
Multiple requests are accumulated on Publisher and will be served later
Fast subscriber demands more elements; publisher don't need to wait for
requests => push mode
Slow subscriber request less; publish waits for requests => pull mode
The dynamic push/pull makes back-pressure possible and ensure all
participating components are resilient to massive load although this may
degrade the performance

Let's fake it till we make it
public interface Publisher<T> {
public void subscribe(Subscriber<? super T> s);
}
abstract public class Flowie<T> implements Publisher<T> {
public static <T> Flowie<T> fromIterable(Iterable<T> iterable) {
...
}
}
"Flowie", a homemade implementation for Reactive Streams
Spring Reactor has "Flux"; RxJava comes with "Flowable".
Why don't we create one of our own?

public class FlowieIterable<T> extends Flowie<T> {
private Iterable<T> iterable;
public FlowieIterable(Iterable<T> iterable) {
this.iterable = iterable;
}
@Override
public void subscribe(Subscriber<? super T> subscriber) {
try {
Iterator<T> iterator = this.iterable.iterator();
subscriber.onSubscribe(new IterableSubscription<>(subscriber, iterator));
subscriber.onError(e);
}
}
...
static class IterableSubscription<T> implements Subscription {
private Subscriber<? super T> subscriber;
private Iterator<? extends T> iterator;
private boolean cancelled = false;
private AtomicLong requested = new AtomicLong(0L);
@Override
public void request(long n) {
...
}
}
}
We take a similar approach to that for Spring Reactor
Put major logic in Subscription object.
This subscription object gets delivered to users via
onSubscribe()

/**
* pending request
*
* The updates to the pending request may come from other threads via {@link Subscription#request(lo
*
* Use AtomicLong to make sure the update and comparison of pending request counter is conducted ato
*
*/
private AtomicLong requested = new AtomicLong(0L);
@Override
public void request(long n) {
long updatedRequested = addRequestAndReturnCurrent(n);
if (updatedRequested == 0L) {
doEmit(n);
}
}
private void doEmit(long n) {
long emitted = 0L;
while (true) {
Great care is taken to make sure multiple asynchronous Subscription.request(n) are
accumulated and served atomically
often leverage java.util.concurrent.atomic components
use CAS (compare and set) to ensure counter is updated atomically at the right
time
Sometimes the price to maintain the correctness of request counter is messing up
the code making it hard to read and understand.

public class FlowieIterable<T> extends Flowie<T> {
private Iterable<T> iterable;
public FlowieIterable(Iterable<T> iterable) {
this.iterable = iterable;
}
@Override
public void subscribe(Subscriber<? super T> subscriber) {
try {
Iterator<T> iterator = this.iterable.iterator();
subscriber.onSubscribe(new IterableSubscription<>(subscriber, iterator));
subscriber.onError(e);
}
}
///////////////////////////////////////////////////////////////////////////
private Subscriber<? super T> subscriber;
private Iterator<? extends T> iterator;
/**
* pending request
*
* The updates to the pending request may come from other threads via {@link Subscription#request(lo
*
* Use AtomicLong to make sure the update and comparison of pending request counter is conducted ato
*
*/

Ya! We have the ﬁrst Flowie implementation
Flowie.fromIterable(Arrays.asList("a", "b", "c")).subscribe(new Subscriber<String>() {
@Override
public void onSubscribe(Subscription s) {
s.request(Long.MAX_VALUE);
}
@Override
public void onNext(String s) {
System.out.println(s);
}
@Override
public void onError(Throwable t) {
}
@Override
public void onComplete() {
}
});
After subscribing, Subscription.request() needs to be
invoked to trigger element emission
a request of Long.MAX_VALUE elements, by the rule of
3.17 (https://github.com/reactive-streams/reactive-streams-jvm/blob/v1.0.2/README.md#speciﬁcation), is
treated as requesting ALL elements from the publisher

Let's pursue the banking service and make it
more "Flowie"
public Flowie<User> get(String userId) {
List<User> user = FakeData.users.containsKey(userId) ?
Collections.singletonList(FakeData.users.get(userId)) :
Collections.emptyList();
return Flowie.fromIterable(user);
}
}
public Flowie<Account> get(String accountNumber) {
return Flowie.fromIterable(
FakeData.accounts.get(accountNumber)
);
}
}

AccountService in callback hell
public class AccountServiceInCallbackHell {
...
/**
* NOTE: This only works in a single-thread execution context.
* The implementation does not catch the timing of "completion" in onComplete() or onError().
* In a muti-thread environment, the elements (User, Account) might not be ready
* when this method returns.
*/
public Flowie<AccountInfo> getAccountInfo(String accountNumber) {
final List<AccountInfo> result = new ArrayList<>();
this.accountRepository.get(accountNumber).subscribe(new Subscriber<Account>() {
@Override
}
@Override
public void onNext(Account account) {
userRepository.get(account.getUserId()).subscribe(new Subscriber<User>() {
@Override
}
@Override
public void onNext(User user) {
result.add(AccountInfo.create(user, account));
}
...
});
}
...

We need a way out of this callback hell.

Can we make Flowie a monad?
This way, we can do monadic composition
of 2 Flowie values like this.
flowieAccuont.flatMap(account ->
flowieUser.map(user ->
// create AccountInfo out of account, user
accountInfo))

Start with the simple one: Flowie.map
abstract public class Flowie<T> implements Publisher<T> {
public <R> Flowie<R> map(Function<? super T, ? extends R> mapper) {
return new FlowieMap<>(this, mapper);
}
}
public class FlowieMap<T, R> extends Flowie<R> {
private Publisher<T> source;
private Function<? super T, ? extends R> mapper;
...
@Override
public void subscribe(Subscriber<? super R> s) {
this.source.subscribe(new MapSubscriber<>(s, this.mapper));
}
static class MapSubscriber<T, R> implements Subscriber<T>, Subscription {
private Subscriber<? super R> actualSubscriber;
private Function<? super T, ? extends R> mapper;
private Subscription upstreamSubscription;
2 levels of decorator pattern adoption
1st level is FlowMap decorating the actual Publisher
2nd level is MapSubscriber decorating the actual
Subscriber
MapSubscriber applies the mapper function in "onNext"

Flowie.flatMap
Compliant flatMap implementation for Reactive Streams
is complicated and needs to tackle the concurrent sub-
streams emissions and elements queuing.
For quick demonstration of functional programming
benefit, we rush a non-compliant implementation that
only works in a single-thread execution context.

Flowie.ﬂatMap Implementation FYI
/**
* This implementation DOES NOT comply with Reactive Streams. It does not take care of
* the situation where elements from sub-streams emitted asynchronously.
*
* The compliant implementation is complicated and usually needs one or more queues
* to store un-consumed elements emitted from sub-streams.
*
* This non-compliant implementation only serves the purpose of demonstrating the
* advantage of making reactive streams functional.
*/
public class FlowieNonCompliantSynchronousFlatMap<T, R> extends Flowie<R> {
private Publisher<T> source;
private Function<? super T, ? extends Publisher<? extends R>> mapper;
public FlowieNonCompliantSynchronousFlatMap(Publisher<T> source, Function<? super T, ? extend
this.source = source;
this.mapper = mapper;
}
@Override
public void subscribe(Subscriber<? super R> s) {
this.source.subscribe(new SynchronousNonThreadSafeFlapMapSubscriber<>(s, this.mapper));
}
///////////////////////////////////////////////////////////////////////////
static class SynchronousNonThreadSafeFlapMapSubscriber<T, R> implements Subscriber<T>, Subscr

With map/ﬂatMap in place, here is the
upgraded Flowie AccountService
...
public Flowie<AccountInfo> getAccountInfo(String accountNumber) {
return this.accountRepository.get(accountNumber).flatMap(account ->
this.userRepository.get(account.getUserId()).map(user ->
new AccountInfo(
user.getId(), user.getName(),
account.getAccountNumber(), account.getBalance())));
}
}

Project ReactorProject Reactor
(Spring Reactor)

Project Reactor implements Reactive Streams, inherently with non-
blocking streaming nature

On top of Reactive Streams, Project Reactor provides its own APIs for
functional streaming handling and adapting other data source.

Reactor oﬀers 2 Publisher implementations
Flux: a reactive stream of 0-N elements
Mono: a reactive stream of 0-1 elements

In addition to Reactive Streams, Reactor extensively
implements APIs deﬁned by Reactive Extensions (Rx)
(http://reactivex.io)

Spring WebFlux runs on top of Reactor non-blocking IO, if
Netty is chosen as the underlying web server among Tomcat,
Jetty, Undertow.

Spring WebClient is built on top of Reactor non-blocking IO

AccountService with Reactor
public Mono<User> get(String userId) {
return Mono.justOrEmpty(FakeData.users.get(userId));
}
}
public Flux<Account> get(String accountNumber) {
return Flux.fromIterable(
FakeData.accounts.get(accountNumber)
);
}
}
...
public FunctionalAccountService(UserRepository userRepository, AccountRepository accountRepository) {
this.userRepository = userRepository;
this.accountRepository = accountRepository;
}
public Flux<AccountInfo> getAccountInfo(String accountNumber) {
return this.accountRepository.get(accountNumber).flatMap(account ->
this.userRepository.get(account.getUserId()).map(user ->
new AccountInfo(
user.getId(), user.getName(),
account.getAccountNumber(), account.getBalance())));
}
}

Mono/Flux.flatMap
3 flavours of flatMap
flatMap
flatMapSequential
concatMap
Generation of inners
and subscription
Ordering of the
flattened values
Interleaving
flatMap Eagerly subscribing to
inners
No Yes
flatMapSequential Eagerly subscribing to
inners
Yes (elements from
late inners are
queued)
No
concatMap subscribing to inners
one by one
Yes No

Threading Control
In Reactor, the execution model and where the execution happens is
determined by the Scheduler that is used. A Scheduler has scheduling
responsibilities similar to an ExecutorService

Default Schedulers

Schedulers.elastic()
An elastic thread pool (Schedulers.elastic()). It creates new worker pools as needed, and reuse idle
ones.

Schedulers.parallel()
a ﬁxed pool of workers that is tuned for parallel work (Schedulers.parallel()). It creates as many
workers as you have CPU cores.

Schedulers.immediate()
the current thread
Create new instances
Schedulers.newElastic
Schedulers.newParallel
Schedulers.newSingle
Schedulers.fromExecutor

Flux/Mono.publishOn
publishOn takes signals from upstream and replays them
downstream while executing the callback on a worker from
the associated Scheduler.
https://projectreactor.io/docs/core/release/reference/#schedulers

Flux/Mono.subscribeOn
https://projectreactor.io/docs/core/release/reference/#schedulers
subscribeOn applies to the subscription process, when that
backward chain is constructed.
As a consequence, no matter where you place the
subscribeOn in the chain, it always aﬀects the context of the
source emission.

Getting closer to Project Reactor
Reactor 3 Reference Guide
https://projectreactor.io/docs/core/release/reference/
Reactor 3 Javadoc
https://projectreactor.io/docs/core/release/api/
methods are illustrated with diagrams

Put it all together
A device simulator that simulator speciﬁed
number of devices.

Each device periodically reports
heartbeat
stats
to a device controller

First create a stream of device MACs
List<String> deviceMacs = new MacGenerator("AA", "BB").generate(deviceCount);
Flux<String> deviceMacFlux = Flux.fromIterable();
Flux<DeviceRequest> createDeviceStatsStream(String deviceMac) {
...
}
Assume we have a method that creates a stream of reporting
requests for a single device.

The stream is a sum of heartbeat/stats request stream
Flux<DeviceRequest> requestFlux = deviceMacFlux.flatMap(mac -> createDeviceRequestStream(mac));
How do we turn a stream of device MAC into a stream of device request stream and
combine each single device request stream into a single massive one?

Yes... ﬂatMap

Dive deeper to implement (deviceMac => Flux<DeviceRequest>)
A device request stream is composed of
heatbeat request stream
stats request stream
private Flux<DeviceRequest> createDeviceRequestStream(String deviceMac) {
return Flux.merge(
createDeviceHeartbeatStream(deviceMac),
createDeviceStatsStream(deviceMac)
);
}
Now what's left is the terminal streams,
heartbeat/stats request streams

heartbeat/stats request streams emit elements in a periodical manner
We need a source stream that generates elements in a ﬁxed time interval.
Then each element is transformed to a DeviceRequest.

Flux.interval(Duration) is what we need
private Flux<DeviceRequest> createDeviceHeartbeatStream(String deviceMac) {
return Flux.interval(Duration.ofSeconds(this.heartbeatIntervalInSeconds))
.map(n -> // create a DeviceRequest out of the given deviceMac);
}
private Flux<DeviceRequest> createDeviceStatsStream(String deviceMac) {
return Flux.interval(Duration.ofSeconds(this.statsIntervalInSeconds))
.map(n -> // create a DeviceRequest out of the given deviceMac);,

List<String> deviceMacs = this.macGenerator.generate(this.deviceCount);
Flux.fromIterable(deviceMacs).flatMap(mac -> createDeviceRequestStream(mac))
.subscribe(request -> /* sending reporting request to device controller via http */ );
Flux<DeviceRequest> createDeviceRequestStream(String deviceMac) {
return Flux.merge(
createDeviceHeartbeatStream(deviceMac),
createDeviceStatsStream(deviceMac)
);
}
Flux<DeviceRequest> createDeviceHeartbeatStream(String deviceMac) {
return Flux.interval(Duration.ofSeconds(this.heartbeatIntervalInSeconds))
.map(n -> new DeviceRequest(
deviceMac,
System.currentTimeMillis(),
ReportType.heartbeat,
this.urlBuilder.url(ReportType.heartbeat, deviceMac)));
}
Flux<DeviceRequest> createDeviceStatsStream(String deviceMac) {
return Flux.interval(Duration.ofSeconds(this.statsIntervalInSeconds))
.map(n -> new DeviceRequest(
deviceMac,
System.currentTimeMillis(),
ReportType.stats,
this.urlBuilder.url(ReportType.stats, deviceMac)));
}

The problem with Flux.flatMap
flatMap eagerly subscribes to inners with configurable value with the default as 256
public static final int SMALL_BUFFER_SIZE = Math.max(16,
Integer.parseInt(System.getProperty("reactor.bufferSize.small", "256")));
We observed that only 256 devices reported heartbeat/stats
even we specified more than 256 devices.
Why?
Each device request stream is an infinite stream, and flatMap
eagerly subscribes first 256 inner streams.

Each device request steam is an infinite stream
(Flux.interval()). That means only first 256 device streams get
to be created and emit data.

More on flatMap
flatMap exposes 2 configurable parameters for eager inner
stream subscription
maxConcurrency
prefetch
flatMap eagerly subscribes "maxConcurrentcy" inner streams when
subscribed (onSubscribe)

When subscribing inner stream, it will also pre-fetch "prefetch" elements by
invoking Subscription.request(prefetch) of inner stream.
To fix this problem, we explicitly configure "maxConcurrency" when flatMapping
Flux.fromIterable(deviceMacs).flatMap(mac -> createDeviceRequestStream(mac), deviceMacs.size())

Being Functional on Reactive Streams with Spring Reactor

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Being Functional on Reactive Streams with Spring Reactor

Similar to Being Functional on Reactive Streams with Spring Reactor (20)

Recently uploaded

Recently uploaded (20)

Being Functional on Reactive Streams with Spring Reactor