This document summarizes a presentation about Alpakka, a Reactive Enterprise Integration library for Java and Scala based on Reactive Streams and Akka Streams. Alpakka provides connectors to various data sources and messaging systems that allow them to be accessed and processed using Akka Streams. Examples of connectors discussed include Kafka, MQTT, JMS, Elasticsearch and various cloud platforms. The document also provides an overview of Akka Streams and how they allow building responsive, asynchronous and resilient data processing pipelines.

1. Let the Alpakka
pull your stream
Enno Runne
Java Forum Nord 2019-09-24

2. Reactive Principles
The Reactive Manifesto discusses how we can
build responsive systems.

3. Akka Streams
Akka Streams is library to model and run high-
performance, non-blocking data flows supporting
back-pressure, with concise APIs for Java and
Scala.

4. Alpakka
Alpakka is a Reactive Enterprise Integration
library for Java and Scala, based on Reactive
Streams and Akka.
The short version: “Endpoints for Akka Streams”

5. • Alpakka Tech Lead @ Lightbend
• Stockholm, Sweden
Enno Runne

6. Reactive?
Why do we talk about

8. Responsive: The system responds in a timely
manner if at all possible. Responsiveness is the
cornerstone of usability and utility, but more than
that, responsiveness means that problems may
be detected quickly and dealt with effectively.
Responsive systems focus on providing rapid
and consistent response times, establishing
reliable upper bounds so they deliver a
consistent quality of service. This consistent
behaviour in turn simplifies error handling, builds
end user confidence, and encourages further
interaction.
https://www.reactivemanifesto.org/

11. Customer support

12. Customer support via chat
Message-driven architecture
• enables resource utilisation
• is easier to scale
• allows for distribution

13. Lightbend Reactive Architecture
A free, online learning experience by Lightbend and IBM
https://lightbend.com/learn/lightbend-reactive-architecture

14. Akka?
How does streaming relate to

15. Build powerful reactive, concurrent,
and distributed applications more easily
Akka
Akka Actors
(typed and classic)
Akka Cluster &
Sharding
Akka Persistence &
Distributed Data
Akka Streams
Akka HTTP
Akka gRPC
Alpakka
(reactive integrations)
Akka Management
Akka Enhancements
(persistence & resilience)

16. Extends Reactive Streams to be used
• for local file access
• over TCP as TCP supports flow control
• for HTTP with Akka HTTP
• for gRPC with Akka gRPC
Akka Streams
Akka Streams
Akka HTTP
Akka gRPC
Alpakka
(reactive integrations)
• Domain Specific Language to describe streams
• Mechanics to map these stream to be executed on top of actors
• Full support to connect to any Reactive Streams compliant library

17. When to use what abstraction
modelling power
complexity
actors
streams
(completable)
futures
java.concurrency

18. Data flow
Step 1 Step 4Step 2 Step 3
Buffering data
• May cure the immediate issue
• All buffering is limited

19. Dynamic push/pull
• Push when downstream is faster
• Pull when upstream is faster
Downstream
Source SinkFlow Flow
Streaming with back-pressure

20. Reactive Streams
reactive-streams.org

21. Part of the JDK since version 9 (java.util.concurrent.Flow)
http://reactive-streams.org/
Compliant libraries allow full interoperability.
Reactive Streams
A standard for asynchronous
stream processing with non-
blocking back-pressure.

22. What is a
stream?
Photo by blucolt - originally posted to Flickr as spearfish creek, CC BY-SA 2.0
50,000 / s
10 / s
1 / month

23. Inter-system
messages with Kafka

24. De-couple systems
Buffering (Queues)
Distribution
Multiple consumers to same data
“Inter-system message-driven”

25. Alpakka Kafka with Akka Streams
Consumer.DrainingControl<Done> control =
Consumer
.sourceWithOffsetContext(consumerSettings, Subscriptions.topics(topic))
.mapAsync(
parallelism: 1,
record -> business(record.key(), record.value())
)
.toMat(Committer.sinkWithOffsetContext(committerSettings), Keep.both())
.mapMaterializedValue(Consumer::createDrainingControl)
.run(materializer);

26. Alpakka Kafka with Akka Streams
Consumer.DrainingControl<Done> control =
Consumer
.sourceWithOffsetContext(consumerSettings, Subscriptions.topics(topic))
.mapAsync(
parallelism: 1,
record -> business(record.key(), record.value())
)
.toMat(Committer.sinkWithOffsetContext(committerSettings), Keep.both())
.mapMaterializedValue(Consumer::createDrainingControl)
.run(materializer);
Source
ConsumerRecord<String, String>

27. Alpakka Kafka with Akka Streams
Consumer.DrainingControl<Done> control =
Consumer
.sourceWithOffsetContext(consumerSettings, Subscriptions.topics(topic))
.mapAsync(
parallelism: 1,
record -> business(record.key(), record.value())
)
.toMat(Committer.sinkWithOffsetContext(committerSettings), Keep.both())
.mapMaterializedValue(Consumer::createDrainingControl)
.run(materializer);
Sink

28. Alpakka Kafka with Akka Streams
Consumer.DrainingControl<Done> control =
Consumer
.sourceWithOffsetContext(consumerSettings, Subscriptions.topics(topic))
.mapAsync(
parallelism: 1,
record -> business(record.key(), record.value())
)
.toMat(Committer.sinkWithOffsetContext(committerSettings), Keep.both())
.mapMaterializedValue(Consumer::createDrainingControl)
.run(materializer);

29. Alpakka Kafka with Akka Streams
Consumer.DrainingControl<Done> control =
Consumer
.sourceWithOffsetContext(consumerSettings, Subscriptions.topics(topic))
.mapAsync(
parallelism: 1,
record -> business(record.key(), record.value())
)
.toMat(Committer.sinkWithOffsetContext(committerSettings), Keep.both())
.mapMaterializedValue(Consumer::createDrainingControl)
.run(materializer);
Flow
ConsumerRecord<String, String> ?

30. Collecting data
via MQTT

31. Collect measurements
Light-weight
Different quality of service
The messaging protocol of the IoT world.
(Eclipse Paho)

32. Alpakka MQTT with Akka Streams
CompletionStage<Done> result =
MqttSource.atLeastOnce(
connectionSettings,
MqttSubscriptions.create(topic, MqttQoS.atLeastOnce()),
bufferSize)
.mapAsync(
parallelism: 1,
msg -> business(msg)
)
.mapAsync(1, MqttMessageWithAck::ack)
.runWith(Sink.ignore(), materializer);

33. Alpakka MQTT with Akka Streams
CompletionStage<Done> result =
MqttSource.atLeastOnce(
connectionSettings,
MqttSubscriptions.create(topic, MqttQoS.atLeastOnce()),
bufferSize)
.mapAsync(1,.mapAsync(
parallelism: 1,
msg -> business(msg)
)
.mapAsync(1, MqttMessageWithAck::ack)
.runWith(Sink.ignore(), materializer);
Source
MqttMessageWithAck

34. Alpakka MQTT with Akka Streams
CompletionStage<Done> result =
MqttSource.atLeastOnce(
connectionSettings,
MqttSubscriptions.create(topic, MqttQoS.atLeastOnce()),
bufferSize)
.mapAsync(
parallelism: 1,
msg -> business(msg)
)
.mapAsync(1, MqttMessageWithAck::ack)
.runWith(Sink.ignore(), materializer);
Sink
MqttMessageWithAck

35. Alpakka MQTT with Akka Streams
CompletionStage<Done> result =
MqttSource.atLeastOnce(
connectionSettings,
MqttSubscriptions.create(topic, MqttQoS.atLeastOnce()),
bufferSize)
.mapAsync(
parallelism: 1,
msg -> business(msg)
)
.mapAsync(1, MqttMessageWithAck::ack)
.runWith(Sink.ignore(), materializer);Flow
MqttMessageWithAck MqttMessageWithAck

36. Listening to
JMS topics

37. Alpakka JMS with Akka Streams
JmsConsumerControl control =
JmsConsumer
.ackSource(consumerSettings)
.mapAsync(
parallelism: 1,
msg -> business(msg)
)
.toMat(Sink.foreach(AckEnvelope::acknowledge), Keep.left())
.run(materializer);

38. Alpakka JMS with Akka Streams
JmsConsumerControl control =
JmsConsumer
.ackSource(consumerSettings)
.mapAsync(
parallelism: 1,
msg -> business(msg)
)
.toMat(Sink.foreach(AckEnvelope::acknowledge), Keep.left())
.run(materializer);
Source
AckEnvelope

39. Alpakka JMS with Akka Streams
JmsConsumerControl control =
JmsConsumer
.ackSource(consumerSettings)
.mapAsync(
parallelism: 1,
msg -> business(msg)
)
.toMat(Sink.foreach(AckEnvelope::acknowledge), Keep.left())
.run(materializer);
Sink
AckEnvelope

40. Alpakka JMS with Akka Streams
JmsConsumerControl control =
JmsConsumer
.ackSource(consumerSettings)
.mapAsync(
parallelism: 1,
msg -> business(msg)
)
.toMat(Sink.foreach(AckEnvelope::acknowledge), Keep.left())
.run(materializer);
Flow
AckEnvelope AckEnvelope

41. Akka Streams
The power of

42. Source
T
Parts of a Stream
Sink
S
A B
Flow

43. A B
Stream operators
map
A B
mapAsync
A B
via
Flow
CompletionStage

44. A Collection<A>
Stream operators
groupedWithin
A A
idleTimeout
A A
throttle
{
{
{
{

45. Composability

46. Composability
Flow<ConsumerRecord<String, String>,
ConsumerRecord<String, String>,
NotUsed> flow =
Flow.<ConsumerRecord<String, String>>create()
.mapAsync(1, msg -> business(msg));
Flow<MqttMessageWithAck,
MqttMessageWithAck,
NotUsed> flow =
Flow.<MqttMessageWithAck>create()
.mapAsync(1, msg -> business(msg));
A B
via
Flow

47. Asynchronous
Threads
Memory
Network
DB Connections
Akka Streams & resources
Backpressure
Multi-core
Multi-processor
Resource utilisation Resource bounds
enables
to make use of
enables to respect
to not exhaust

48. Akka Streams: Message order
1234

49. Akka Streams: Concurrent execution
1
2
3
4

50. Akka Streams: At-least-once delivery
1123

52. Alpakka
Alpakka is a Reactive Enterprise Integration
library for Java and Scala, based on Reactive
Streams and Akka.
The short version: “Endpoints for Akka Streams”

53. Alpakka connectors for cloud services
Amazon DynamoDB
Amazon Kinesis data
streams & firehose
AWS Lambda
Amazon S3
Amazon SNS
Amazon SQS
Google Cloud Pub/Sub
+ gRPC-based
Google Firebase Cloud
Messaging
Google Cloud Storage
Azure Storage Queue

54. Alpakka connectors for data stores
Elasticsearch

55. Alpakka connectors for messaging
(Eclipse Paho &
Akka native)
AMQP
(RabbitMQ)
IronMQ
JMS
Java Messaging Service
… not as fancy logos, but very well suited for the streaming approach.

56. Community connectors to Akka Streams
Apache Camel
Eventuate
FS2
Pulsar
… if you know of more, please tell us.

58. Code, Issues, Pull Requests @ Github
• https://github.com/akka/alpakka
• https://github.com/akka/alpakka-kafka
Questions, Discussions
• https://discuss.akka.io
Alpakka community

59. Thank you!
@ennru
enno.runne@lightbend.com
https://akka.io
@akkateam
Join the Alpakka community
at github.com/akka/alpakka