Systems integration is everywhere, not because we want it, but because we need it. It's the download of exchange rates, the list of yesterday's orders and the latest inventory. Not long time ago, we'd pull this kind of information in overnight batches and every system had something to work on. That was the age where we had printed newspapers. Today, data needs to be there. Instantaneously. Or "as fast as possible". We don't want to transfer huge piles of data once every night, but have the updates coming by - just after the change happened. We want streaming data. In this talk, we exemplify the path to move from overnight file exchanges to streaming data by using Alpakka, which is an integration library based on Reactive Streams and Akka.

1. From Overnight
to Always On
Enno Runne
Jfokus 2019-02-06

2. Akka Streams
Akka Streams is library to model and run high-
performance, non-blocking data flows supporting
back-pressure, with concise APIs for Java and
Scala.

3. Alpakka
Alpakka is a Reactive Enterprise Integration
library for Java and Scala, based on Reactive
Streams and Akka.
The short version: “Endpoints for Akka Streams”

4. File content as
streams

5. Let’s call it source and sink
Disk
Read
from file
Disk
Write
to file
Source<Byte…> Sink<Byte…>

6. FileIO.fromPath(sourceFile) FileIO.toPath(targetFile)
File source and sink with Akka Streams
Disk
Read
from file
Disk
Write
to file
Source<ByteString, …> Sink<ByteString, …>

7. Copying a file with Akka Streams
Read
from file
Write
to file
CompletionStage<IOResult> handle =
FileIO.fromPath(sourceFile)
.runWith(
FileIO.toPath(targetFile),
materializer
);

8. Data flow
Step 1 Step 4Step 2 Step 3
Buffering data
• May cure the immediate issue
• All buffering is limited

9. Downstream
Source SinkFlow Flow
Streaming with back-pressure
Dynamic push/pull
• Push when downstream is faster
• Pull when upstream is faster

10. Overnight
Batching

11. Make our file-copy more useful
Disk
Read
from file
Disk
Write
to file
Detect
new file

12. Detect a new file in the directory
DirectoryChangesSource.create(
sourceDir,
pollingInterval,
maxChangesKept
)
Detect
new file
.filter(pathChange ->
DirectoryChange.Creation
== pathChange.second()
)
// Pair<Path, DirectoryChange>
.map(Pair::first);
Source<Path, NotUsed> newFileDetector =

13. We built our
own source!

14. Combine the source with a stream with a sink
newFileDetector
.mapAsync(8, p -> {
Path targetFile = targetDir.resolve(p.getFileName());
return createFileToFile(p, targetFile);
})
.runWith(Sink.ignore(), materializer);
Nesting a stream execution within a stream

15. Outer and inner flows
Disk
Read
from file
Disk
Write
to file
Detect
new file

16. Overcoming
file-based

17. Disk
Read
from file
Parse
CSV
bytes messages
Parse
JSON
Parse
XML
Overcoming file-based integration

18. Parse as CSV with Alpakka
Parse
CSV
lines
byteStringSource
.via(CsvParsing.lineScanner())
Convert
CSV lines
to maps
ByteString Collection<ByteString> Map<String, String>
.via(CsvToMap.toMapAsStrings(StandardCharsets.UTF_8));

19. Flow<ByteString, Map<String, String>, NotUsed> csvBytesToMap =
Flow.of(ByteString.class)
.via(CsvParsing.lineScanner())
.via(CsvToMap.toMapAsStrings(
StandardCharsets.UTF_8
));
Build your own flow

20. JsonNodeFactory jsonNodeFactory = JsonNodeFactory.instance;
FileIO.fromPath(p)
.via(csvBytesToMap)
.map(data -> { // Using raw Jackson to create JSON objects
ObjectNode objectNode = jsonNodeFactory.objectNode();
data.forEach(objectNode::put);
return objectNode;
})
Use your own flow and apply data mapping
Flow<ByteString, Map<String, String>, NotUsed> csvBytesToMap =
Flow.of(ByteString.class)
.via(CsvParsing.lineScanner())
.via(CsvToMap.toMapAsStrings(StandardCharsets.UTF_8));

21. Overcoming file-based integration
Disk
Read
from file
Parse
CSV
bytes JSONMap Map to
JSON

22. Reactive File Integration
Disk
Read
from file
Destination
Detect
new file
Turn
into
messages
Other
technology

24. Alpakka connectors for cloud services
Amazon DynamoDB
Amazon Kinesis data
streams & firehose
AWS Lambda
Amazon S3
Amazon SNS
Amazon SQS
Google Cloud Pub/
Sub
Google Firebase
Cloud Messaging
Azure Storage Queue

25. Alpakka connectors for data stores
Elasticsearch

26. Alpakka connectors for messaging
(Eclipse Paho
and Akka native)
AMQP
(RabbitMQ)
IronMQ
JMS
Java Messaging Service
Apache Kafka
… not as colourful logos, but very well suited for the streaming approach.

27. Thank you!
@ennru
enno.runne@lightbend.com
https://www.lightbend.com/alpakka
@akkateam
Enno Runne
• Alpakka Tech Lead @ Lightbend
• Stockholm, Sweden