Kafka is a distributed streaming platform whose main strength is the ability to serve as the single message hub for applications of massive scale.It relies on a topic-based publish-subscribe model, where each topic may have multiple partitions to be consumed/published from/into. Kafka diverges from regular message queues in a lot of its functionalities. A significant change from other standard message queues is that each consumer has an associated offset, representing the identifier of the last consumed message from the subscribed topic. This allows for replaying of messages in cases of failures, deployment issues, and other occurrences. How can fully functional applications interact with Kafka and its features while maintaining the characteristics of the functional domain? The Scala ecosystem has many stream-based frameworks that can be leveraged to use Kafka. Two of the most used are Akka-Streams and FS2. These frameworks imply different approaches for processing data streams and dealing with Kafka’s features. The goal of this talk is to provide insight into these differences, in terms of functionality, performance, and their impact in maintaining the code, keeping it functional, robust and readable.

1. Functional streams with
Kafka
A comparison between Akka-streams and
FS2
12th May 2017
Luis Reis & Rui Batista

3. Apache Kafka

4. Apache Kafka

5. Read
and Write
Offset
management
Parallelism
Which features?

6. val props = new Properties()
props.put("bootstrap.servers", "localhost:9092")
props.put("group.id", "consumer-tutorial")
props.put("key.deserializer", StringDeserializer.class.getName())
props.put("value.deserializer", StringDeserializer.class.getName())
consumer = new KafkaConsumer[String, String](props)
consumer.subscribe(List("foo", "bar").asJava)
//One thread
try {
while (true) {
val records = consumer.poll(Long.MAX_VALUE);
records.iterator.asScala foreach { record => println(record.value) }
}
} catch {
case e: WakeupException => // ignore for shutdown
} finally {
consumer.close()
}
//separate thread
consumer.wakeup()
Java API
val props = new Properties()
props.put("bootstrap.servers", "localhost:9092")
props.put("group.id", "consumer-tutorial")
props.put("key.deserializer", StringDeserializer.class.getName())
props.put("value.deserializer", StringDeserializer.class.getName())
consumer = new KafkaConsumer[String, String](props)
consumer.subscribe(List("foo", "bar").asJava)

7. Java API

8. Functional Streams
Akka
Streams
FS2
Monix
Kafka
Streams

9. Functional Streams
Akka Streams + FS2
Akka
Streams
Reactive-Kafka
FS2
FS2-Kafka
(by Rui)

10. Functional Streams
Akka Streams

11. Functional Streams
FS2
Stream[F, O]
F[_] --> fs2.Task || scalaz.concurrent.Task || cats.effect.IO
//A Function from Stream[F, I] to Stream[F, O]
Pipe[F, I, O] =:= (Stream[F, I] => Stream[F, O])
Sink[F, O] =:= Pipe[F, O, Unit]
Stream[Task, Int](1).map(_.toString).runLast.unsafeRun()

12. Read Write

13. Read Write
Akka Streams
val bootstrapServers = "localhost:9092"
val consumerGroup = "retweets"
val stringSerializer = new StringSerializer
val stringDeserializer = new StringDeserializer
val bootstrapServers = "localhost:9092"
val consumerGroup = "retweets"
val stringSerializer = new StringSerializer
val stringDeserializer = new StringDeserializer
val producerSettings = ProducerSettings(system, stringSerializer, stringSerializer)
.withBootstrapServers(bootstrapServers)
val consumerSettings = ConsumerSettings(system, stringDeserializer, stringDeserializer)
.withBootstrapServers(bootstrapServers)
.withGroupId(consumerGroup)
val regularSubscription = Subscriptions.topics("topic1")
val bootstrapServers = "localhost:9092"
val consumerGroup = "retweets"
val stringSerializer = new StringSerializer
val stringDeserializer = new StringDeserializer
val producerSettings = ProducerSettings(system, stringSerializer, stringSerializer)
.withBootstrapServers(bootstrapServers)
val consumerSettings = ConsumerSettings(system, stringDeserializer, stringDeserializer)
.withBootstrapServers(bootstrapServers)
.withGroupId(consumerGroup)
val regularSubscription = Subscriptions.topics("topic1")
Consumer.plainSource(consumerSettings, regularSubscription)
.map { message =>
parseTweet[Message](record, _.value())
}.map { tweet =>
val count = tweet.retweet_count.toString
new ProducerRecord[String, String]("topic2", count)
}.runWith(Producer.plainSink(producerSettings))

14. Read Write
FS2
val bootstrapServers = "localhost:9092"
val consumerGroup = "retweets"
val bootstrapServers = "localhost:9092"
val consumerGroup = "retweets"
val consumerSettings = ConsumerSettings[String, String](50 millis)
.withBootstrapServers(bootstrapServers)
.withGroupId(consumerGroup)
val producerSettings = ProducerSettings[String, String]()
.withBootstrapServers(bootstrapServers)
val subscription = Subscriptions.topics("topic1")
val bootstrapServers = "localhost:9092"
val consumerGroup = "retweets"
val consumerSettings = ConsumerSettings[String, String](50 millis)
.withBootstrapServers(bootstrapServers)
.withGroupId(consumerGroup)
val producerSettings = ProducerSettings[String, String]()
.withBootstrapServers(bootstrapServers)
val subscription = Subscriptions.topics("topic1")
// stream definition
val stream = Consumer[Task, String, String](consumerSettings)
.simpleStream
.plainMessages(subscription)
.map(msg => parseTweet[ConsumerRecord[String, String]](msg, _.value))
.map(_.retweet_count)
.map(count => new ProducerRecord[String, String]("topic2", "key", count.toString))
.to(Producer[Task, String, String](producerSettings).sendAsync)
// run at end of universe
stream.run.unsafeRun()

15. Offset Management

16. Offset Management
Akka Streams
//Automatically
Consumer.committableSource(consumerSettings, regularSubscription)
.map { message =>
(message, parseTweet[Message](message, _.record.value()))
}.map { case (message, tweet) =>
Util.messageWithOffset(message.committableOffset, tweet, "topic2")
}.runWith(Producer.commitableSink(producerSettings))
//Automatically
Consumer.committableSource(consumerSettings, regularSubscription)
.map { message =>
(message, parseTweet[Message](message, _.record.value()))
}.map { case (message, tweet) =>
Util.messageWithOffset(message.committableOffset, tweet, "topic2")
}.runWith(Producer.commitableSink(producerSettings))
//External offset storage
Source.fromFuture(offsetDB.loadOffset())
.flatMapConcat { offset =>
val subscription = Subscriptions.assignmentWithOffsets(
Map(new TopicPartition("topic1", 0) -> offset)
)
Consumer.committableSource(consumerSettings, subscription)
.mapAsync(producerSettings.parallelism) { record =>
val offset = record.committableOffset.partitionOffset.offset
offsetDB.save(offset + 1)
}
}.runWith(Sink.ignore)

17. Offset Management
FS2
//Automatically
Consumer[Task, String, String](consumerSettings)
.simpleStream
.commitableMessages(subscription)
.map(_.map(r => parseTweet[Message](r, _.value)))
// do stuff with tweet
.map { message =>
Util.messageWithOffset(message.committableOffset, tweet, "topic2")
}
.to(Producer[Task, String, String](producerSettings).sendCommitable)
//Automatically
Consumer[Task, String, String](consumerSettings)
.simpleStream
.commitableMessages(subscription)
.map(_.map(r => parseTweet[Message](r, _.value)))
// do stuff with tweet
.map { message =>
Util.messageWithOffset(message.committableOffset, tweet, "topic2")
}
.to(Producer[Task, String, String](producerSettings).sendCommitable)
//External offset storage
Stream.eval(offsetDB.loadOffset)
.flatMap { offset =>
val assignment = Subscriptions.assignmentWithOffsets(
Map(new TopicPartition("topic1", 0) -> offset)
)
Consumer[Task, String, String](consumerSettings)
.simpleStream
.commitableMessages(assignment)
// do stuff with message
.evalMap(msg => offsetDB.save(msg.commitableOffset.partitionOffset.offset + 1))
}

18. Parallelism

19. Parallelism
Akka Streams
//One source per partition
Consumer.committablePartitionedSource(consumerSettings, regularSubscription)
//One source per partition
Consumer.committablePartitionedSource(consumerSettings, regularSubscription)
.map {
case (topicPartition, source) =>
source
.via(logicFlow)
.map { flowResponse => createKafkaMessage(producerTopic, flowResponse) }
.runWith(Producer.committablePartitionedSink(producerSettings))
}
.mapAsyncUnordered(producerSettings.parallelism)(identity)
.runWith(Sink.ignore)

20. Parallelism
FS2
def parallelCount(keyFunc: KeyFunc[Task, Message],
signal: Signal[Task, Map[String, Int]])
def parallelCount(keyFunc: KeyFunc[Task, Message],
signal: Signal[Task, Map[String, Int]]) = {
val partitioned = Consumer[Task, String, String](consumerSettings)
.partitionedStreams
.commitableMessages(Subscriptions.topics("topic1"))
.map {
case (_, innerStream) =>
innerStream.evalMap(keyFunc)
}
//still needs to join streams
def parallelCount(keyFunc: KeyFunc[Task, Message],
signal: Signal[Task, Map[String, Int]]) = {
val partitioned = Consumer[Task, String, String](consumerSettings)
.partitionedStreams
.commitableMessages(Subscriptions.topics("topic1"))
.map {
case (_, innerStream) =>
innerStream.evalMap(keyFunc)
}
// join streams and aggregate key counts
fs2.concurrent.join(100)(partitioned)
.scan(Map.empty[String, Int]) { case (current, key) =>
current |+| Map(key -> 1)
}
.evalMap(signal.set _)
}

21. Functional Streams
API Design for FS2-Kafka
Typesafe
API
Resource
acquisition
State
management
Back
Pressure

22. trait Consumer[F[_], K, V]
FS2-Kafka
Typesafe API
trait Consumer[F[_], K, V] {
val simpleStream = new StreamType {
type OutStreamType[A] = Stream[F, A]
// ...
}
val partitionedStreams = new StreamType {
type OutStreamType[A] = Stream[F, (TopicPartition, Stream[F, A])]
// ...
}
}

23. trait Consumer[F[_], K, V] {
private[kafka] def createConsumer: F[ConsumerControl[F, K, V]]
trait StreamType {
type OutStreamType[_] <: Stream[F, _]
type CMessage = CommitableMessage[F, ConsumerRecord[K, V]]]
private[kafka] def makeStream(
subscription: Subscription,
builder: MessageBuilder[F, K, V]
)(implicit F: Async[F]): OutStreamType[builder.Message]
def commitableMessages(subscription: Subscription)
(implicit F: Async[F]): OutStreamType[CMessage]] =
makeStream(subscription, new CommitableMessageBuilder[F, K, V])
}
}
FS2-Kafka
Typesafe API

24. trait Producer[F[_], K, V] {
//...
type PMessage[P] = ProducerMessage[K, V, P]
def send[P](implicit F: Async[F]): Pipe[F, PMessage[P], ProducerMetadata[P]]
def sendAsync: Sink[F, ProducerRecord[K, V]]
def sendCommitable[P <: Commitable[F]](implicit F: Async[F]): Sink[F, PMessage[P]]
}
Resource Acquisition
def bracket[F[_],R,A](r: F[R])
(use: R => Stream[F,A],
release: R => F[Unit]): Stream[F,A]
Stream.bracket(createConsumer)({consumer => ???}, _.close)
FS2-Kafka
Typesafe API

25. type Record = ConsumerRecord[K, V]
//Mutable queues and references
openPartitions: Async.Ref[F, Map[TopicPartition, Queue[F, Option[Chunk[Record]]]]]
//Notify assigned partitions
openPartitionsQueue: Queue[F, (TopicPartition, Stream[F, Record])]
FS2-Kafka
State Management

26. FS2-Kafka
Back pressure

27. References
Apache Kafka
http://kafka.apache.org/
http://blog.cloudera.com/blog/2014/09/apache
-kafka-for-beginners/
http://nverma-tech-
blog.blogspot.pt/2015/10/apache-kafka-quick-
start-on-windows.html
FS2
https://github.com/ragb/fs2-kafka
https://github.com/functional-
streams-for-scala/fs2
Akka-Streams
http://akka.io/docs/
https://pt.slideshare.net/Lightben
d/understanding-akka-streams-
back-pressure-and-
asynchronous-architectures

28. Thank you !!!!!