The document discusses the concept of containerizing distributed pipes, highlighting the use of Apache Kafka for stream processing and its integration with Docker containers. It covers various aspects of Kafka, including its architecture, message processing, and tool application stack, while also referencing the UNIX philosophy for stream processing. The summary concludes with a recap of the challenges faced in implementing Kafka streams and the need for durable storage and capacity planning.

1. Containerising
Distributed Pipes
a story about convenience
Hagen Tönnies
www.linkedin.com/in/hagen-toennies
1

2. Agenda
• Introduction
• A Distributed Pipe
• Tool Application Stack
• Recap
2

3. Unix Pipe Recap
• […]In Unix-like computer operating systems, a
pipeline is a sequence of processes chained
together by their standard streams, so that the
output of each process (stdout) feeds directly as
input (stdin) to the next one.
3

4. Unix Philosophy
• Write programs that do one thing and do it well.
• Write programs to work together.
• Write programs to handle text streams,
because that is a universal interface.
Peter H. Salus says about the unix philosophy
4

5. Apache Kafka
5

6. Apache Kafka Cluster
Kafka Broker
6
Zookeeper

7. Apache Kafka Topic
oldernewer
7
topic

8. Apache Kafka Partitions
8

9. Apache Kafka Replications
9

10. Apache Kafka Distribute
Partitions
10

11. Apache Kafka Producer
oldernewer
Producer
11

12. Apache Kafka Consumer
Consumer
oldernewer
12

13. Apache Kafka Consumer
Groups
oldernewer
Consumer1Consumer1Group1
Consumer1Consumer1Group2
13

14. Kafka Streaming API
14
Low-Level High-Level
• Topology Builder
• Stream and Table
abstractions
• Custom Aggregators • Simple Transformation
• Custom Processors
• Simple Joins of Tables
and Streams

15. Kafka Streaming API
15
Table Stream (change-log)
alice | 1
alice | 1
charlie | 1
alice | 2
charlie | 1
(„alice“ , 1)
(„charlie, 1)
(„alice, 2)

16. Kafka featuring Unix
(split %1 "s")
16

17. Kafka featuring Unix
(split %1 "s")
kafka
Message Broker Stream processing job
17

18. Kafka featuring Unix
(sketch %1)(split %1 "s") (store %1)(agg-by-key %1)
18

19. Containers
• […] Operating-system-level virtualization is a
server virtualization method in which the kernel of
an operating system allows the existence of
multiple isolated user-space instances
19

20. Containers
• docker run -t -i CONTAINER-NAME …args
• java -jar …args
20

21. Our Distributed Pipe
Application Stack in Docker
ZooKeeper
kafka
Discovering and configuring services in your infrastructure.
Distributed append log a.k.a Message Broker
enables highly reliable distributed coordination.
Provides a distributed full-text search engine
21

22. consul:
image: qnib/alpn-consul
networks:
- network
cpu_shares: 4
mem_limit: 1g
environment:
- DC_NAME=es
ports:
- “8500:8500"
22

23. zookeeper:
image: qnib/zookeeper
extends:
file: base.yml
ports:
- "2181:2181"
ZooKeeper
23

24. kafka-broker:
image: kafka:0.10.0.1
extends:
file: base.yml
service: gaikai
volumes:
- /tmp/kafka-logs
ports:
- "9092:9092"
kafka
24

25. elasticsearch:
image: elasticsearch:1.7
command: "elasticsearch -Des.cluster.name=dp-es -Dnetwork.bind_host=0.0.0.0"
environment:
- ES_HEAP_SIZE=2g
ports:
- "9200:9200"
- "9300:9300"
25

26. 26

27. Stream Processing Recap
27
Power
Simplicity
kafka

28. 28
kafka
kafka streams
Stream Processing Recap

29. • clj-kstream-cutter
• clj-kstream-hh
• clj-kstream-string-long-window-aggregate
• clj-kstream-elasticsearch-sink
https://github.com/sojoner
Our Distributed Tool
Application Stack in Docker
29

30. https://xkcd.com/297/
30

31. clj-kstream-cutter
Edmilson Alves 0 Edmilson Alves -LRB- born February
17 , 1976 -RRB- , is a Brazilian midfielder who currently
plays for Roasso Kumamoto in the J. League Division 2 .
[ Edmilson, Alves, 0, Edmilson, Alves, LRB, born …]
31
Input:
Output:

32. [ Edmilson, Alves, 0, Edmilson, Alves, LRB, born, …]
Edmilson ~10 Alves ~8
clj-kstream-hh
32
Input:
Output:

33. 33
Count Min (CM) sketch

34. 34
CM sketch retrieval

35. And ~10
Bob ~7
Alice ~5
Foo ~3
Bar ~2
take top_n
retrieve sketched value
Heavy Hitter for t_1
35
Heavy Hitter

36. (defn- heavy-hitter-processor
"Main stream processor takes a configuration and a mapper function to apply."
[conf]
(let [streamBuilder (-> (new TopologyBuilder)
(.addSource (:name conf) string_dser string_dser (into-array [(:input-topic conf)]))
(.addProcessor "HeavyHitter"
(reify ProcessorSupplier
(get [this]
(get-processor)))
(into-array [(:name conf)]))
(.addStateStore
(->> (Stores/create storeName)
(.withStringKeys)
(.withLongValues)
(.inMemory)
(.build))
(into-array ["HeavyHitter"]))
(.addSink
"Sink"
(:output-topic conf)
string_ser
string_ser
(into-array ["HeavyHitter"])))]
(.start
(KafkaStreams.
streamBuilder
(get-props conf)))))
36
clj-kstream-hh Topology

37. (defn ^Processor get-processor []
(reify org.apache.kafka.streams.processor.Processor
(init [this context]
(.schedule (:context @application-state) (:time-window @application-state))
(swap! hh/state assoc
:top-n 5
:number-of-hashfn 10N
:bucket-size 1000N)
(reset! hh/hitter ^(priority-map))
(reset! hh/min-sketch (make-array Integer/TYPE 10N 1000N)) …)
(process [this key value]
(debug "Process (k,v)::" key value)
(hh/sketch-value value)
(hh/add-to-hitter value) …)
(punctuate [this timestamp …)
(close [this]
(.close (:store @application-state)))))
37
clj-kstream-hh Processor

38. 38
clj-kstream-hh Container
clj-kstream-hh:
image: sojoner/clj-kstream-hh:0.1.0
hostname: clj-kstream-hh
container_name: clj-kstream-hh
extends:
file: base.yml
service: sojoner
command: "--broker kafka-broker:9092 --input-topic mapped-test-json
--output-topic heavy-hitters --window-size 1 --name stream-hh"

39. Input:
Output:
Alves ~8 Alves ~10 Edmilson ~5 Edmilson~3
Alves ~18 Edmilson ~8
(key, value)
39
clj-kstream-string-long-
window-aggregate

40. clj-kstream-elasticsearch-sink
Input:
Output:
{
“name“: “Alves“,
“count“: 18,
“time“: “January 26th 2017, 17:03:00.000”
}
Alves ~18
40

41. Our Distributed Tool
Application Stack in Docker
Edmilson Alves 0 Edmilson Alves -LRB- born February 17 , 1976 -RRB- , is
a Brazilian midfielder who currently plays for Roasso Kumamoto in the J.
League Division 2 .
[ Edmilson, Alves, 0, Edmilson, Alves, LRB, born …]
Alves ~10 Alves ~8
{“name“: “Alves“, “count“: 18, “time“: “January 26th 2017, 17:03:00.000”}
Alves 18
41

42. Our Distributed Tool
Application Stack in Docker
42

43. From the development
setup…
$ export DOCKER_HOST=tcp://my.desktop.de:2576
43

44. …to a datacenter setup
$ export DOCKER_HOST=tcp://my.datacenter.de:2576
Build a Docker Swarm
44

45. Current Challenges
• Kafka Streams still at least once
• Persistent and durable Storage
• Still need capacity planning
• Testing / Debugging is still a challenge
• Consistency of the state storage
• Processing Time vs. Event Time
• How about Amdahl’s law
45

46. Recap
(split %1 "s")
46
as Message broker as Stream Processor
(split %1 "s")

47. Containerising
Distributed Pipes
a story about convenience
(thanks (listening [this]))
47

48. Sources 1
• http://kafka.apache.org/
• https://martin.kleppmann.com/2015/05/06/data-agility-at-strata.html
• https://speakerdeck.com/ept/kafka-and-samza-distributed-stream-
processing-in-practice
• https://github.com/mhausenblas/dnpipes
• https://en.wikipedia.org/wiki/Pipeline_%28Unix%29
• https://zookeeper.apache.org/doc/trunk/zookeeperOver.html
• https://github.com/sojoner/container-stacks/tree/master/
kafkaelasticsearch
48

49. Sources 2
• https://kafka.apache.org/documentation/streams#streams_processor
• https://kafka.apache.org/documentation/streams#streams_dsl
• https://hub.docker.com/r/sojoner/clj-kstream-elasticsearch-sink/
• https://hub.docker.com/r/sojoner/clj-kstream-cutter/
• https://hub.docker.com/r/sojoner/clj-kstream-hh/
• https://hub.docker.com/r/sojoner/clj-kstream-string-long-window-
aggregate/
• https://blog.acolyer.org/2016/07/21/time-adaptive-sketches-ada-
sketches-for-summarizing-data-streams/
49