Watch the webcast here: https://videos.confluent.io/watch/WwmbzUR7bpakR9NLwRhEAX? Speaker: Kenneth Cheung

4. Massive volumes of
new data generated
every day
Mobile Cloud Microservices Internet of
Things
Machine
Learning
Distributed across
apps, devices,
datacenters, clouds
Structured,
unstructured
polymorphic

5. ●
●
●
●
●

9. “Architectural Patterns with Apache
Kafka

10. RDBMS
CDC
Data Lakes

11. payment event
customer
customer payments
Stream
Processing
RDBMS CDC

12. customer
Stream
Processin
g
RDBMS
customer payments
payment event

13. Stream
Processing
RDBMS
Existing
App
New App
<x>

14. Stream
Processing
RDBMS
Existing
App
New App
<x>
New App
<y>

22. my-topic
my-topic-partition-0
my-topic-partition-1
my-topic-partition-2
broker-1
broker-2
broker-3

24. $ kafka-topics --zookeeper zk:2181
--create
--topic my-topic
--replication-factor 3
--partitions 3
AdminClient

26. C CC

32. ProducerProducer Record
Topic
[Partition]
[Timestamp]
Value
Serializer Partitioner
Topic A
Partition 0
Batch 0
Batch 1
Batch 2
Topic B
Partition 1
Batch 0
Batch 1
Batch 2
Kafka
Broker
Send()
Retry
?
Fail
?
Yes
No
Can’t retry,
throw exception
Success: return
metadata
Yes
[Headers]
[Key]

33. JSON
CSV
Avro
XML
SERIALIZERS
01001010 01010011 01001111 01001110
01000011 01010011 01010110
01001010 01010011 01001111 01001110
01010000 01110010 01101111 01110100 ...
01011000 01001101 01001100
(if you must)
Reference
https://kafka.apache.org/10/documentation/streams/developer-guide/datatypes.html
Protobuf

34. private Properties kafkaProps = new Properties();
kafkaProps.put(“bootstrap.servers”, “broker1:9092,broker2:9092”);
kafkaProps.put(“key.serializer”, “org.apache.kafka.common.serialization.StringSerializer”);
kafkaProps.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
producer = new KafkaProducer<String, SpecificRecord>(kafkaProps);
Reference
https://kafka.apache.org/10/documentation/streams/developer-guide/datatypes.html

35. Producer Record
Topic
[Partition]
[Key]
Value
partitioner

36. Producer Record
Topic
[Partition]
AAAA
Value
partition = hash(key) % numPartitions
partitioner

37. Producer Record
Topic
[Partition]
BBBB
Value
partition = hash(key) % numPartitions
partitioner

38. Producer Record
Topic
[Partition]
CCCC
Value
partition = hash(key) % numPartitions
partitioner

39. Producer Record
Topic
[Partition]
DDDD
Value
partition = hash(key) % numPartitions
partitioner

40. Car·di·nal·i·ty
/ˌkärdəˈnalədē/
Noun

42. final Consumer<String, String> consumer = new KafkaConsumer<String, String>(props);
consumer.subscribe(Arrays.asList(topic));
try {
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records) {
-- Do Some Work --
}
}
} finally {
consumer.close();
}
}

43. C

44. CC
C1
CC
C2

45. C C
C C

46. 0 1
2 3

47. 0 1
2 3

48. 0, 3 1
2 3

49. ●
●
●
●
●
●

56. Download your Apache Kafka and Stream Processing
O'Reilly Book Bundle
Download free here