Kafka is an event processing platform based on a distributed architecture. It handles millions of messages through distributed brokers and partitions across topics. Messages are immutable records written sequentially to topics and cannot be deleted or edited. Producers send messages to Kafka topics, while consumers subscribe to topics to receive messages. Common client libraries for Kafka include Java, Python, and Go.

1. A Lunchtime Introduction to Kafka

2. Wrote this….
Works with this amazing bunch.

3. The Rules Haven’t Changed……
You may heckle via Twitter…..
@jasonbelldata
It’s Friday, you might as well.
It’s in print, no one can argue…..

4. What is Kafka?

5. It’s an immutable log.

7. Think of it like this…..

8. My Kafka Log...

9. Opinionated thought
approaching.
#grumpy_yorkshireman

10. IT

11. IS

12. NOT

13. A

14. DATABASE

15. Kafka is an event processing platform which is based on a
distributed architecture.

16. Kafka is an event processing platform which is based on a
distributed architecture.
To producers and consumer subscribed to the system it
would appear as a standalone processing engine but
production systems are built on many machines.

17. Kafka is an event processing platform which is based on a
distributed architecture.
To producers and consumer subscribed to the system it
would appear as a standalone processing engine but
production systems are built on many machines.
It can handle millions of messages throughput, dependent on
physical disk and RAM on the machines, and is fault tolerant.

19. Messages are sent to topics which are written sequentially in
an immutable log. Kafka can support many topics and can be
replicated and partitioned.

20. Once the records are appended to the topic log
they can’t be deleted or amended.

21. (If the customer wants you to
“replay from the start”, well they can’t)

22. It’s a very simple data structure
where each message is byte encoded.

23. Each Message has:
A key
A value (payload)
A timestamp
A header (for meta data)

24. Producers and consumers can serialise and deserialise data
to various formats.
(I’ll talk about that later)

25. The Kafka Ecosystem

26. Image source: Confluent Inc.

27. The Kafka Cluster

28. Brokers

34. Partitions

37. Topics

38. Creating a Topic

39. bin/kafka-topics --create -–zookeeper localhost:2181 --replication-factor 4
--partitions 10 --topic testtopic --config min.insync.replicas=2

40. Listing Topics

41. bin/kafka-topics --zookeeper localhost:2181 --list

42. Deleting a Topic

43. $ bin/kafka-topics --zookeeper localhost:2181 --delete --topic
testtopic
Topic testtopic is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set
to true.

44. Topic Rentention

45. Rentention by Time

46. log.retention.hours
log.retention.minutes
log.retention.ms
The default is 168 hours (7 days) if not set.
Note: The smallest value setting take priority, be careful!

47. Rentention by Size

48. log.retention.bytes
Defines volume of log bytes retained. Applied per partition, so if set to 1GB
and you have 8 partitions, you are storing 8GB.

49. Client Libraries

50. C/C++ github.com/edenhill/librdkafka
Go github.com/confluentinc/confluent-kafka-go
Java Kafka Consumer and Kafka Producer
JMS JMS Client
.NET github.com/confluentinc/confluent-kafka-dotnet
Python github.com/confluentinc/confluent-kafka-python
Client Library Language Support

51. Client Libraries: Java and Go.
Java: Full support for all API features.
Go: Exactly once semantics
Kafka Streams
Schema Registry
Simplified Installation
Are NOT supported.

52. Producers

53. Producers
(Sends messages to the
Kafka Cluster)

54. import java.util.Properties;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
public class SimpleProducer {
public static void main(String[] args) throws Exception{
String topicName = “testtopic”;
Properties props = new Properties();
props.put("bootstrap.servers", “localhost:9092");
props.put("acks", “all");
props.put("retries", 0);
props.put("batch.size", 10);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer
<String, String>(props);
for(int i = 0; i < 10; i++)
producer.send(new ProducerRecord<String, String>(topicName,
Integer.toString(i), “This is my message: “ + Integer.toString(i)));
System.out.println(“Message sent successfully”);
producer.close();
}
}

55. import java.util.Properties;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
public class SimpleProducer {
public static void main(String[] args) throws Exception{
String topicName = “testtopic”;
Properties props = new Properties();
props.put("bootstrap.servers", “localhost:9092");
props.put("acks", “all");
props.put("retries", 0);
props.put("batch.size", 10);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer
<String, String>(props);
for(int i = 0; i < 10; i++)
producer.send(new ProducerRecord<String, String>(topicName,
Integer.toString(i), “This is my message: “ + Integer.toString(i)));
System.out.println(“Message sent successfully”);
producer.close();
}
}

56. props.put("acks", “all");
Value Action
-1/all Message has been received by the leader and followers.
0 Fire and Forget
1 Once the message is received by the leader.

58. public class ProducerWithCallback implements ProducerInterceptor{
private int onSendCount;
private int onAckCount;
private final Logger logger = LoggerFactory.getLogger(ProducerWithCallback.class);
@Override
public ProducerRecord onSend(final ProducerRecord record) {
onSendCount++;
System.out.println(String.format("onSend topic=%s key=%s value=%s %d n",
record.topic(), record.key(), record.value().toString(),
record.partition()));
return record;
}
@Override
public void onAcknowledgement(final RecordMetadata metadata, final Exception exception) {
onAckCount++;
System.out.println(String.format("onAck topic=%s, part=%d, offset=%dn",
metadata.topic(), metadata.partition(), metadata.offset()));
}
@Override
public void close() {
System.out.println("Total sent: " + onSendCount);
System.out.println("Total acks: " + onAckCount);
}
@Override
public void configure(Map<String,?> configs) {
}
}

59. import java.util.Properties;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
public class SimpleProducer {
public static void main(String[] args) throws Exception{
String topicName = “testtopic”;
Properties props = new Properties();
props.put("bootstrap.servers", “localhost:9092");
props.put("acks", “all");
props.put("retries", 0);
props.put("batch.size", 10);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer
<String, String>(props);
for(int i = 0; i < 10; i++)
producer.send(new ProducerRecord<String, String>(topicName,
Integer.toString(i), “This is my message: “ + Integer.toString(i)));
System.out.println(“Message sent successfully”);
producer.close();
}
}

60. //Only one in-flight messages per Kafka broker connection
// - max.in.flight.requests.per.connection (default 5)
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION,1);
//Set the number of retries - retries
props.put(ProducerConfig.RETRIES_CONFIG, 3);
//Request timeout - request.timeout.ms
props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 15000);
//Only retry after one second.
props.put(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, 1000);

61. import java.util.Properties;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
public class SimpleProducer {
public static void main(String[] args) throws Exception{
String topicName = “testtopic”;
Properties props = new Properties();
props.put("bootstrap.servers", “localhost:9092");
props.put("acks", “all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer
<String, String>(props);
for(int i = 0; i < 10; i++)
producer.send(new ProducerRecord<String, String>(topicName,
Integer.toString(i), “This is my message: “ + Integer.toString(i)));
System.out.println(“Message sent successfully”);
producer.close();
}
}

62. byte[] Serdes.ByteArray(), Serdes.Bytes()
ByteBuffer Serdes.ByteBuffer()
Double Serdes.Double()
Integer Serdes.Integer()
Long Serdes.Long()
String Serdes.String()
JSON JsonPOJOSerializer()/Deserializer()
Serialize/Deserialize Types

63. Consumers

64. public class SimpleConsumer {
public static void main(String[] args) throws Exception {
if(args.length == 0){
System.out.println("Enter topic name");
return;
}
//Kafka consumer configuration settings
String topicName = args[0].toString();
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test");
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("session.timeout.ms", "30000");
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer
<String, String>(props);
//Kafka Consumer subscribes list of topics here.
consumer.subscribe(Arrays.asList(topicName))
//print the topic name
System.out.println("Subscribed to topic " + topicName);
int i = 0;
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records)
// print the offset,key and value for the consumer records.
System.out.printf("offset = %d, key = %s, value = %sn",
record.offset(), record.key(), record.value());
}
}
}

65. Consumer Groups

67. Consumer Group Offsets

68. The offset is a simple integer number that is used
by Kafka to maintain the current position of a
consumer.

69. kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group mygroup
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
mygroup t1 0 1 3 2 test-consumer

70. GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
mygroup t1 0 1 12 11 test-consumer

71. Streaming API

72. public class WordCountLambdaExample {
static final String inputTopic = "streams-plaintext-input";
static final String outputTopic = "streams-wordcount-output";
/**
* The Streams application as a whole can be launched like any normal Java application that has a `main()` method.
*/
public static void main(final String[] args) {
final String bootstrapServers = args.length > 0 ? args[0] : "localhost:9092";
// Configure the Streams application.
final Properties streamsConfiguration = getStreamsConfiguration(bootstrapServers);
// Define the processing topology of the Streams application.
final StreamsBuilder builder = new StreamsBuilder();
createWordCountStream(builder);
final KafkaStreams streams = new KafkaStreams(builder.build(), streamsConfiguration);
streams.cleanUp();
streams.start();
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
}
static Properties getStreamsConfiguration(final String bootstrapServers) {
final Properties streamsConfiguration = new Properties();
streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "wordcount-lambda-example");
streamsConfiguration.put(StreamsConfig.CLIENT_ID_CONFIG, "wordcount-lambda-example-client");
streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 10 * 1000);
streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getAbsolutePath());
return streamsConfiguration;
}
static void createWordCountStream(final StreamsBuilder builder) {
final KStream<String, String> textLines = builder.stream(inputTopic);
final Pattern pattern = Pattern.compile("W+", Pattern.UNICODE_CHARACTER_CLASS);
final KTable<String, Long> wordCounts = textLines
.flatMapValues(value -> Arrays.asList(pattern.split(value.toLowerCase())))
.groupBy((keyIgnored, word) -> word)
.count();
wordCounts.toStream().to(outputTopic, Produced.with(Serdes.String(), Serdes.Long()));
}
}

73. https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/

74. Supports exactly once transactions! Yay!

75. I know what you’re thinking…...

77. processing.guarantee=exactly_once

78. What is Kafka Connect?

79. Provides a mechanism for
data sources and data
sinks.

80. Image source: Confluent Inc.

81. Data Source
Data from a thing (Database,
Twitter stream, Logstash etc)
going to Kafka.

82. Data Sink
Data going from Kafka to a
thing (Database, file,
ElasticSearch)

83. There are many
community connector
plugins that will fit most
needs.

84. A Source Example

85. {
"name": "twitter_source_json_01",
"config": {
"connector.class":
"com.github.jcustenborder.kafka.connect.twitter.TwitterSourceConnector",
"twitter.oauth.accessToken": "xxxxxxxxxxxxx",
"twitter.oauth.consumerSecret": "xxxxxxxxxxxx",
"twitter.oauth.consumerKey": "xxxxxxxxxxxx",
"twitter.oauth.accessTokenSecret": "xxxxxxxxxxx",
"kafka.delete.topic": "twitter_deletes_json_01",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": false,
"key.converter.schemas.enable": false,
"kafka.status.topic": "twitter_json_01",
"process.deletes": true,
"filter.keywords": "#instil, #virtualdevbash"
}
}

86. curl -XPOST -H ‘Content-type: application/json’ -d @myconnector.json
http://localhost:8083/connectors

87. A Sink Example

88. {
"name": "string-jdbc-sink",
"config": {
"connector.class": “com.domain.test.StringJDBCSink”,
"connector.type": "sink",
"tasks.max": "1",
"topics": “topic_to_read”,
"topic.type": "avro",
"tablename":"testtopic",
"connection.user": "dbuser",
"connection.password": “dbpassword_encoded”,
"connection.url": "jdbc:mysql://localhost:3307/connecttest",
"query.string": "INSERT INTO testtopic (payload) VALUES(?);",
"db.driver": "com.mysql.jdbc.Driver",
"key.converter":"org.apache.kafka.connect.storage.StringConverter",
"value.converter":"io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://localhost:8081"
}
}

89. Dead Letter Queues

90. Sometimes stuff doesn’t
work……

91. Image source: Confluent Inc.

92. “errors.tolerance":"all",
"errors.log.enable":true,
"errors.log.include.messages":true
“errors.deadletterqueue.topic.name":"dlq_topic_name",
"errors.deadletterqueue.topic.replication.factor":1,
"errors.deadletterqueue.context.headers.enable":true,
"errors.retry.delay.max.ms": 60000,
"errors.retry.timeout": 300000

93. Monitoring

94. At present the main broker Kafka metrics (via JMX)
kafka.server:*
kafka.controller:*
kafka.coordinator.group:*
kafka.coordinator.transaction:*
kafka.log:*
kafka.network:*
kafka.server:*
kafka.utils:*

95. Topic Level Fetch Metrics
MBean: kafka.consumer:type=consumer-fetch-manager-metrics,client-id=([-.w]+),topic=([-.w]+)
Key Name Description
fetch-size-avg The average number of bytes fetched per
request for a specific topic.
fetch-size-max The maximum number of bytes fetched per
request for a specific topic.
bytes-consumed-rate The average number of bytes consumed
per second for a specific topic.
records-per-request-avg The average number of records in each
request for a specific topic.
records-consumed-rate The average number of records consumed
per second for a specific topic.

96. Producer Level Application Metrics (using JConsole)

97. Consumer Level Application Metrics (using JConsole)

98. Thank you.
Twitter: @digitalis_io Web: https://digitalis.io