The presentation is about the kafka 9 features and the bug fixes. The presentation also talks about the new consumer. Explains the state diagram of consumer and co-ordinator

1. ©2016 LinkedIn Corporation. All Rights Reserved.
Kafka 0.9, Things you should know

2. ©2016 LinkedIn Corporation. All Rights Reserved.
Ratish Ravindran
Site Reliability Engineer
LinkedIn, Data Infrastructure Streaming

3. ©2016 LinkedIn Corporation. All Rights Reserved. 3
Agenda
 Security
 Kafka Connect
 User defined quota
 New consumer
 Notable improvements and fixes
 Upgrading from kafka 0.8
 Kafka 0.10 - highlights

4. ©2016 LinkedIn Corporation. All Rights Reserved. 4
Security
 Why ?
 Multitenant cluster
 Multiple clusters
 Network ACLs

5. ©2016 LinkedIn Corporation. All Rights Reserved. 5
Security
 Authentication
 Kerberos
 TLS
 Unix like permission
 "Principal P is [Allowed/Denied] Operation O From Host H On Resource R"
bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --
allow-principal User:Bob --allow-principal User:Alice --allow-host 198.51.100.0 --
allow-host 198.51.100.1 --operation Read --operation Write --topic Test-topic
 Encryption

6. ©2016 LinkedIn Corporation. All Rights Reserved. 6
Kafka Connect
 Why ?
 Multiple tools for importing and exporting
 High engineering and operational overhead
 Some tools are poor fit for job

7. ©2016 LinkedIn Corporation. All Rights Reserved. 7
Kafka Connect
Kafka
Data
source
C
Data
source
B
Data
source
A
Data
sink 3
Data
sink 2
Data
sink 1T1
T2
T3 T6
T5
T4

8. ©2016 LinkedIn Corporation. All Rights Reserved. 8
Kafka Connect
Kafka
Data
source
C
Data
source
B
Data
source
A
Data
sink 3
Data
sink 2
Data
sink 1
KafkaConnect
KafkaConnect

9. ©2016 LinkedIn Corporation. All Rights Reserved. 9
Kafka Connect
Key Properties:
 Broad copying by default
 Streaming and batch
 Scales to application
 Focus on copying data only
 Parallel
 Connector API

10. ©2016 LinkedIn Corporation. All Rights Reserved. 10
Kafka Connect
Advantages :
 Fault tolerance
 Partitioning
 Offset management
 Delivery semantics
 Operations
 Monitoring

11. ©2016 LinkedIn Corporation. All Rights Reserved. 11
User defined quota
 Why ?
 High reads
 High writes
 SLAs

12. ©2016 LinkedIn Corporation. All Rights Reserved. 12
User defined quota
 Single large cluster
 Producer side (quota.producer.default)
 Consumer side (quota.consumer.default)
 Per client , Per broker
 Quota override
./bin/kafka-config.sh --alter
--add-config ‘producer_byte_rate=1048576,consumer_byte_rate=1048576’
--entity-type clients
--entity-name TestTopic
--zookeeper localhost:2181

13. ©2016 LinkedIn Corporation. All Rights Reserved. 13
New consumer
Motivation :
 Thin consumer client
 Central co-ordination
 Allow manual partition assignment
 Allow manual offset management
 Invocation of user specified callback on rebalance
 Non blocking consumer APIs

14. ©2016 LinkedIn Corporation. All Rights Reserved. 14
New consumer
Features:
 Group management protocol
 Consumer

15. ©2016 LinkedIn Corporation. All Rights Reserved. 15
New consumer
State Diagram - consumer

16. ©2016 LinkedIn Corporation. All Rights Reserved. 16
New consumer
Features:
 Group management protocol
 Consumer
 Co-ordinator

17. ©2016 LinkedIn Corporation. All Rights Reserved. 17
New consumer
State Diagram – Co-ordinator

18. ©2016 LinkedIn Corporation. All Rights Reserved. 18
New consumer
Features:
 Group management protocol
 Consumer
 Co-ordinator
 Failure detection protocol

19. ©2016 LinkedIn Corporation. All Rights Reserved. 19
New consumer
Interesting scenarios:
 Co-ordinator failover/connection loss
 Partition changes for subscribed topics
 Offset commit during rebalance
 Hearbeats during rebalance
 Slow consumers

20. ©2016 LinkedIn Corporation. All Rights Reserved. 20
Notable improvements and fixes
 Automated replica lag tuning (replica.lag.time.max.ms)
 New purgatory design – low memory overhead
 Auto-assign node ids
 No data loss in Mirror Maker – unclean shutdown
 Log compaction for compressed topics
 Handling of corrupt index files

21. ©2016 LinkedIn Corporation. All Rights Reserved. 21
Upgrading from kafka 0.8
 inter.broker.protocol.version=0.8.2.x
 Update code and restart
 inter.broker.protocol.version=0.9.0.0
 Restart brokers again

22. ©2016 LinkedIn Corporation. All Rights Reserved. 22
Potential Breaking Changes:
 Java 1.6 and Scala 2.9 are not supported
 Broker IDs > 1000 ( reserved.broker.max.id and
broker.id.generation.enable )
 replica.lag.max.messages removed
 replica.lag.time.max.ms
 No compaction for topics without key
Upgrading from kafka 0.8

23. ©2016 LinkedIn Corporation. All Rights Reserved. 23
Potential Breaking Changes contd….
 Changes in default JVM options
Upgrading from kafka 0.8

24. ©2016 LinkedIn Corporation. All Rights Reserved. 24
Why not kafka 0.10 ?

25. ©2016 LinkedIn Corporation. All Rights Reserved. 25
Kafka 0.10 - Highlights
 Kafka Streams
 Rack Awareness
 Timestamps in messages
 SASL improvements
 Kafka consumer max record
 Protocol version improvements

26. ©2016 LinkedIn Corporation. All Rights Reserved. 26
References
 http://kafka.apache.org/090/documentation.html
 http://www.confluent.io/blog/apache-kafka-0.9-is-released
 http://www.confluent.io/blog/announcing-apache-kafka-0.10-and-confluent-
platform-3.0
 https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+R
ewrite+Design

27. ©2016 LinkedIn Corporation. All Rights Reserved.