More Related Content Similar to Kafka 0.9, Things you should know (20) Kafka 0.9, Things you should know3. ©2016 LinkedIn Corporation. All Rights Reserved. 3
Agenda
Security
Kafka Connect
User defined quota
New consumer
Notable improvements and fixes
Upgrading from kafka 0.8
Kafka 0.10 - highlights
5. ©2016 LinkedIn Corporation. All Rights Reserved. 5
Security
Authentication
Kerberos
TLS
Unix like permission
"Principal P is [Allowed/Denied] Operation O From Host H On Resource R"
bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --
allow-principal User:Bob --allow-principal User:Alice --allow-host 198.51.100.0 --
allow-host 198.51.100.1 --operation Read --operation Write --topic Test-topic
Encryption
6. ©2016 LinkedIn Corporation. All Rights Reserved. 6
Kafka Connect
Why ?
Multiple tools for importing and exporting
High engineering and operational overhead
Some tools are poor fit for job
7. ©2016 LinkedIn Corporation. All Rights Reserved. 7
Kafka Connect
Kafka
Data
source
C
Data
source
B
Data
source
A
Data
sink 3
Data
sink 2
Data
sink 1T1
T2
T3 T6
T5
T4
8. ©2016 LinkedIn Corporation. All Rights Reserved. 8
Kafka Connect
Kafka
Data
source
C
Data
source
B
Data
source
A
Data
sink 3
Data
sink 2
Data
sink 1
KafkaConnect
KafkaConnect
9. ©2016 LinkedIn Corporation. All Rights Reserved. 9
Kafka Connect
Key Properties:
Broad copying by default
Streaming and batch
Scales to application
Focus on copying data only
Parallel
Connector API
10. ©2016 LinkedIn Corporation. All Rights Reserved. 10
Kafka Connect
Advantages :
Fault tolerance
Partitioning
Offset management
Delivery semantics
Operations
Monitoring
12. ©2016 LinkedIn Corporation. All Rights Reserved. 12
User defined quota
Single large cluster
Producer side (quota.producer.default)
Consumer side (quota.consumer.default)
Per client , Per broker
Quota override
./bin/kafka-config.sh --alter
--add-config ‘producer_byte_rate=1048576,consumer_byte_rate=1048576’
--entity-type clients
--entity-name TestTopic
--zookeeper localhost:2181
13. ©2016 LinkedIn Corporation. All Rights Reserved. 13
New consumer
Motivation :
Thin consumer client
Central co-ordination
Allow manual partition assignment
Allow manual offset management
Invocation of user specified callback on rebalance
Non blocking consumer APIs
18. ©2016 LinkedIn Corporation. All Rights Reserved. 18
New consumer
Features:
Group management protocol
Consumer
Co-ordinator
Failure detection protocol
19. ©2016 LinkedIn Corporation. All Rights Reserved. 19
New consumer
Interesting scenarios:
Co-ordinator failover/connection loss
Partition changes for subscribed topics
Offset commit during rebalance
Hearbeats during rebalance
Slow consumers
20. ©2016 LinkedIn Corporation. All Rights Reserved. 20
Notable improvements and fixes
Automated replica lag tuning (replica.lag.time.max.ms)
New purgatory design – low memory overhead
Auto-assign node ids
No data loss in Mirror Maker – unclean shutdown
Log compaction for compressed topics
Handling of corrupt index files
21. ©2016 LinkedIn Corporation. All Rights Reserved. 21
Upgrading from kafka 0.8
inter.broker.protocol.version=0.8.2.x
Update code and restart
inter.broker.protocol.version=0.9.0.0
Restart brokers again
22. ©2016 LinkedIn Corporation. All Rights Reserved. 22
Potential Breaking Changes:
Java 1.6 and Scala 2.9 are not supported
Broker IDs > 1000 ( reserved.broker.max.id and
broker.id.generation.enable )
replica.lag.max.messages removed
replica.lag.time.max.ms
No compaction for topics without key
Upgrading from kafka 0.8
23. ©2016 LinkedIn Corporation. All Rights Reserved. 23
Potential Breaking Changes contd….
Changes in default JVM options
Upgrading from kafka 0.8
25. ©2016 LinkedIn Corporation. All Rights Reserved. 25
Kafka 0.10 - Highlights
Kafka Streams
Rack Awareness
Timestamps in messages
SASL improvements
Kafka consumer max record
Protocol version improvements
26. ©2016 LinkedIn Corporation. All Rights Reserved. 26
References
http://kafka.apache.org/090/documentation.html
http://www.confluent.io/blog/apache-kafka-0.9-is-released
http://www.confluent.io/blog/announcing-apache-kafka-0.10-and-confluent-
platform-3.0
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+R
ewrite+Design