Presented at 2nd Apache Kafka Bratislava meetup.

1. Cell based security in
Kafka
Tibor Varkonyi
2017-11-29

2. 2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Agenda
Kafka
Security
Prototype
Demo
Future

3. 3 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Kafka
 Zookeeper
 Producers
 Consumers
 Brokers
 Arbitrary data
 Serializers / Deserializers
https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/

4. 4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Agenda
Kafka
Security
Prototype
Demo
Future

5. 5 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Security
 Access Control List (ACL)
 Simple Authentication and Security
Layer (SASL)
 Topic level security
 Structured data
 Access control for fields
 Broker side authorization
 Message level security
What we have What we need

6. 6 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Agenda
Kafka
Security
Prototype
Demo
Future

7. 7 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Prototype
 Real life use case based on bank requirement
 Different types of consumers
 Different types of users
 Store sensitive information
– Credit card number
– Other real-life ID
 Bind security metadata to messages

8. 8 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
000010000015fd8f55a280000015fd8f55d94ffffffffffffffffffffffffffff00000002 16000000 010a68656c6c6f00 1800d80d 02 010a776f726c6400
000000000015fd8fc3a380000015fd8fc3a38ffffffffffffffffffffffffffff00000001 14000000 01087961617900
000000000015fd90132950000015fd9013295ffffffffffffffffffffffffffff00000001 12000000 010665797900
000000000015fd90468140000015fd9046814ffffffffffffffffffffffffffff00000001 12000000 010661736400
000000000015fd911d2380000015fd911d238ffffffffffffffffffffffffffff00000001 12000000 010661736400
00000000015fd91542830000015fd9154283ffffffffffffffffffffffffffff00000001 16000000 010a6a6f7a736900
000000000015fd94bd6c40000015fd94bd6c4ffffffffffffffffffffffffffff00000001 16000000 010a6a6f7a736900
000000000015fd94d07010000015fd94d0701ffffffffffffffffffffffffffff00000001 1c000000 01106b6a6b6b6a6b6c6a00
000000000015fd951179b0000015fd951179bffffffffffffffffffffffffffff00000001 12000000 010661736400
000000000015fd9512ad00000015fd9512ad0ffffffffffffffffffffffffffff00000001 12000000 010677746600
00000000015fd95132580000015fd9513258ffffffffffffffffffffffffffff00000001 16000000 010a445347445300
00000000015fd952cd070000015fd952cd07ffffffffffffffffffffffffffff00000001 18000000 010c6b6a686a6b6800
000000000015fd952da530000015fd952da53ffffffffffffffffffffffffffff00000001 1c000000 01106867686a66676a6600
00000000015fd952e1820000015fd952e182ffffffffffffffffffffffffffff00000001 12000000 010661626300
000000000015fd952e6130000015fd952e613ffffffffffffffffffffffffffff00000001 12000000 010661626300
000010000015fd952ec630000015fd952efabffffffffffffffffffffffffffff00000002 12000000 0106616e6300 1400900d 02 010661626300
000000000015fd956a7eb0000015fd956a7ebffffffffffffffffffffffffffff00000001 14000000 01086b6a6b6a00
000000000015fd956e4240000015fd956e424ffffffffffffffffffffffffffff00000001 18000000 010c6c6b6a6c6b6a00
00000000015fd956e94c0000015fd956e94cffffffffffffffffffffffffffff00000001 16000000 010a6b6a6b6c6a00
000000000015fd9597b030000015fd9597b03ffffffffffffffffffffffffffff00000001 1a000000 010e68626a6862686a00
000000000015fd959b7d60000015fd959b7d6ffffffffffffffffffffffffffff00000001 14000000 01086161616200
000000000015fd96ebef30000015fd96ebef3ffffffffffffffffffffffffffff00000001 12000000 010661736400
000000000015fd97170130000015fd9717013ffffffffffffffffffffffffffff00000001 22000000 011668656c6c6f20776f726c6400
000000000015fd973b9460000015fd973b946ffffffffffffffffffffffffffff00000001 12000000 010661736400
000000000015fd974827e0000015fd974827effffffffffffffffffffffffffff00000001 12000000 010661736400
000000000015fd976f0800000015fd976f080ffffffffffffffffffffffffffff00000001 14000000 01086173646600
00000000015fd977964a0000015fd977964affffffffffffffffffffffffffff00000001 12000000 010661736400
000000000015fd97a3f280000015fd97a3f28ffffffffffffffffffffffffffff00000001 16000000 010a646667646600
000000000015fd97cd8710000015fd97cd871ffffffffffffffffffffffffffff00000001 12000000 010661736400
000000000015fd97ec1650000015fd97ec165ffffffffffffffffffffffffffff00000001 14000000 01087364666700
Prototype
 Structured data as valid JSON
 Extend message headers with security data
 Filter per-message by security data
 Has backward compatibility for producers, consumers, and brokers
 Access management through a Scala trait
 Performance overhead visible

9. 9 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Consumer
Prototype
JSON Serialize Binary data
Binary data Deserialize JSON
Producer Broker
Broker

10. 10 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
ConsumerBroker
Prototype
JSON Serialize Binary data
Binary data Deserialize JSON
Producer Broker
Deserialize
Serialize
Filter

11. 11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Agenda
Kafka
Security
Prototype
Demo
Future

12. 12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Agenda
Kafka
Security
Prototype
Demo
Future

13. 13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Future
 Topic metadata
 Filter full messages
 Handle user groups
 Support Ranger

14. 14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Thank You