Spring Kafka beyond the basics - Lessons learned on our Kafka journey (Tim van Baarsen, ING Bank) Kafka Summit 2020
•
0 likes•2,122 views
You know the fundamentals of Apache Kafka. You are a Spring Boot developer and working with Apache Kafka. You have chosen Spring Kafka to integrate with Apache Kafka. You implemented your first producer, consumer and maybe some Kafka streams, it's working... Hurray! You are ready to deploy to production what can possibly go wrong? In this talk, Tim will take you on a journey beyond the basics of Spring Kafka and will share his knowledge, pitfalls and lessons learned based on real-life Kafka projects that are running in production for many years at ING Bank in the Netherlands. This deep-dive talk is a balanced combination of code examples, fully working demos and theory behind the lessons learned like: * Survive consuming a poison pill from a Kafka topic without writing gigabytes of log lines and blow up your server's hard drive? * Understand how to deal with exceptions in different cases * Validate incoming data consumed from a Kafka topic. After joining this talk, you will be inspired to bring your Spring Kafka projects to the next level and share some practical tips and tricks with your fellow developer colleagues!
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Spring Kafka beyond the basics - Lessons learned on our Kafka journey (Tim van Baarsen, ING Bank) Kafka Summit 2020
Similar to Spring Kafka beyond the basics - Lessons learned on our Kafka journey (Tim van Baarsen, ING Bank) Kafka Summit 2020 (20)
More from confluent
More from confluent (20)
Recently uploaded
Recently uploaded (20)
Spring Kafka beyond the basics - Lessons learned on our Kafka journey (Tim van Baarsen, ING Bank) Kafka Summit 2020
- 1. Spring Kafka beyond the basics Lessons learned on our Kafka journey at ING Bank Tim van Baarsen KAFKA SUMMIT – AUSTIN (VIRTUAL)
- 2. Who I am? Tim van Baarsen Software Engineer @ ING Netherlands – Tribe Investments The Netherlands Tribe InvestmentsAmsterdam @TimvanBaarsen
- 3. Kafka @ ING 3 Frontrunners in Kafka Running in production: • 6 years • 1800+ topics • Serving 450+ development teams • 20k to 350k messages per second • Traffic grew 1100% in the last three years • Self service topic management • Many different use cases
- 4. After the honeymoon phase is over 4 🧨 😱 🚀 😎 Traps Lessons learned The hard way! Apply in your own project(s) Share with your fellow developers Tips & Tricks %
- 6. Domain Lingo AAPL ‘quote’ = price last trade of a stock Constant stream of price updates on stocks
- 7. 💊 Scenario: ‘Poison Pill’ (Deserialization exception) 7 💊💀 Corrupted record What is a Poison Pill? Deserialization failure A record that always fails when consumed, no matter how many times it is attempted. Different forms:
- 8. 💊 Scenario: ‘Poison Pill’ (Deserialization exception) 8 Scenario: • Consumer of topics • Someone produced a ‘poison’ message • Consumer fails to deserialize • Consequence • Blocks consumption of the topic/partition • application can’t ‘swallow the pill’ • Try again and again and again (very fast) • Log line for every failure AAPL NASDAQ $243.65 INGA AEX €10.34 NFLX NASDAQ $280.30 INGA AEX €10.15 💊PILL ? Consumer Kafka client Consumes from: Kafka topic: ‘stock-quotes’ Consumer Another Consumer (different application)
- 9. 💊 Scenario: ‘Poison Pill’ (Deserialization exception) 9 • Result: log file will flood your disc • Impact: High • How to survive this scenario? • Wait until the retention period of the topic has passed • Change consumer group • Manually / Programmatically update the offset • Configure ErrorHandlingDeserializer (Provided by Spring Kafka) 👎 👍 👎 👎 AAPL NASDAQ $243.65 INGA AEX €10.34 NFLX NASDAQ $280.30 INGA AEX €10.15 💊PILL ? Consumer Kafka client Consumes from: Kafka topic: ‘stock-quotes’
- 10. ConsumerConsumer 💊 Scenario: ‘Poison Pill’ (Deserialization exception) 10 0 1 2 3 4 5 6 7 8 old new Kafka topic: ‘stock-quotes’ Producer Kafka client Kafka broker - micro batching - serialization • key • value Consumer Kafka client - micro batching - deserialization • key • value Kafka Broker is not responsible for: - Type checking - Schema validation - Other constraints Data in a Kafka topic are just stored as bytes CustomSerializer BytesSerializer KafkaAvroDeserializer JsonSerializer CustomDeserializer BytesDeserializer JsonDeserializer KafkaAvroSerializer StringSerializerStringDeserializer 0100101001101 0100101001101 poll send
- 11. ConsumerConsumer 💊 Scenario: ‘Poison Pill’ (Deserialization exception) 11 0 1 2 3 4 5 6 7 8 old new Kafka topic: ‘stock-quotes’ Producer Kafka client Kafka broker - micro batching - serialization • key • value Consumer Kafka client - micro batching - deserialization • key • value Kafka Broker is not responsible for: - Type checking - Schema validation - Other constraints Data in a Kafka topic are just stored as bytes CustomSerializer BytesSerializer KafkaAvroDeserializer JsonSerializer CustomDeserializer BytesDeserializer JsonDeserializer KafkaAvroSerializer StringSerializerStringDeserializer 0100101001101 0100101001101 poll send
- 12. 💊 Scenario: ‘Poison Pill’ (Deserialization exception) 12 0 1 2 3 4 5 6 7 8 old new Kafka topic: ‘stock-quotes’ Producer Kafka broker Consumer KafkaAvroDeserializer KafkaAvroSerializer StringSerializer 010010100110 010010100110 poll send Producersend >_ 💊 Confluent Schema Registry REST API Load schema Register schema Schema validation - Avro
- 13. 💊 Scenario: ‘Poison Pill’ - Demo 13
- 14. 🎓 Lessons learned 14 • Consumer: • Expect the unexpected! • Handle Deserialization exceptions a.k.a. ‘poison pills’ • Propper exception handling • Dead letter topic • Producer: • Don’t change your serializers • Shared secrets • Leverage Apache Avro + Confluent Schema registry • Security • Who can produce data to your topics? • Who can consume?
- 15. Questions 15 🤔
- 16. Thanks for joining my talk! Tim van Baarsen Software Engineer @ ING Netherlands @TimvanBaarsen https://www.medium.com/@TimvanBaarsen Example codebase on Github