Watch this talk here: https://www.confluent.io/online-talks/how-apache-kafka-works-on-demand Pick up best practices for developing applications that use Apache Kafka, beginning with a high level code overview for a basic producer and consumer. From there we’ll cover strategies for building powerful stream processing applications, including high availability through replication, data retention policies, producer design and producer guarantees. We’ll delve into the details of delivery guarantees, including exactly-once semantics, partition strategies and consumer group rebalances. The talk will finish with a discussion of compacted topics, troubleshooting strategies and a security overview. This session is part 3 of 4 in our Fundamentals for Apache Kafka series.

1. 1
Fundamentals for Apache Kafka®
How Apache Kafka Works
Mike Bingham, Technical Trainer
Confluent Certified Developer, Confluent Certified Administrator

2. 2
Session Schedule
● Session 1: Benefits of Stream Processing and Apache Kafka Use Cases
● Session 2: Apache Kafka Architecture & Fundamentals Explained
● Session 3: How Apache Kafka Works
● Session 4: Integrating Apache Kafka into your Environment

3. 3
Learning Objectives
After this module you will be able to:
● Give a high level description of the programming logic
in Kafka producer and consumer clients
● Explain how EOS works to an interested lay person
● List the means with which Kafka provides durability
and HA
● Illustrate on a high level, how you can secure your
Kafka cluster

4. 4
Development:
A Basic
Producer in
Java

5. 5
Development:
A Basic
Consumer in
.NET/C#

6. 6
Partition Leadership & Replication

7. 7
Partition Leadership & Replication

8. 8
Data Retention Policy
How long do I want or can I store my data?
• How long (default: one week)
• Set globally or per topic
• Business decision
• Cost factor
• Compliance factor (e.g., GDPR)

9. 9
Compacted Topics

10. 10
Producer Design

11. 11
Producer
Guarantees

12. 12
Delivery Guarantees

13. 13
Idempotent
Producers

14. 14
Exactly Once Semantics (EOS)
What is it?
● Strong transactional guarantees for
Kafka
● Prevents clients from processing
duplicate messages
● Handles failures gracefully
Use Cases
● Tracking ad views
● Processing financial transactions
● Stream processing

15. 15
Partitioning Strategies
Why partitioning?
● Consumers need to aggregate or join by some key
● Consumers need ordering guarantee
● Concentrate data for storage efficiency and/or indexing

16. 16
Consumer Groups

17. 17
Consumer
Rebalances

18. 18
Troubleshooting
● Confluent Control Center
● Log files
● Special config settings
○ SSL Logging
○ Authorizer debugging

19. 19
Security Overview
● Kafka supports Encryption in Transit
● Kafka supports Authorization and Authentication
● No Encryption at Rest out of the box
● Clients can be mixed with & without Encryption & Authentication

20. 20
Client Side Security Features
● Encryption of Data in Transit
● Client Authentication
● Client Authorization

21. 21
Continue your Apache Kafka Education!
● Confluent Operations for Apache Kafka
● Confluent Developer Skills for Building Apache Kafka
● Confluent Stream Processing using Apache Kafka
Streams and KSQL
● Confluent Advanced Skills for Optimizing Apache Kafka
For more details, seehttp://confluent.io/training

22. 2222
Certifications
Confluent Certified Developer
for Apache Kafka
(aligns to Confluent Developer Skills
for Building Apache Kafka course)
Confluent Certified
Administrator for Apache
Kafka
(aligns to Confluent Operations Skills
for Apache Kafka)
What you Need to Know
○ Qualifications: 6-to-9 months hands-on
experience
○ Duration: 90 mins
○ Availability: Live, online 24/7
○ Cost: $150
○ Register online:
www.confluent.io/certification

23. 2323
cnfl.io/slack
Stay in touch!
cnfl.io/kafka-trainingcnfl.io/download

24. 25
Thank you for attending!
• Thank you for attending thesession!
• Feedback to: training-admin@confluent.io

25. 26
Copyright ©Confluent, Inc. 2014-2019. Privacy Policy | Terms &Conditions.
Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of
the Apache Software Foundation