Scaling big with Apache Kafka
•
2 likes•261 views
Apache Kafka sits at the core of the modern scalable event driven architecture. It’s no longer used only as logging infrastructure, but as a core component in thousands of companies around the world. It has the unique capability to provide low-latency, fault-tolerant pipeline at scale that is very important for today’s world of big data. During this talk we’ll see what makes Apache Kafka perfect for the job. We’ll explore how to optimize it for throughput or for durability. And we’ll also go over the messaging semantics it provides. Last but not least, we’ll see how Apache Kafka can help us solve some everyday problems that we face when we build large scale systems in an elegant way.
Recommended
Recommended
More Related Content
What's hot
What's hot (15)
Similar to Scaling big with Apache Kafka
Similar to Scaling big with Apache Kafka (20)
More from Nikolay Stoitsev
More from Nikolay Stoitsev (20)
Recently uploaded
Recently uploaded (20)
Scaling big with Apache Kafka
- 2. Going big with Apache Kafka Nikolay Stoitsev - Sr. Software Engineer @ Uber
- 5. Topic Message Message Message Message Message Message
- 6. Partition - ordered, immutable sequence Message Message Message Message Message Message Message Message Message Message Message Message Message Message Message Partition 0 Partition 1 Partition 2
- 7. Offset Message Message Message Message MessagePartition 0 Offset: 1
- 8. Offset Message Message Message Message MessagePartition 0 Offset: 2
- 9. Multi Subscriber Message Message Message Message Message Offset: 5Offset: 3 Consumer: 2 Consumer: 1
- 13. Broker Broker Broker P0 P0 P0 P1 P1 P1 P2 P2 P2 Producer Producer Consumer Consumer Consumer Group 1 Consumer Consumer Consumer Consumer Group 2 Consumers
- 14. Broker Broker P0 P0 P1 P1 P2 P2 Producer Consumer Consumer Consumer Group 1 ZooKeeper Get Broker ID Update Offset
- 16. System for “collecting and delivering high volumes of log data with low latency”
- 18. Publish - Subscribe Or Kafka for interservice communication
- 19. Good throughput
- 20. Built-in partitioning, replication, and fault-tolerance
- 21. Durability
- 23. One leader for every partition Follower Leader Follower Follower Producer Consumer
- 27. Tune for lower latency ● Acknowledgement after persisted on the leader ● Can lost message on leadership changes ● At-most-once semantic
- 28. Tune for durability ● Acknowledgement after persisted on all ISR (after committed) ● No data loss ● At-least-once sematic
- 29. At-most-once cluster for logging
- 30. At-least-one cluster for message bus
- 31. Kafka as a message bus KafkaUpstream Downstream
- 33. Message queueing
- 35. Avro
- 36. KafkaUpstream Downstrea m Schema Registry Publish Schema Fetch Schema https://docs.confluent.io/current/schema-registry/docs/index.html
- 37. How to make sure something is durably stored and will be processed exactly once?
- 44. Idempotency
- 45. Idempotency + at least once delivery Process 1 Process 2Kafka Consumer
- 46. Out of the box exactly once delivery after 0.11
- 47. https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/
- 50. Order Service Kafka Payment Consumer Payment Provider Dead Letter Queue Payment Retry 0 Payment Retry 1 Dead Letter Queue
- 51. Order Service Kafka Payment Consumer Payment Provider Dead Letter Queue Payment Retry 0 Payment Retry 1 Dead Letter Queue Payment Consumer Payment Consumer
- 52. Multi Data Center Application
- 53. Regional Cluster Local Producer Regional Kafka DC1 Local Producer Regional Kafka DC2
- 54. Aggregated Cluster Local Producer Regional Kafka DC1 Local Producer Regional Kafka DC2 Kafka Replicator Aggregated Kafka DC3
- 58. How to monitor Kafka?
- 60. Detect data loss, lag and duplication
- 61. Audit Library Regional Kafka Service Kafka REST Proxy Audit Library Audit Library
- 62. Chaperone Service Regional Kafka Service Kafka REST Proxy Audit Library Audit Library Chaperone Service
- 63. Chaperone Collector Regional Kafka Service Kafka REST Proxy Audit Library Audit Library Aggregate Kafka Chaperone Service Chaperone Service Chaperone Collector DB
- 68. Summary Tune for durability Define Avro Schemas
- 69. Summary Tune for durability Define Avro Schemas Use Kafka REST Proxy
- 70. Summary Tune for durability Define Avro Schemas Use Kafka REST Proxy Add idempotency checks
- 71. Summary Tune for durability Define Avro Schemas Use Kafka REST Proxy Add idempotency checks Use Dead Letter Queue
- 72. Summary Tune for durability Define Avro Schemas Use Kafka REST Proxy Add idempotency checks Use Dead Letter Queue Monitor everything
- 73. THANK YOU! Nikolay Stoitsev Sr. Software Engineer @ Uber