Consistency and Completeness: Rethinking Distributed Stream Processing in Apa...
Your SlideShare is downloading.
×
Check these out next
Recommended
More Related Content
Slideshows for you (20)
Similar to Top Ten Kafka® Configs (20)
More from confluent (20)
Recently uploaded (20)
Top Ten Kafka® Configs
- 1. 1 Top Ten Kafka Configs Important Configurations for Optimum Kafka Performance and Robustness
- 2. 2 Brief Introduction • Worked at Confluent (Streams Team) 2 years • Apache Kafka Committer • Author Kafka Streams in Action
- 3. 3 Special Thanks • Dustin Cote • Justin Manchester • Koelli Mungee • Ryan Prigeon
- 4. 4 Agenda • Quick Intro to Kafka • Ground Rules • Broker Configs • Client Configs • Consumer • Producer
- 5. 5 Intro to Kafka Broker 1 Broker 2 Broker 3
- 6. 6 Intro to Kafka Broker 1 Broker 2 Broker 3 Topic
- 7. 7 Intro to Kafka Topic 0 1 2 3 Next record
- 8. 8 Intro to Kafka 0 1 2 3 Producer
- 9. 9 Intro to Kafka 0 1 2 3 Producer Consumer
- 10. 10 Intro to Kafka 0 1 2 3 Producer Consumer Consumer reads available records then commits the offsets
- 11. 11 Configuration Ground Rules • Kafka is the central nervous system of your Event based platform • Proper configuration is important
- 12. 12 Configuration Ground Rules Not All Configurations need to be tuned!
- 13. 13 Configuration Ground Rules • Make sure you understand the behavior and what you’re trying to achieve before changing configs • Focus on what’s essential for your needs
- 14. 14 Broker Configs Producer Broker Consumer
- 15. 15 Broker Configs • JMX metrics • UncleanLeaderElection • Retention (Topic and Broker) • Min InSync Replicas (discussed in Producer section)
- 16. 16 JMX Metrics • JMX metrics not exposed remotely by default • No insight into broker activity without them
- 17. 17 JMX Metrics • Set the environment variable JMX_PORT to enable remote JMX metrics
- 18. 18 JMX Metrics • With environment variable JMX_PORT set Kafka uses these default properties for JMX -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
- 19. 19 JMX Metrics • To override/customize the JMX settings, i.e. provide authentication set the KAFKA_JMX_OPTS env variable
- 20. 20 JMX Metrics Viewing JConsole https://docs.oracle.com/javase/8/docs/technotes/guid es/management/jconsole.html
- 21. 21 JMX Metrics Connecting ssh -fN -D 7777 -i ~/.ssh/<key name> <instance-public-ip> jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 -J-DsocksNonProxyHosts= service:jmx:rmi:///jndi/rmi://localhost:<JMX_PORT>/jmxrmi
- 22. 22 JMX Metrics BrokerTopicMetrics • MessagesInPerSec • BytesInPerSec • BytesOutPerSec
- 23. 23 JMX Metrics RequestMetrics • Requests per second • Produce - requests from producer to send records • FetchConsumer – requests from consumer to retrieve records • FetchFollower – requests from follower broker to keep in sync with leader for given topic-partition
- 24. 24 JMX Metrics RequestChannel • ReqeustQueueSize Special thanks to Dustin Cote for the Broker Load metrics
- 25. 25 UncleanLeaderElection Leader Broker Follower Broker Leader has records up to offset 300 Follower is only up to 150
- 26. 26 UncleanLeaderElection Leader Broker Follower Broker Leader has records up to offset 300 Follower is only up to 150
- 27. UncleanLeaderElection Allow Follower to become leader • Almost immediately available • Potential for missing records
- 28. UncleanLeaderElection Wait to bring back old leader broker • Not highly available • Downtime until old leader recovered • No potential for missing records
- 29. UncleanLeaderElection • Default setting is false • Use with caution
- 30. 30 Retention • Size • Age of records • now – largest timestamp > retention time
- 31. 31 Timestamps - CreateTime Producer sets timestamp Producer Broker Topic
- 32. 32 Timestamps -LogAppendTime Producer sets timestamp Producer Broker Topic
- 33. 33 Retention Topic A retention 1 week Topic B retention 2 days Kafka Streams Application
- 34. 34 Consumer Configs Producer Broker Consumer
- 35. 35 Consumer Configurations • Max Poll Interval • Committing
- 36. 36 Max Poll Interval max.poll.interval.ms • max time between poll calls • If exceeded then consumer kicked out of group, rebalancing while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5)); for (ConsumerRecord<String, String> record : records) { //do something with recrods } }
- 37. 37 Max Poll Interval max.poll.interval.ms = 30000/30 seconds while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5)); for (ConsumerRecord<String, String> record : records) { //This loop must complete in under 30 seconds } }
- 38. 38 Max Poll Interval Processing time for fetched records on consumer takes 45 seconds while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5)); for (ConsumerRecord<String, String> record : records) { //Processing in this loop takes 45 seconds this consumer kicked out of group, and a rebalance is triggered } }
- 39. 39 Max Poll Interval max.poll.interval.ms solutions - • Increase the poll interval time – maybe • Possibly decrease max.poll.records while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5)); for (ConsumerRecord<String, String> record : records) { //This loop must complete in under 30 seconds } }
- 40. 40 Max Poll Interval max.poll.interval.ms solutions - • Decrease processing time • Take processing out of loop while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5)); for (ConsumerRecord<String, String> record : records) { //This loop must complete in under 30 seconds } }
- 41. 41 Committing Partition A Consumer By default auto commit enabled every 5 seconds
- 42. 42 Committing Partition A Consumer Only commit last record processed
- 43. 43 Committing 1 2 3 4 5 6 7 8 9 10 6 last processed offset 10 Records in batch
- 44. 44 Committing 1 2 3 4 5 6 7 8 9 10 Committing offset 11 means 7, 8, and 9 committed as well!
- 45. 45 Producer Configs Producer Broker Consumer
- 46. 46 Producer Configs • Linger • Acks • Retries and Delivery Timeouts • Keeping Records in Order
- 47. 47 Linger Producer Record Producer Record Buffer
- 48. 48 Linger – Normal Traffic Producer Record Buffer
- 49. 49 Linger – Lighter Traffic Producer Record Buffer
- 50. 50 Linger Trade Offs • Wait for more records • Waiting to send a batch • Decreased number of requests to broker
- 51. 51 Linger Trade Offs • Set linger.ms to zero • Send batches faster • More requests to broker
- 52. 52 Acks • Number of acknowledgements from broker that it has persisted the message • Zero • One • All
- 53. 53 Acks Zero Producer “Fire” and forget don’t wait for any response Broker (Leader) Broker (Follower) Broker (Follower)
- 54. 54 Acks One Producer Produce records and wait for leader broker to acknowledge Broker (Leader) Broker (Follower) Broker (Follower) Got it!
- 55. 55 Acks All Producer Produce waits for leader broker to acknowledge after all followers have acknowledged Broker (Leader) Broker (Follower) Broker (Follower) Got it!
- 56. 56 Acks All Producer Produce waits for leader broker to acknowledge after all followers have acknowledged Broker (Leader) Broker (Follower) Broker (Follower) Got it! Got it!
- 57. 57 InSync Replicas • acks = “all” • Leader broker is considered a replica!
- 58. 58 InSync Replicas Producer acks = “all” min.insync.replicas = 1 Broker (Leader) Broker (Follower) Broker (Follower) Got it!
- 59. 59 InSync Replicas Producer acks = “all” min.insync.replicas = 1 possible data loss Broker (Leader) Broker (Follower) Broker (Follower) Got it!
- 60. 60 InSync Replicas Producer acks = “all” min.insync.replicas = 2 no data loss Broker (Leader) Broker (Follower) Broker (Follower) Got it!
- 61. 61 InSync Replicas Producer acks = “all” min.insync.replicas = 2 no data loss Broker (Leader) Broker (Follower) Broker (Follower) Got it! Got it!
- 62. 62 InSync Replicas Producer acks = “all” min.insync.replicas = 2 no data loss Broker (Leader) Broker (Follower) Broker (Follower) Got it!
- 63. 63 Retries Topic A Partition 0 Producer Retriable Exception
- 64. 64 Retries Topic A Partition 0 Producer Retriable Exception 0.. Max Retries
- 65. 65 Retries Topic A Partition 0 Producer Retriable Exception 0.. Max Retries or
- 66. 66 Retries Topic A Partition 0 Producer Retriable Exception Use delivery.timeout.ms for limit of time from sending to receiving acknowledgement see KIP-91 for details
- 67. 67 Keeping Records in Order Topic A Partition 0 Producer Retriable Exception Batch 1
- 68. 68 Keeping Records in Order Topic A Partition 0 Producer Retriable Exception Batch 1 Batch 2
- 69. 69 Keeping Records in Order Topic A Partition 0 Producer Retriable Exception Batch 1 Batch 2 Batch 1
- 70. 70 Keeping Records in Order Topic A Partition 0 Producer Retriable Exception Batch 1 Batch 2 Batch 1 Batch 2 processed before Batch 1 !
- 71. 71 Keeping Records in Order Topic A Partition 0 Producer Retriable Exception Batch 1 Batch 2 Batch 1 max.in.flight.requests.per.connection = 1
- 72. 72 Keeping Records in Order Topic A Partition 0 Producer Retriable Exception Batch 1 Batch 1 max.in.flight.requests.per.connection = 1 Batch 2
- 73. 73 Summary • Configuration is important to the health of a Kafka Cluster/ • Need to know which knobs to turn • Measure the before and after • Try it in development first
- 74. 74 Summary • Kafka The Definitive Guide - http://shop.oreilly.com/product/0636920044123.do • Apache Kafka docs - https://kafka.apache.org/documentation/
- 75. NOMINATE YOURSELF OR A PEER AT CONFLUENT.IO/NOMINATE
- 76. KS19Meetup. CONFLUENT COMMUNITY DISCOUNT CODE 25% OFF* *Standard Priced Conference pass
- 77. 77 Thanks! Stay in Touch! • https://slackpass.io/confluentcommunity • https://www.confluent.io/blog/ • Twitter @bbejeck • We are hiring! https://www.confluent.io/careers/
