Downloaded 25 times
More Related Content
Similar to Paris Kafka Meetup - patterns anti-patterns
Paris Kafka Meetup - patterns anti-patterns
- 1. #DevoxxFR 1 Patterns – Anti-Patterns FlorentRamière | florent@confluent.io Damien Gasparina | damien@confluent.io
- 2.
- 3. 3 Silos explained byData Gravity concept As data accumulates (builds mass) there is a greater likelihood that additional services and applications will be attracted to this data. This is the same effect gravity has on objects around a planet. As the mass or density increases, so does the strength of gravitational pull.
- 4.
- 5.
- 6. 6 Store & ETLProcess Publish & Subscribe In short
- 7. 7 From a simpleidea
- 8. 8 From a simpleidea
- 9. 9 with great properties! • Scalability • Retention • Durability • Replication • Security • Resiliency • Throughput • Ordering • Exactly Once Semantic • Transaction • Idempotency • Immutability • …
- 10.
- 11. 11 11 What could potentiallygo wrong ?
- 12.
- 13. 13 13 …which is truefor any data systems
- 14. 14 14 Not thinking aboutDurability
- 15. 15 Data durability Ifyou didn’t think about it… it’s not durable!
- 16.
- 17.
- 18.
- 19. 19 19 And you mighthave lost data!
- 20. 20 Data durability Kafkais not waiting for a disk flush by default. Durability is achieved through replication.
- 21.
- 22.
- 23.
- 24.
- 25. 25 Data durability Itdepends on your configuration...
- 26.
- 27.
- 28.
- 29.
- 30. 30 Data durability acks=1(default value) good for latency acks=all good for durability
- 31.
- 32. 32 acks=all The leaderwill wait for the full set of in-sync replicas to acknowledge the record.
- 33.
- 34.
- 35.
- 36. 36 min.insync.replicas minimum numberof replicas that must acknowledge. Default is 1.
- 37.
- 38. 38 38 Data Durability whileProducing ? Tune it with the parameters acks and min.insync.replicas
- 39. 39 defaults The defaultvalues are optimized for availability & latency. If durability is more important, tune it!
- 40. 40 40 Deploying on multidatacenters ?
- 41.
- 42. 42 Multi-dc It’s quitecomplicated... It’s easy to make it wrong on many levels. It could be a 3h talk.
- 43. 43 Multi-dc Disaster recoveryfor multi datacenter
- 44. 44 44 What about theconsumers ?
- 45. 45 consumers Consumer canread only committed data.
- 46.
- 47. 47 47 Think about datadurability and decide of the best trade-off for you
- 48. 48 Throughput, latency, durability, availability Optimizingyour Apache Kafka deployment
- 49. 49 49 Focusing only onthe happy path
- 50.
- 51.
- 52.
- 53. 53 retries It willcause the client to resend any record whose send fails with a potentially transient error. Default value : 0
- 54.
- 55. 55 retries Use builtin retries ! Bump it from 0 to infinity!
- 56. 56 retries But youare exposed to a different kind of issue…
- 57.
- 58. 58 enable.idempotence When setto 'true', the producer will ensure that exactly one copy of each message is written. Default value: false
- 59.
- 60.
- 61. 61 61 Use built inidempotency!
- 62. 62 62 But it doesnot save you from - Managing exception and failure - Developing Idempotent consumer
- 63.
- 64.
- 65. 65 65 At least once(default) At most once Exactly Once
- 66.
- 67.
- 68.
- 69.
- 70.
- 71.
- 72. 72 commit Manually committing aggressively... Adda huge workload on Apache Kafka
- 73.
- 74. 74 commit Manually committing aggressively... Doesnot provide exactly once semantic
- 75.
- 76. 76 76 Rely on KafkaStreams with Exactly Once !
- 77.
- 78.
- 79.
- 80. 80 Future<RecordMetadata> send(ProducerRecord<K, V>record, Callback callback);
- 81.
- 82. 82 error handling Wedon’t expect the unexpected until the unexpected is expected.
- 83.
- 84. 84 error handling Amessage can not be processed
- 85. 85 error handling Amessage can not be processed A message doesn’t have the expected schema
- 86.
- 87.
- 88.
- 89.
- 90. 90 90 Write to adead letter queue and continue
- 91.
- 92.
- 93.
- 94.
- 95. 95 95 Handle the exceptions! https://eng.uber.com/reliable-reprocessing/
- 96.
- 97.
- 98.
- 99.
- 100. 100 governance Changes inproducers might impact consumers
- 101.
- 102.
- 103.
- 104.
- 105. 105 105 Let bad citizenswander around
- 106.
- 107. 107 107 Leverage Security, ACLand Quota Security Authorization and ACLs Enforcing Client Quotas
- 108. 108 108 Installing prod onSunday night
- 109.
- 110. 110 configuration If youuse the default configuration… You will have issues!
- 111. 111 111 Please read thedoc Running Kafka in Production Running ZooKeeper in Production
- 112.
- 113.
- 114. 114 os Tune atleast your open file descriptors and mmap count.
- 115. 115 115 Configure your os RunningKafka in Production
- 116.
- 117.
- 118.
- 119.
- 120.
- 121.
- 122.
- 123.
- 124. 124 124 Call external servicesin Kafka Streams
- 125.
- 126.
- 127. 127 Set up secureKafka & build your first app Understand streaming Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Infrastructure & apps across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Self-service on shared Kafka Infrastructure & applications across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streamingUnderstand streaming Pre-streamingValue Stream Everything 05Break Silos 04 03 Go To Production 02 Learn Kafka 01 Investment & Time Solve A Critical Need Maturity model
- 128. 128 Set up secureKafka & build your first app Understand streaming Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Infrastructure & apps across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Self-service on shared Kafka Infrastructure & applications across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streamingUnderstand streaming Pre-streamingValue Stream Everything 05Break Silos 04 03 Go To Production 02 Learn Kafka 01 Solve A Critical Need Maturity model
- 129. 129 Set up secureKafka & build your first app Understand streaming Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Infrastructure & apps across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Self-service on shared Kafka Infrastructure & applications across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streamingUnderstand streaming Pre-streamingValue Stream Everything 05Break Silos 04 03 Go To Production 02 Learn Kafka 01 Solve A Critical Need Maturity model
- 130. 130 130 Not having thebusiness context!
- 131.
- 132. 132 132 2 DC –Stretched
- 133. 133 133 Questions ? Florent Ramière| florent@confluent.io Damien Gasparina | damien@confluent.io (Yes we answer your emails!)
Editor's Notes
- #5 This is how a company is built Event-driven architecture isn’t new, but different enough now that it is a new type of animal. If we’re successful this will be a major data platform in companies and will redefine the architecture of a digital company. The people here will be part of making that happen.
- #7 So what is a streaming platform? There are a set of core capabilities around data streams you have to have... The first is the ability to publish and subscribe to streams of data. This is something that’s been around for a long time. Messaging systems have been able to do this. What’s different now is the ability to store data and do it properly in a replicated manner. The final capability is to be able to process these streams of data. Initially starting off as a messaging system, over the years Kafka has evolved into a full-fledged distributed streaming platform that embodies the quintessential characteristics of this new category of infrastructure .