Recommended
Recommended
More Related Content
What's hot
What's hot (14)
Similar to React to Event-Driven World with IBM Event Streams & Apache Kafka
Similar to React to Event-Driven World with IBM Event Streams & Apache Kafka (20)
More from Grace Jansen
More from Grace Jansen (20)
Recently uploaded
Recently uploaded (20)
React to Event-Driven World with IBM Event Streams & Apache Kafka
- 1. Reacting to an Event-Driven world IBM Event StreamsApache Kafka GIDS.ARCHITECTURE Live 2020 Grace Jansen and Kate Stanley
- 2. The never-ending growth of data © 2020 IBM Corporation https://www.researchgate.net/figure/The-growth-of-structured-versus-unstructured-data-over-the-past-decade-41_fig1_335927263
- 3. The never-ending growth of data © 2020 IBM Corporation https://www.researchgate.net/figure/The-growth-of-structured-versus-unstructured-data-over-the-past-decade-41_fig1_335927263
- 4. Event Driven Architecture © 2020 IBM Corporation S1 S3 S2 S4 Event-driven messaging backboneMicroservice publishing events Microservice consuming events
- 5. Now for an example, let’s get some coffee! © 2020 IBM Corporation
- 6. © 2020 IBM Corporation https://github.com/cescoffier/quarkus-coffeeshop-demo
- 7. Barista Example: © 2020 IBM Corporation Coffee Lovers
- 8. Barista Example: © 2020 IBM Corporation Coffee Shop Coffee Lovers HTTP
- 9. Barista Example: © 2020 IBM Corporation Coffee Shop Barista Coffee Lovers HTTP HTTP
- 10. Barista Example: © 2020 IBM Corporation Coffee Shop Coffee Shop Barista Coffee Lovers Coffee Lovers HTTP HTTP HTTP
- 11. Barista Example: © 2020 IBM Corporation Coffee Shop Coffee Shop Barista Coffee Lovers Coffee Lovers HTTP Orders, Event Backbone HTTP HTTP
- 12. Barista Example: © 2020 IBM Corporation Coffee Shop Coffee Shop Barista Barista Coffee Lovers Coffee Lovers HTTP Orders, Queue Event Backbone HTTP HTTP
- 13. Barista Example: © 2020 IBM Corporation Coffee Shop Coffee Shop Barista Barista Board Coffee Lovers Coffee Lovers HTTP Orders, Queue Event Backbone HTTP HTTP
- 14. © 2020 IBM Corporation Q: Is your microservice system non- blocking and highly responsive?
- 15. © 2020 IBM Corporation Q: Is your microservice system non- blocking and highly responsive? A:Yes I’m using Kafka!
- 16. © 2020 IBM Corporation Q: Is your microservice system non- blocking and highly responsive? A:Yes I’m using Kafka!
- 17. © 2020 IBM Corporation Microservice 2 Microservice 3 Microservice 1 How do we make a highly responsive app?
- 18. © 2020 IBM Corporation Futures Libraries Reactive Streams Futures Libraries Reactive Streams Futures Reactive Streams Microservice 2 Microservice 3 Microservice 1 How do we make a highly responsive app? Libraries
- 19. © 2020 IBM Corporation Futures Libraries Reactive Streams Futures Reactive Streams Futures Reactive Streams Microservice 2 Microservice 3 Microservice 1 Reactive Programming How do we make a highly responsive app? Libraries Libraries
- 20. © 2020 IBM Corporation Futures Reactive Streams Futures Libraries Reactive Streams Futures Libraries Reactive Streams Microservice 2 Microservice 3 Microservice 1 Reactive Programming Reactive Systems How do we make a highly responsive app? Libraries
- 21. Reactive Programming © 2020 IBM Corporation A subset of asynchronous programming and a paradigm where the availability of new information drives the logic forward rather than having control flow driven by a thread-of-execution.
- 22. Reactive Programming Patterns © 2020 IBM Corporation Futures: a promise to hold the result of some operation once that operation completes Reactive programming libraries: for composing asynchronous and event-based programs. (e.g. RxJava, SmallRye Mutiny) Reactive Streams: a programming concept for handling asynchronous data streams in a non-blocking manner while providing backpressure to stream publishers
- 23. Reactive Manifesto © 2020 IBM Corporation
- 24. Reactive Manifesto © 2020 IBM Corporation Message- Driven
- 25. Reactive Manifesto © 2020 IBM Corporation ResilientElastic Message- Driven
- 26. Reactive Manifesto © 2020 IBM Corporation ResilientElastic Message- Driven Responsive
- 27. Reactive Architecture design patterns CQRS Event Sourcing Saga Sharding © 2020 IBM Corporation APP WRITE API APP READ API APP APP APP APP APP APP
- 28. How do I configure Kafka for a reactive system? © 2020 IBM Corporation
- 29. Event-Driven vs Message-Driven © 2020 IBM Corporation ResilientElastic Message- Driven Responsive
- 30. Reactive Systems rely on asynchronous message-passing © 2020 IBM Corporation
- 31. Messages “An item of data sent to a specific location.” Events “A signal emitted by a component upon reaching a given state.” © 2020 IBM Corporation A message can contain an encoded event in its payload.
- 32. Apache Kafka is an open source, distributed streaming platform © 2020 IBM Corporation Publish and subscribe to streams of records Store records in durable way Process streams of records as they occur
- 33. Resiliency in Kafka © 2020 IBM Corporation ResilientElastic Message- Driven Responsive
- 34. Resilient © 2020 IBM Corporation
- 35. Resilient © 2020 IBM Corporation
- 36. Resilient © 2020 IBM Corporation
- 37. Resilient © 2020 IBM Corporation KAFKA CLIENTS
- 38. Resilient Topic A Partition 1 Broker 1 © 2020 IBM Corporation
- 39. Resilient Topic A Partition 1 Broker 1 Topic A Partition 1 Offline Leader KAFKA CLIENTS © 2020 IBM Corporation
- 40. Resilient producers Delivery guarantees: At most once At least once Configuration: Acks Retries © 2020 IBM Corporation
- 41. Resilient consumers © 2020 IBM Corporation 0 1 2 3 4 5 Offset If a consumer dies it needs to know where to pick up from again Committing offsets: Manual Automatic
- 42. Baristas with auto commit Barista TOPIC Coffee Cappuccino Latte © 2020 IBM Corporation
- 43. Baristas with auto commit Barista TOPIC Coffee Cappuccino Latte Coffee © 2020 IBM Corporation
- 44. Baristas with auto commit Barista TOPIC Coffee Cappuccino Latte Coffee © 2020 IBM Corporation
- 45. Baristas with auto commit Barista TOPIC Coffee Cappuccino Latte © 2020 IBM Corporation
- 46. Baristas with auto commit Barista TOPIC Coffee Cappuccino Latte Cappuccino © 2020 IBM Corporation
- 47. Baristas with auto commit Barista TOPIC Coffee Cappuccino Latte Cappuccino Latte © 2020 IBM Corporation
- 48. Baristas with auto commit Barista TOPIC Coffee Cappuccino Latte Cappuccino Latte Baristas with manual commit Coffee Cappuccino Latte TOPIC Barista © 2020 IBM Corporation
- 49. Baristas with auto commit Barista TOPIC Coffee Cappuccino Latte Cappuccino Latte Baristas with manual commit Coffee Cappuccino Latte TOPIC Barista Coffee © 2020 IBM Corporation
- 50. Baristas with auto commit Barista TOPIC Coffee Cappuccino Latte Cappuccino Latte Baristas with manual commit Coffee Cappuccino Latte TOPIC Barista Coffee © 2020 IBM Corporation
- 51. Baristas with auto commit Barista TOPIC Coffee Cappuccino Latte Cappuccino Latte Baristas with manual commit Coffee Cappuccino Latte TOPIC Barista Coffee LatteCappuccino © 2020 IBM Corporation
- 52. Scalability in Kafka © 2020 IBM Corporation ResilientElastic Message- Driven Responsive
- 53. Scalability in Kafka © 2020 IBM Corporation
- 54. Elastic Producers 0 1 2 3 TOPIC 0 1 2 3 0 1 2 3 PARTITION 0 PARTITION 1 PARTITION 2 key:a val:A1 © 2020 IBM Corporation
- 55. Elastic Producers 0 1 2 3 4 TOPIC 0 1 2 3 0 1 2 3 4 PARTITION 0 PARTITION 1 PARTITION 2 4 © 2020 IBM Corporation
- 56. Elastic Producers 0 1 2 3 4 5 TOPIC 0 1 2 3 0 1 2 3 4 5 PARTITION 0 PARTITION 1 PARTITION 2 4 5 © 2020 IBM Corporation
- 57. Elasticity in Consumers © 2020 IBM Corporation
- 58. © 2019 IBM Corporation Consumer Groups CONSUMER GROUP A p0, offset 7 p1, offset 3 p2, offset 5 TOPIC 0 1 2 3 0 1 2 3 4 5 PARTITION 0 PARTITION 1 PARTITION 2 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER CONSUMER GROUP B p0, offset 7 p1, offset 3 p2, offset 5 CONSUMER CONSUMER © 2019 IBM Corporation
- 59. © 2019 IBM Corporation Consumer Groups CONSUMER GROUP A p0, offset 7 p1, offset 3 p2, offset 5 TOPIC 0 1 2 3 0 1 2 3 4 5 PARTITION 0 PARTITION 1 PARTITION 2 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER CONSUMER GROUP B p0, offset 7 p1, offset 3 p2, offset 5 CONSUMER CONSUMER © 2019 IBM Corporation
- 60. © 2019 IBM Corporation Consumer Groups CONSUMER GROUP A p0, offset 7 p1, offset 3 p2, offset 5 TOPIC 0 1 2 3 0 1 2 3 4 5 PARTITION 0 PARTITION 1 PARTITION 2 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER CONSUMER GROUP B p0, offset 7 p1, offset 3 p2, offset 5 CONSUMER CONSUMER © 2019 IBM Corporation
- 61. © 2019 IBM Corporation Consumer Groups CONSUMER GROUP A p0, offset 7 p1, offset 3 p2, offset 5 TOPIC 0 1 2 3 0 1 2 3 4 5 PARTITION 0 PARTITION 1 PARTITION 2 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER GROUP B p0, offset 7 p1, offset 3 p2, offset 5 CONSUMER CONSUMER © 2019 IBM Corporation
- 62. How do we get started?
- 63. © 2020 IBM Corporation https://kafka.apache.org/quickstart
- 64. © 2020 IBM Corporation
- 65. Reactive Frameworks for Kafka Reactor Kafka MicroProfile Reactive Messaging Alpakka Kafka Connector Vert.x Kafka Client © 2020 IBM Corporation
- 66. Reactor Kafka © 2020 IBM Corporation Reactor Kafka is a thin layer of reactive streams over Kafka client Hides the complexity of Kafka Built-in backpressure support
- 67. MicroProfile Reactive Messaging © 2020 IBM Corporation This specification provides asynchronous messaging support based on Reactive Streams for MicroProfile. Implementations include: • SmallRye Reactive Messaging 1.0.0 • Open Liberty 19.0.0.9 (via SmallRye) • Quarkus 1.0.0.Final (via SmallRye) • WebSphere Liberty 19.0.0.9 (via SmallRye) Reactive Messaging uses a model of annotated methods which are connected by named channels.
- 68. Alpakka Kafka Connector © 2020 IBM Corporation Enables connection between Apache Kafka and Akka Streams. Hides complexity of Kafka Built-in backpressure support Akka Streams uses Akka actors as its foundation
- 69. What is Eclipse Vert.x? Polyglot Tool-kit Based on Reactor pattern Runs on the JVM Non-blocking Event-driven Includes distributed event-bus Code is single-threaded © 2020 IBM Corporation
- 70. Why? © 2020 IBM Corporation Simplified Kafka APIs that are reactive Built-in back pressure Enabling asynchronous, per-record processing
- 71. Demo app © 2020 IBM Corporation https://github.com/ibm-messaging/kafka-java-vertx-starter Vert.x app produce consume
- 72. Demo app © 2020 IBM Corporation https://github.com/ibm-messaging/kafka-java-vertx-starter Vert.x app websocket produce consume
- 73. Apache Kafka Java Client © 2020 IBM Corporation
- 74. Apache Kafka Java Client © 2020 IBM Corporation
- 75. Apache Kafka Java Client © 2020 IBM Corporation
- 76. Vert.x Client © 2020 IBM Corporation
- 77. Vert.x Client © 2020 IBM Corporation
- 78. © 2020 IBM Corporation https://github.com/ibm-messaging/kafka-java-vertx-starter
- 79. © 2020 IBM Corporation ibm.biz/ExperiencesWritingAReactiveKafkaApp
- 80. Summary Building reactive can help you! Kafka is useful, but think carefully when designing your system You are not alone! Lots of reactive frameworks to help you along the way © 2020 IBM Corporation
- 81. Reactive Systems Explained © 2020 IBM Corporation ibm.biz/ReactiveReport
- 82. MicroProfile Reactive Messaging Guide © 2020 IBM Corporation ibm.biz/ReactiveMessagingGuide
- 83. IBM Event Streams Apache Kafka for the enterprise © 2020 IBM Corporation ibm.biz/aboutEventStreams Powerful Ops Tooling Award Winning User Experience Connector Catalog Support you can Trust Unrivalled MQ connectivity Schema Registry Geo-replication for DR
- 84. Thank you Grace Jansen | @gracejansen27 Kate Stanley | @katestanley91 © 2020 IBM Corporation IBM Event Streams: ibm.biz/aboutEventStreams Reactive resources: https://www.reactivemanifesto.org https://ibm.biz/GettingStartedWithReactive Getting started with Kafka: https://kafka.apache.org/quickstart https://strimzi.io Reactive Kafka libraries https://vertx.io/docs/vertx-kafka-client/java/ https://github.com/eclipse/microprofile-reactive- messaging https://projectreactor.io/docs/kafka https://github.com/akka/alpakka-kafka http://heidloff.net/article/comparing-synchronous- asynchronous-access-postgresql/
Editor's Notes
- Start with our “journey” to microservices How do you architect your microservices so that your clients get a nice experience Talk about response time, data-driven level first, but we live in an event-driven world! Start with demo showing http vs Kafka (video) Show that Kafka is much quicker Why? – delve into code? Show that we are being event-driven, what does that mean, why is it quicker (timeout diagrams) Talk about data centric vs event-centric But this is only looking at the architecture, what is happening inside your microservices Leads into reactive intro Why reactive architecture exists, how it fits into Kafka, what are the cornerstones What happens if we set up Kafka in a non-reactive way? Ok let’s fix it so it is reactive, and now switch to a reactive app. At the end, running Kafka in reactive way and implementing with vertx, includes showing the vertx Kafka client etc. Run app in a container? Options for Kafka on Kube End resources Non-resilient, or non-elastic we could have failure at some point Non-resilient – only replicating on one broker Non-elastic – how does vertx do elasticity? First app is a basic Kafka client app, then later introduce vertx
- Every second, ~ 6,000 tweets are tweeted >40,000 Google queries are searched >2 million emails are sent Photo uploads total 300 million per day. Emphasizing how much data applications are expected to handle Also impact in terms of fluctuation e.g. black Friday Also people wanting to have split second responsiveness Banking apps -> needing up to date information (Internet Live Stats, a website of the international Real Time Statistics Project)
- Event Driven Architecture (EDA) is a popular architectural approach that enables events to be placed at the heart of our systems Consists of Events Events are records of something that has happened, a change in state - immutable and are ordered in sequence of their creation. Interested parties can be notified of these state changes by subscribing to published events and then acting on information using their chosen business logic. An event-driven architecture, refers to a system of loosely coupled microservices that exchange information between each other through the production and consumption of events.
- Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
- Clement 4pm C3
- Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better Clement’s session C3 – 4pm It is possible to do http requests without blocking the thread, but even with that switch you are still approaching from a request/response perspective
- Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better Clement’s session C3 – 4pm It is possible to do http requests without blocking the thread, but even with that switch you are still approaching from a request/response perspective
- Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better Clement’s session C3 – 4pm It is possible to do http requests without blocking the thread, but even with that switch you are still approaching from a request/response perspective
- Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
- Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
- Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
- Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
- No! This isn’t reasonable! Kafka is a good tool, but it isn’t enough to have a good tool, you need to use it in the right way You also need to think about your applications and other services, Kafka isn’t your whole architecture – integration between components is key! Can we just use Kafka to create a Reactive application? Short answer: NO While Kafka look after the messaging part, we still need a Reactive Microservice implementation, for instance, using the actor model to replace thread synchronization with queued message processing or the supervisor model to handle failures and self-healing. We definitely need both Akka and Kafka to build Reactive Microservices based responsive, resilient and elastic systems.
- No! This isn’t reasonable! Kafka is a good tool, but it isn’t enough to have a good tool, you need to use it in the right way You also need to think about your applications and other services, Kafka isn’t your whole architecture – integration between components is key! Can we just use Kafka to create a Reactive application? Short answer: NO While Kafka look after the messaging part, we still need a Reactive Microservice implementation, for instance, using the actor model to replace thread synchronization with queued message processing or the supervisor model to handle failures and self-healing. We definitely need both Akka and Kafka to build Reactive Microservices based responsive, resilient and elastic systems.
- No! This isn’t reasonable! Kafka is a good tool, but it isn’t enough to have a good tool, you need to use it in the right way You also need to think about your applications and other services, Kafka isn’t your whole architecture – integration between components is key! Can we just use Kafka to create a Reactive application? Short answer: NO While Kafka look after the messaging part, we still need a Reactive Microservice implementation, for instance, using the actor model to replace thread synchronization with queued message processing or the supervisor model to handle failures and self-healing. We definitely need both Akka and Kafka to build Reactive Microservices based responsive, resilient and elastic systems.
- Kafka = gives us reactive data layer G Reactive architecture patterns = give us reactivity in the architecture of the system Reactive programming = gives us reactivity within the microservices (Designed well together)
- Kafka = gives us reactive data layer G Reactive architecture patterns = give us reactivity in the architecture of the system Reactive programming = gives us reactivity within the microservices (Designed well together)
- Kafka = gives us reactive data layer G Reactive architecture patterns = give us reactivity in the architecture of the system Reactive programming = gives us reactivity within the microservices (Designed well together)
- Kafka = gives us reactive data layer G Reactive architecture patterns = give us reactivity in the architecture of the system Reactive programming = gives us reactivity within the microservices A reactive system is an architectural style that allows multiple individual applications to coalesce as a single unit, reacting to its surroundings, while remaining aware of each other—this could manifest as being able to scale up/down, load balancing, and even taking some of these steps proactively. It’s possible to write a single application in a reactive style (i.e. using reactive programming); however, that’s merely one piece of the puzzle. Though each of the above aspects may seem to qualify as “reactive,” in and of themselves they do not make a system reactive. (Designed well together)
- Asynchronous code allows independent IO operations to run concurrently, resulting in efficient code. However, this improved efficiency comes at a cost — straightforward synchronous code may become a mess of nested callbacks. Futures - Enables us to combine the simplicity of synchronous code with the efficiency of the asynchronous approach. Future represents the result of an asynchronous computation. Methods are provided to check if the computation is complete, to wait for its completion, and to retrieve the result of the computation. A Publisher is the source of events T in the stream, and a Subscriber is a consumer for those events. A Subscriber subscribes to a Publisher by invoking a “factory method” in the Publisher that will push the stream items <T> starting a new Subscription. This is also called Reactor Pattern.
- Asynchronous code allows independent IO operations to run concurrently, resulting in efficient code. However, this improved efficiency comes at a cost — straightforward synchronous code may become a mess of nested callbacks. Futures - Enables us to combine the simplicity of synchronous code with the efficiency of the asynchronous approach. Future represents the result of an asynchronous computation. Methods are provided to check if the computation is complete, to wait for its completion, and to retrieve the result of the computation. A Publisher is the source of events T in the stream, and a Subscriber is a consumer for those events. A Subscriber subscribes to a Publisher by invoking a “factory method” in the Publisher that will push the stream items <T> starting a new Subscription. This is also called Reactor Pattern.
- Reactive Manifest 2.0 Reactive architecture is an architecture approach aims to use asynchronous messaging or event driven architecture to build Responsive, Resilient and Elastic systems. Reactive Microservices is capitalizing on the Reactive approach while supporting faster time to market using Microservices. Reactive Microservices is using asynchronous messaging to minimize or isolate the negative effects of resource contention, coherency delays and inter-service communication network latency. By using an event driven architecture we can have both agile development and build responsive systems.
- Reactive Manifest 2.0 Reactive architecture is an architecture approach aims to use asynchronous messaging or event driven architecture to build Responsive, Resilient and Elastic systems. Reactive Microservices is capitalizing on the Reactive approach while supporting faster time to market using Microservices. Reactive Microservices is using asynchronous messaging to minimize or isolate the negative effects of resource contention, coherency delays and inter-service communication network latency. By using an event driven architecture we can have both agile development and build responsive systems.
- Reactive Manifest 2.0 Reactive architecture is an architecture approach aims to use asynchronous messaging or event driven architecture to build Responsive, Resilient and Elastic systems. Reactive Microservices is capitalizing on the Reactive approach while supporting faster time to market using Microservices. Reactive Microservices is using asynchronous messaging to minimize or isolate the negative effects of resource contention, coherency delays and inter-service communication network latency. By using an event driven architecture we can have both agile development and build responsive systems.
- Reactive Manifest 2.0 Reactive architecture is an architecture approach aims to use asynchronous messaging or event driven architecture to build Responsive, Resilient and Elastic systems. Reactive Microservices is capitalizing on the Reactive approach while supporting faster time to market using Microservices. Reactive Microservices is using asynchronous messaging to minimize or isolate the negative effects of resource contention, coherency delays and inter-service communication network latency. By using an event driven architecture we can have both agile development and build responsive systems.
- Reactive adopts a set of design patterns such as: - CQRS – separates the reads and writes - Event Sourcing - persists the state of a business entity such an Order or a Customer as a sequence of state-changing events. Whenever the state of a business entity changes, a new event is appended to the list of events. - SAGA - a mechanism to take traditional transactions that we would have done in a monolithic architecture and do it in a distributed way. We create multi “micro” transactions that have fallback behaviour to account for things going wrong part way through. It’s a sequence of local transactions where each transaction updates data within a single service - Sharding - distributes and replicates the data across a pool of databases that do not share hardware or software. Each individual database is known as a shard. Java applications can linearly scale up or scale down by adding databases (shard) to the pool or by removing databases (shards) from the pool. These patterns trade off eventual consistency, availability and scalability for strong consistency (CAP Theorem). Kafka is a perfect fit for those design patterns.
- So Kafka claims to have scalable consumption and resiliency, do I just get that for free when I start Kafka? How does it work?
- Talking about message driven vs event driven
- Ultimately, founders of Reactive manifesto believed that by switching from Event-Driven to Message-Driven, they could more accurately articulate and define the other traits. The difference being that messages are directed, events are not—a message has a clear addressable recipient while an event just happen for others (0-N) to observe it.
- Open sourced distributed streaming platform, often being adopted as the “de-facto” event streaming technology Arrived at the right time, captured mindshare among developers and so exploded in popularity Kafka has deliberately moved away from the word “events”… instead uses records now
- Reactive architecture is an architecture approach aims to use asynchronous messaging or event driven architecture to build Responsive, Resilient and Elastic systems. Reactive Microservices is capitalizing on the Reactive approach while supporting faster time to market using Microservices. Reactive Microservices is using asynchronous messaging to minimize or isolate the negative effects of resource contention, coherency delays and inter-service communication network latency. By using an event driven architecture we can have both agile development and build responsive systems.
- A Kafka cluster consists of a set of brokers. A cluster has a minimum of 3 brokers.
- Kafka broken down into topics Records on a topic split into different partitions Partitions distributed across Kafka brokers
- For each partition, one of the brokers is the leader, and the other brokers are the followers. Replication works by the followers repeatedly fetching messages from the leader. This is done automatically by Kafka. For production we recommend at least 3 replicas: you’ll see why in a minute.
- In order to improve availability, each topic can be replicated onto multiple brokers. For each partition, one of the brokers is the leader, and the other brokers are the followers. Replication works by the followers repeatedly fetching messages from the leader. This is done automatically by Kafka. For production we recommend at least 3 replicas: you’ll see why in a minute.
- Imagine a broker goes down, this means the leader of Topic A, partition 1 is offline
- Imagine a broker goes down, this means the leader of Topic A, partition 1 is offline
- Can’t do fire and forget if you want full resiliency cause if the broker goes down your messages get lost Two different guarantees, way you get them is through confirguartion At most once, you may lose some messages (not completely relisient) At least once, guaranteed delivery but may get duplicates Retries is if acks times out/fails – how many times do you retry producing the event (how will the retry affect ordering)
- Replacing scalable with elastic….Truly Reactive Systems should react to changes in the input rate by increasing or decreasing the resources allocated to service these inputs, not just expanded according to its usage (which is the definition of Scalable). Three different things to consider – Kafka itself, the consumers and the producers
- Elasticity in Kafka itself Scale out brokers, can’t scale down (where do events go if you did?) Can scale out partitions but can’t scale them down again Can add topics, and delete topics if you don’t care about them
- An individual record is made up of a key and a value. You can scale up producers and scale down producers as you want - it’s the key you attach to the records they produce that determine which partition they are assigned to and their ordering Kafka guarantee that all messages with the same non-empty key will be sent to the same partition
- When no key is set, producers records will be appended in a round-robin fashion
- An individual record is made up of a key and a value. When no key is set, producers records will be appended in a round-robin fashion
- To allow scalability of consumers, consumers are grouped into consumer groups. Consumer declare what group they are in using a group id For consumers we use Consumer groups to enable elasticity
- If you added an extra consumer to consumer group A it would sit idle, since there aren’t any spare partitions – this isn’t ideal but could be useful if you want it to quickly pick up the slack if one of the other consumers went down Key message – you can scale up and scale down consumers – CAVEAT! You can only scale up consumers to match the number of partitions So black Friday, make sure you have enough partitions! If you scale up consumers to more then partitions, you’ll have some sitting idle, only use of this is if a consumer goes down then you have a backup
- If you added an extra consumer to consumer group A it would sit idle, since there aren’t any spare partitions – this isn’t ideal but could be useful if you want it to quickly pick up the slack if one of the other consumers went down
- If you added an extra consumer to consumer group A it would sit idle, since there aren’t any spare partitions – this isn’t ideal but could be useful if you want it to quickly pick up the slack if one of the other consumers went down
- If you added an extra consumer to consumer group A it would sit idle, since there aren’t any spare partitions – this isn’t ideal but could be useful if you want it to quickly pick up the slack if one of the other consumers went down
- At the moment Kafka uses Zookeeper for storing metadata, in the future the metadata will be stored in topics in Kafka, so Zookeeper won’t be required any more.
- Reactive adopts a set of design patterns such as: - CQRS - event sourcing - command sourcing - sharding These patterns trade off eventual consistency, availability and scalability for strong consistency (CAP Theorem). Kafka is a perfect fit for those design patterns.
- Applications using Kafka as a message bus using this API may consider switching to Reactor Kafka if the application is implemented in a functional style. Based on top of Project Reactor Uses Kafka Java client (Kafka Producer/Consumer API) under the hood
- The actor model is a conceptual model to deal with concurrent computation. An actor is the primitive unit of computation. It’s the thing that receives a message and do some kind of computation based on it. Messages are sent asynchronously to an actor, that needs to store them somewhere while it’s processing another message. The mailbox is the place where these messages are stored. Actors communicate with each other by sending asynchronous messages. Those messages are stored in other actors' mailboxes until they're processed. It allows consuming/producing from Kafka with Akka Streams, leveraging the reactive interface of this streaming library, its backpressure, and resource safety. It hides a lot of complexity, especially when your streaming logic is non-trivial like sub-streaming per partition and handling commits in custom ways.
- Polyglot Java, Javascript, Groovy, Ceylon, Scala and Kotlin The reactor pattern is one implementation technique of event-driven architecture. In simple terms, it uses a single threaded event loop blocking on resource-emitting events and dispatches them to corresponding handlers and callbacks. It receives messages, requests, and connections coming from multiple concurrent clients and processes these posts sequentially using event handlers. The purpose of the Reactor design pattern is to avoid the common problem of creating a thread for each message, request, and connection. Then it receives events from a set of handlers and distributes them sequentially to the corresponding event handlers. It’s single-threaded – so you must not block the thread! The Kafka client is becoming more popular and e.g. it is used by SmallRye Reactive messaging
- Reactive adopts a set of design patterns such as: - CQRS - event sourcing - command sourcing - sharding These patterns trade off eventual consistency, availability and scalability for strong consistency (CAP Theorem). Kafka is a perfect fit for those design patterns.
- Demo the starter app working Key takeaways: Choosing a reactive framework makes it easier to work with Kafka Strimzi, cool open source project that provides a Kubernetes operator for Kafka, just been accepted into CNCF (Cloud Native Computer Foundation) Kate active contributer to Strimzi and I was interested in Vert.x
- Demo the starter app working Key takeaways: Choosing a reactive framework makes it easier to work with Kafka Strimzi, cool open source project that provides a Kubernetes operator for Kafka, just been accepted into CNCF (Cloud Native Computer Foundation) Kate active contributer to Strimzi and I was interested in Vert.x
- Show/talk about the normal way to use the Kafka clients
- Show/talk about the normal way to use the Kafka clients
- Show/talk about the normal way to use the Kafka clients
- Show/talk about the normal way to use the Kafka clients
- Show/talk about the normal way to use the Kafka clients
- IBM Event Streams is fully supported Apache Kafka® with value-add capabilities