Event Sourcing and CQRS are two popular patterns for implementing a Microservices architectures. With Event Sourcing we do not store the state of an object, but instead store all the events impacting its state. Then to retrieve an object state, we have to read the different events related to a certain object and apply them one by one. CQRS (Command Query Responsibility Segregation) on the other hand is a way to dissociate writes (Command) and reads (Query). Event Sourcing and CQRS are frequently grouped and used together to form something bigger. While it is possible to implement CQRS without Event Sourcing, the opposite is not necessarily correct. In order to implement Event Sourcing, an efficient Event Store is needed. But is that also true when combining Event Sourcing and CQRS? And what is an event store in the first place and what features should it implement? This presentation will first discuss what functionalities an event store should offer and then present how Apache Kafka can be used to implement an event store. But is Kafka good enough or do specific event store solutions such as AxonDB or Event Store provide a better solution?
Kafka as an Event Store - is it Good Enough?Guido Schmutz
Event Sourcing and CQRS are two popular patterns for implementing a Microservices architectures. With Event Sourcing we do not store the state of an object, but instead store all the events impacting its state. Then to retrieve an object state, we have to read the different events related to a certain object and apply them one by one. CQRS (Command Query Responsibility Segregation) on the other hand is a way to dissociate writes (Command) and reads (Query). Event Sourcing and CQRS are frequently grouped and used together to form something bigger. While it is possible to implement CQRS without Event Sourcing, the opposite is not necessarily correct. In order to implement Event Sourcing, an efficient Event Store is needed. But is that also true when combining Event Sourcing and CQRS? And what is an event store in the first place and what features should it implement? This presentation will first discuss what functionalities an event store should offer and then present how Apache Kafka can be used to implement an event store. But is Kafka good enough or do specific event store solutions such as AxonDB or Event Store provide a better solution?
Event Driven Architecture with a RESTful Microservices Architecture (Kyle Ben...confluent
Tinder’s Quickfire Pipeline powers all things data at Tinder. It was originally built using AWS Kinesis Firehoses and has since been extended to use both Kafka and other event buses. It is the core of Tinder’s data infrastructure. This rich data flow of both client and backend data has been extended to service a variety of needs at Tinder, including Experimentation, ML, CRM, and Observability, allowing backend developers easier access to shared client side data. We perform this using many systems, including Kafka, Spark, Flink, Kubernetes, and Prometheus. Many of Tinder’s systems were natively designed in an RPC first architecture.
Things we’ll discuss decoupling your system at scale via event-driven architectures include:
– Powering ML, backend, observability, and analytical applications at scale, including an end to end walk through of our processes that allow non-programmers to write and deploy event-driven data flows.
– Show end to end the usage of dynamic event processing that creates other stream processes, via a dynamic control plane topology pattern and broadcasted state pattern
– How to manage the unavailability of cached data that would normally come from repeated API calls for data that’s being backfilled into Kafka, all online! (and why this is not necessarily a “good” idea)
– Integrating common OSS frameworks and libraries like Kafka Streams, Flink, Spark and friends to encourage the best design patterns for developers coming from traditional service oriented architectures, including pitfalls and lessons learned along the way.
– Why and how to avoid overloading microservices with excessive RPC calls from event-driven streaming systems
– Best practices in common data flow patterns, such as shared state via RocksDB + Kafka Streams as well as the complementary tools in the Apache Ecosystem.
– The simplicity and power of streaming SQL with microservices
Building Event-Driven (Micro) Services with Apache KafkaGuido Schmutz
This talk begins with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each eachother in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Kafka as an event store - is it good enough?Guido Schmutz
Event Sourcing and CQRS are two popular patterns for implementing a Microservices architectures. With Event Sourcing we do not store the state of an object, but instead store all the events impacting its state. Then to retrieve an object state, we have to read the different events related to a certain object and apply them one by one. CQRS (Command Query Responsibility Segregation) on the other hand is a way to dissociate writes (Command) and reads (Query). Event Sourcing and CQRS are frequently grouped and used together to form something bigger. While it is possible to implement CQRS without Event Sourcing, the opposite is not necessarily correct. In order to implement Event Sourcing, an efficient Event Store is needed. But is that also true when combining Event Sourcing and CQRS? And what is an event store in the first place and what features should it implement?
This presentation will first discuss what functionalities an event store should offer and then present how Apache Kafka can be used to implement an event store. But is Kafka good enough or do specific event store solutions such as AxonDB or Event Store provide a better solution?
A banking system has some special constraints that are very hard to achieve together: High scalability, availability, auditing, and specially, dealing with parallelism respecting the invariances of a typical bank account. In this proposal, Kafka Platform is the backbone of this solution, but It’s not alone. To deal fully with the constraints, a combination with others technologies is necessary and Akka is the main chosen partner. The talk will contextualize the problem, how the Kafka Platform deal with it and the gaps filled by Akka.
Batch and streaming visualization in big data reference architecture, architecture blueprints for streaming visualization, implementations of the blueprints in a fast data solution.
Most data visualisation solutions today still work on data sources which are stored persistently in a data store, using the so called “data at rest” paradigms. More and more data sources today provide a constant stream of data, from IoT devices to Social Media streams. These data stream publish with high velocity and messages often have to be processed as quick as possible. For the processing and analytics on the data, so called stream processing solutions are available. But these only provide minimal or no visualisation capabilities. Therefore, one can use a dedicated Streaming Visualisation solution. They are specially built for streaming data and often do not support batch data. A much better solution would be to have one tool capable of handling both, batch and streaming data. This talk presents different architecture blueprints for integrating data visualisation into a fast data solution and highlights some of the products available to implement these blueprints.
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesKai Wähner
Technical thought leadership presentation to discuss how leading organizations move to real-time architecture to support business growth and enhance customer experience. This is a forum to discuss use cases with your peers to understand how other digital-native companies are utilizing data in motion to drive competitive advantage.
Agenda:
- Data in Motion with Event Streaming and Apache Kafka
- Streaming ETL Pipelines
- IT Modernisation and Hybrid Multi-Cloud
- Customer Experience and Customer 360
- IoT and Big Data Processing
- Machine Learning and Analytics
Kafka as an Event Store - is it Good Enough?Guido Schmutz
Event Sourcing and CQRS are two popular patterns for implementing a Microservices architectures. With Event Sourcing we do not store the state of an object, but instead store all the events impacting its state. Then to retrieve an object state, we have to read the different events related to a certain object and apply them one by one. CQRS (Command Query Responsibility Segregation) on the other hand is a way to dissociate writes (Command) and reads (Query). Event Sourcing and CQRS are frequently grouped and used together to form something bigger. While it is possible to implement CQRS without Event Sourcing, the opposite is not necessarily correct. In order to implement Event Sourcing, an efficient Event Store is needed. But is that also true when combining Event Sourcing and CQRS? And what is an event store in the first place and what features should it implement? This presentation will first discuss what functionalities an event store should offer and then present how Apache Kafka can be used to implement an event store. But is Kafka good enough or do specific event store solutions such as AxonDB or Event Store provide a better solution?
Event Driven Architecture with a RESTful Microservices Architecture (Kyle Ben...confluent
Tinder’s Quickfire Pipeline powers all things data at Tinder. It was originally built using AWS Kinesis Firehoses and has since been extended to use both Kafka and other event buses. It is the core of Tinder’s data infrastructure. This rich data flow of both client and backend data has been extended to service a variety of needs at Tinder, including Experimentation, ML, CRM, and Observability, allowing backend developers easier access to shared client side data. We perform this using many systems, including Kafka, Spark, Flink, Kubernetes, and Prometheus. Many of Tinder’s systems were natively designed in an RPC first architecture.
Things we’ll discuss decoupling your system at scale via event-driven architectures include:
– Powering ML, backend, observability, and analytical applications at scale, including an end to end walk through of our processes that allow non-programmers to write and deploy event-driven data flows.
– Show end to end the usage of dynamic event processing that creates other stream processes, via a dynamic control plane topology pattern and broadcasted state pattern
– How to manage the unavailability of cached data that would normally come from repeated API calls for data that’s being backfilled into Kafka, all online! (and why this is not necessarily a “good” idea)
– Integrating common OSS frameworks and libraries like Kafka Streams, Flink, Spark and friends to encourage the best design patterns for developers coming from traditional service oriented architectures, including pitfalls and lessons learned along the way.
– Why and how to avoid overloading microservices with excessive RPC calls from event-driven streaming systems
– Best practices in common data flow patterns, such as shared state via RocksDB + Kafka Streams as well as the complementary tools in the Apache Ecosystem.
– The simplicity and power of streaming SQL with microservices
Building Event-Driven (Micro) Services with Apache KafkaGuido Schmutz
This talk begins with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each eachother in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Kafka as an event store - is it good enough?Guido Schmutz
Event Sourcing and CQRS are two popular patterns for implementing a Microservices architectures. With Event Sourcing we do not store the state of an object, but instead store all the events impacting its state. Then to retrieve an object state, we have to read the different events related to a certain object and apply them one by one. CQRS (Command Query Responsibility Segregation) on the other hand is a way to dissociate writes (Command) and reads (Query). Event Sourcing and CQRS are frequently grouped and used together to form something bigger. While it is possible to implement CQRS without Event Sourcing, the opposite is not necessarily correct. In order to implement Event Sourcing, an efficient Event Store is needed. But is that also true when combining Event Sourcing and CQRS? And what is an event store in the first place and what features should it implement?
This presentation will first discuss what functionalities an event store should offer and then present how Apache Kafka can be used to implement an event store. But is Kafka good enough or do specific event store solutions such as AxonDB or Event Store provide a better solution?
A banking system has some special constraints that are very hard to achieve together: High scalability, availability, auditing, and specially, dealing with parallelism respecting the invariances of a typical bank account. In this proposal, Kafka Platform is the backbone of this solution, but It’s not alone. To deal fully with the constraints, a combination with others technologies is necessary and Akka is the main chosen partner. The talk will contextualize the problem, how the Kafka Platform deal with it and the gaps filled by Akka.
Batch and streaming visualization in big data reference architecture, architecture blueprints for streaming visualization, implementations of the blueprints in a fast data solution.
Most data visualisation solutions today still work on data sources which are stored persistently in a data store, using the so called “data at rest” paradigms. More and more data sources today provide a constant stream of data, from IoT devices to Social Media streams. These data stream publish with high velocity and messages often have to be processed as quick as possible. For the processing and analytics on the data, so called stream processing solutions are available. But these only provide minimal or no visualisation capabilities. Therefore, one can use a dedicated Streaming Visualisation solution. They are specially built for streaming data and often do not support batch data. A much better solution would be to have one tool capable of handling both, batch and streaming data. This talk presents different architecture blueprints for integrating data visualisation into a fast data solution and highlights some of the products available to implement these blueprints.
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesKai Wähner
Technical thought leadership presentation to discuss how leading organizations move to real-time architecture to support business growth and enhance customer experience. This is a forum to discuss use cases with your peers to understand how other digital-native companies are utilizing data in motion to drive competitive advantage.
Agenda:
- Data in Motion with Event Streaming and Apache Kafka
- Streaming ETL Pipelines
- IT Modernisation and Hybrid Multi-Cloud
- Customer Experience and Customer 360
- IoT and Big Data Processing
- Machine Learning and Analytics
Architecting Microservices Applications with Instant Analyticsconfluent
View recording here: https://www.confluent.io/online-talks/architecting-microservices-applications-with-instant-analytics
The next generation architecture for exploring and visualizing event-driven data in real-time requires the right technology. Microservices deliver significant deployment and development agility, but raise questions of how data will move between services and how it will be analyzed. This online talk explores how Apache Druid and Apache Kafka® can turn a microservices ecosystem into a distributed real-time application with instant analytics. Apache Kafka and Druid form the backbone of an architecture that meet the demands imposed on the next generation applications you are building right now. Join industry experts Tim Berglund, Confluent, and Rachel Pedreschi, Imply, as they discuss architecting microservices apps with Druid and Apache Kafka.
Building event-driven Microservices with Kafka EcosystemGuido Schmutz
This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each each other in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing system can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Use Apache Gradle to Build and Automate KSQL and Kafka Streams (Stewart Bryso...confluent
KSQL is an easy-to-use and easy-to-understand streaming SQL engine for Apache Kafka built on top of Kafka Streams. The ability to write streaming applications using only SQL makes Apache Kafka available to a whole range of new developers and potential use cases, either as a stand-alone solution, or as a single component to a broader Kafka Streams implementation. Inspired by a customer project now in production, experience the lifecycle of a streaming application developed using KSQL and Kafka Streams. With Apache Gradle as our build framework, we’ll explore the open-source Gradle plugin we built during this project to improve developer efficiency and automate the deployment of KSQL pipelines, user-defined functions, and Kafka Streams microservices.
We’ll demonstrate the deployment process live, and discuss design decisions around incorporating SQL-based processes into an overall streaming application.
Key Takeaways
1. KSQL is a natural choice for expressing data-driven applications, but it may not naturally fit into established DevOps processes and automations.
2. We built an open-source Gradle plugin to handle all aspects of deploying a Kafka-based streaming application: KSQL pipelines, KSQL user-defined functions, and Kafka Streams microservices.
3. KSQL pipelines can be deployed using either a server start script, or the KSQL REST API, and our Gradle plugin fully supports both options.
Jay Kreps is the CEO of Confluent, Inc., a company backing the popular Apache Kafka® messaging system. Prior to founding Confluent, he was formerly the lead architect for data infrastructure at LinkedIn. He is among the original authors of several open source projects including Project Voldemort (a key-value store). Apache Kafka (a distributed messaging system) and Apache Samza (a stream processing system).
Ingesting streaming data into Graph DatabaseGuido Schmutz
This talk presents the experience of a customer project where we built a stream-based ingestion into a graph database. It is one thing to load the graph first and then querying it. But it is another story if the data to be added to the graph is constantly streaming in, while querying it. Data is easy to add, if each single message ends up as a new vertex in the graph. But if a message consists of hierarchical information, it most often means creating multiple new vertices as well adding edges to connect this information. What if a node already exists in the graph? Do we create it again or do we rather add edges which link to the existing node? Creating multiple nodes for the same real-life entity is not the best choice, so we have to check for existence first. We end up requiring multiple operations against the graph, which demonstrated to be a bottle neck. This talk presents the implementation of an ingestion pipeline and the design choice we made to improve performance.
A guide through the Azure Messaging services - Update ConferenceEldert Grootenboer
https://www.updateconference.net/en/2019/session/a-guide-through-the-azure-messaging-services
A guide through the Azure Messaging services - Update Conference
Most data visualization solutions today still work on the “data at rest” paradigm, where data is persisted first and then analyzed. But data sources today often come as a constant stream of data, from IoT devices to Social Media streams. These data streams publish information with high velocity and messages often have to be processed as quick as possible. For the processing and analytics, so called stream processing solutions are available. But these only provide minimal or no visualization capabilities. So how do we solve the visualization of high velocity data streams? An option is to first persist the data and then use a traditional data visualization solution to present the data. If latency is not an issue, this might be good enough. A NoSQL database might be an ideal solution, but not all traditional visualization tools might easily integrate with the specific data store. Another option is to use a dedicated Streaming Visualization solution. They are specially built for streaming data but on the other hand often do not support batch data. This talk presents different architecture blueprints for visualizing fast data and shows some products for implementing these blueprints.
Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will dig into how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to
hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Building event-driven (Micro)Services with Apache Kafka Guido Schmutz
What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will start with quick recap of how we created systems over the past 20 years and how different architectures evolved from it. The talk will show how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Event Hub (i.e. Kafka) in Modern Data (Analytics) ArchitectureGuido Schmutz
Today's modern data architectures and the their implementations contain an Event Hub. What are the benefits of placing an Event Hub in a Modern Data (Analytics) Architecture? What exactly is an Event Hub and what capabilities should it provide? Why is Apache Kafka the most popular realization of an Event Hub? These and many other questions will be answered in this session. The talk will start with a vendor-neutral definition of the capabilities of an Event Hub. Then the session will highlight the different architecture styles which can be supported using an Event Hub (Kafka), such as Streaming Data Integration, Stream Analytics and Decoupled Event-Driven Applications and how can these be combined into a unified architecture, making the Event Hub the central nervous system of an enterprise architecture. We will end with an overview of the Kafka ecosystem and a placement of the various components onto the Modern Data (Analytics) Architecture.
Building event-driven (Micro)Services with Apache KafkaGuido Schmutz
This talk begins with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each eachother in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. Storing such huge event streams into HDFS or a NoSQL datastore is feasible and not such a challenge anymore. But if you want to be able to react fast, with minimal latency, you can not afford to first store the data and doing the analysis/analytics later. You have to be able to include part of your analytics right after you consume the data streams. Products for doing event processing, such as Oracle Event Processing or Esper, are avaialble for quite a long time and used to be called Complex Event Processing (CEP). In the past few years, another family of products appeared, mostly out of the Big Data Technology space, called Stream Processing or Streaming Analytics. These are mostly open source products/frameworks such as Apache Storm, Spark Streaming, Flink, Kafka Streams as well as supporting infrastructures such as Apache Kafka. In this talk I will present the theoretical foundations for Stream Processing, discuss the core properties a Stream Processing platform should provide and highlight what differences you might find between the more traditional CEP and the more modern Stream Processing solutions.
Ingesting and Processing IoT Data - using MQTT, Kafka Connect and KSQLGuido Schmutz
Internet of Things use cases are a perfect match for processing with a streaming platform such as Kafka and the Confluent Platform. Some of the questions to be answered are: How do we feed the data from our devices into Kafka? Do we directly send data to Kafka? Is Kafka accessible from outside the organization over the internet? What if we want to use a more specific IoT protocol such as MQTT or CoAP in between? How would we integrate it with Kafka? How can we enrich IoT streaming data with static data sitting in a traditional system?
This session will provide answers to these and other questions using a fictitious use case of a trucking company. Trucks are constantly sending data about position and driving habits, which can be used to derive real-time information and actions. A large part of the presentation will be a live demo. The demo will show the implementation of the pipeline incrementally: starting with sending the truck movement events directly to Kafka, then adding MQTT to the sensor data ingestion, followed by using Kafka Streams and KSQL to apply stream processing on the information received. The final pipeline will demonstrate the application of Kafka Connect with MQTT and JDBC source connectors for data ingestion and event stream enrichment, and Kafka Streams and KSQL for stream processing. The key takeaway is the live demonstration of a working end-to-end IoT streaming data ingestion pipeline using Kafka technologies.
Jun Rao, Confluent | Kafka Summit SF 2019 Keynote ft. Chris Kasten, Walmart Labsconfluent
Jun Rao, Confluent Co-Founder discusses the power of Kafka, why it was created at LinkedIn, and what it's used for at Kafka Summit SF 2019's keynote. Featuring Chris Kasten, VP Walmart Cloud.
Building Event-Driven Applications with Apache Kafka & Confluent Platformconfluent
Watch this talk here: https://www.confluent.io/online-talks/building-event-driven-applications-apache-kafka-and-confluent-platform
Apache Kafka® has become the de facto technology for real-time event streaming. Confluent Platform, developed by the creators of Apache Kafka, is an event-streaming platform that enables the ingest and processing of massive amounts of data in real time.
In this session, we will cover the easiest ways to start developing event-driven applications with Apache Kafka using Confluent Platform. We will also demo a contextual event-driven application built using our ecosystem of connectors, REST proxy, and a variety of native clients.
View now to learn:
-How to create Apache Kafka topics in minutes and process event streams in real time
-Check the health of an Apache Kafka broker using Confluent Control Center
-The latest enhancements to Confluent Platform that make it easier to run Apache Kafka at scale
-How to use KSQL, streaming SQL for Apache Kafka, to process event streams in real time using simple SQL queries
More info: https://cnfl.io/cloud-native-experience-for-kafka-in-cloud | Neha Narkhede is co-founder and CTO at Confluent, a company backing the popular Apache Kafka messaging system. Prior to founding Confluent, Neha led streams infrastructure at LinkedIn, where she was responsible for LinkedIn’s streaming infrastructure built on top of Apache Kafka and Apache Samza. She is one of the initial authors of Apache Kafka and a committer and PMC member on the project.
Building Event Driven (Micro)services with Apache KafkaGuido Schmutz
What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will start with quick recap of how we created systems over the past 20 years and how different architectures evolved from it. The talk will show how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so.
Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Developing event-driven microservices with event sourcing and CQRS (phillyete)Chris Richardson
Modern, cloud-native applications typically use a microservices architecture in conjunction with NoSQL and/or sharded relational databases. However, in order to successfully use this approach you need to solve some distributed data management problems including how to maintain consistency between multiple databases without using 2PC. In this talk you will learn more about these issues and how to solve them by using an event-driven architecture. We will describe how event sourcing and Command Query Responsibility Separation (CQRS) are a great way to realize an event-driven architecture. You will learn about a simple yet powerful approach for building, modern, scalable applications.
Building microservices with Scala, functional domain models and Spring Boot (...Chris Richardson
In this talk you will learn about a modern way of designing applications that’s very different from the traditional approach of building monolithic applications that persist mutable domain objects in a relational database.We will talk about the microservice architecture, it’s benefits and drawbacks and how Spring Boot can help. You will learn about implementing business logic using functional, immutable domain models written in Scala. We will describe event sourcing and how it’s an extremely useful persistence mechanism for persisting functional domain objects in a microservices architecture.
Architecting Microservices Applications with Instant Analyticsconfluent
View recording here: https://www.confluent.io/online-talks/architecting-microservices-applications-with-instant-analytics
The next generation architecture for exploring and visualizing event-driven data in real-time requires the right technology. Microservices deliver significant deployment and development agility, but raise questions of how data will move between services and how it will be analyzed. This online talk explores how Apache Druid and Apache Kafka® can turn a microservices ecosystem into a distributed real-time application with instant analytics. Apache Kafka and Druid form the backbone of an architecture that meet the demands imposed on the next generation applications you are building right now. Join industry experts Tim Berglund, Confluent, and Rachel Pedreschi, Imply, as they discuss architecting microservices apps with Druid and Apache Kafka.
Building event-driven Microservices with Kafka EcosystemGuido Schmutz
This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each each other in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing system can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Use Apache Gradle to Build and Automate KSQL and Kafka Streams (Stewart Bryso...confluent
KSQL is an easy-to-use and easy-to-understand streaming SQL engine for Apache Kafka built on top of Kafka Streams. The ability to write streaming applications using only SQL makes Apache Kafka available to a whole range of new developers and potential use cases, either as a stand-alone solution, or as a single component to a broader Kafka Streams implementation. Inspired by a customer project now in production, experience the lifecycle of a streaming application developed using KSQL and Kafka Streams. With Apache Gradle as our build framework, we’ll explore the open-source Gradle plugin we built during this project to improve developer efficiency and automate the deployment of KSQL pipelines, user-defined functions, and Kafka Streams microservices.
We’ll demonstrate the deployment process live, and discuss design decisions around incorporating SQL-based processes into an overall streaming application.
Key Takeaways
1. KSQL is a natural choice for expressing data-driven applications, but it may not naturally fit into established DevOps processes and automations.
2. We built an open-source Gradle plugin to handle all aspects of deploying a Kafka-based streaming application: KSQL pipelines, KSQL user-defined functions, and Kafka Streams microservices.
3. KSQL pipelines can be deployed using either a server start script, or the KSQL REST API, and our Gradle plugin fully supports both options.
Jay Kreps is the CEO of Confluent, Inc., a company backing the popular Apache Kafka® messaging system. Prior to founding Confluent, he was formerly the lead architect for data infrastructure at LinkedIn. He is among the original authors of several open source projects including Project Voldemort (a key-value store). Apache Kafka (a distributed messaging system) and Apache Samza (a stream processing system).
Ingesting streaming data into Graph DatabaseGuido Schmutz
This talk presents the experience of a customer project where we built a stream-based ingestion into a graph database. It is one thing to load the graph first and then querying it. But it is another story if the data to be added to the graph is constantly streaming in, while querying it. Data is easy to add, if each single message ends up as a new vertex in the graph. But if a message consists of hierarchical information, it most often means creating multiple new vertices as well adding edges to connect this information. What if a node already exists in the graph? Do we create it again or do we rather add edges which link to the existing node? Creating multiple nodes for the same real-life entity is not the best choice, so we have to check for existence first. We end up requiring multiple operations against the graph, which demonstrated to be a bottle neck. This talk presents the implementation of an ingestion pipeline and the design choice we made to improve performance.
A guide through the Azure Messaging services - Update ConferenceEldert Grootenboer
https://www.updateconference.net/en/2019/session/a-guide-through-the-azure-messaging-services
A guide through the Azure Messaging services - Update Conference
Most data visualization solutions today still work on the “data at rest” paradigm, where data is persisted first and then analyzed. But data sources today often come as a constant stream of data, from IoT devices to Social Media streams. These data streams publish information with high velocity and messages often have to be processed as quick as possible. For the processing and analytics, so called stream processing solutions are available. But these only provide minimal or no visualization capabilities. So how do we solve the visualization of high velocity data streams? An option is to first persist the data and then use a traditional data visualization solution to present the data. If latency is not an issue, this might be good enough. A NoSQL database might be an ideal solution, but not all traditional visualization tools might easily integrate with the specific data store. Another option is to use a dedicated Streaming Visualization solution. They are specially built for streaming data but on the other hand often do not support batch data. This talk presents different architecture blueprints for visualizing fast data and shows some products for implementing these blueprints.
Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will dig into how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to
hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Building event-driven (Micro)Services with Apache Kafka Guido Schmutz
What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will start with quick recap of how we created systems over the past 20 years and how different architectures evolved from it. The talk will show how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Event Hub (i.e. Kafka) in Modern Data (Analytics) ArchitectureGuido Schmutz
Today's modern data architectures and the their implementations contain an Event Hub. What are the benefits of placing an Event Hub in a Modern Data (Analytics) Architecture? What exactly is an Event Hub and what capabilities should it provide? Why is Apache Kafka the most popular realization of an Event Hub? These and many other questions will be answered in this session. The talk will start with a vendor-neutral definition of the capabilities of an Event Hub. Then the session will highlight the different architecture styles which can be supported using an Event Hub (Kafka), such as Streaming Data Integration, Stream Analytics and Decoupled Event-Driven Applications and how can these be combined into a unified architecture, making the Event Hub the central nervous system of an enterprise architecture. We will end with an overview of the Kafka ecosystem and a placement of the various components onto the Modern Data (Analytics) Architecture.
Building event-driven (Micro)Services with Apache KafkaGuido Schmutz
This talk begins with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each eachother in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. Storing such huge event streams into HDFS or a NoSQL datastore is feasible and not such a challenge anymore. But if you want to be able to react fast, with minimal latency, you can not afford to first store the data and doing the analysis/analytics later. You have to be able to include part of your analytics right after you consume the data streams. Products for doing event processing, such as Oracle Event Processing or Esper, are avaialble for quite a long time and used to be called Complex Event Processing (CEP). In the past few years, another family of products appeared, mostly out of the Big Data Technology space, called Stream Processing or Streaming Analytics. These are mostly open source products/frameworks such as Apache Storm, Spark Streaming, Flink, Kafka Streams as well as supporting infrastructures such as Apache Kafka. In this talk I will present the theoretical foundations for Stream Processing, discuss the core properties a Stream Processing platform should provide and highlight what differences you might find between the more traditional CEP and the more modern Stream Processing solutions.
Ingesting and Processing IoT Data - using MQTT, Kafka Connect and KSQLGuido Schmutz
Internet of Things use cases are a perfect match for processing with a streaming platform such as Kafka and the Confluent Platform. Some of the questions to be answered are: How do we feed the data from our devices into Kafka? Do we directly send data to Kafka? Is Kafka accessible from outside the organization over the internet? What if we want to use a more specific IoT protocol such as MQTT or CoAP in between? How would we integrate it with Kafka? How can we enrich IoT streaming data with static data sitting in a traditional system?
This session will provide answers to these and other questions using a fictitious use case of a trucking company. Trucks are constantly sending data about position and driving habits, which can be used to derive real-time information and actions. A large part of the presentation will be a live demo. The demo will show the implementation of the pipeline incrementally: starting with sending the truck movement events directly to Kafka, then adding MQTT to the sensor data ingestion, followed by using Kafka Streams and KSQL to apply stream processing on the information received. The final pipeline will demonstrate the application of Kafka Connect with MQTT and JDBC source connectors for data ingestion and event stream enrichment, and Kafka Streams and KSQL for stream processing. The key takeaway is the live demonstration of a working end-to-end IoT streaming data ingestion pipeline using Kafka technologies.
Jun Rao, Confluent | Kafka Summit SF 2019 Keynote ft. Chris Kasten, Walmart Labsconfluent
Jun Rao, Confluent Co-Founder discusses the power of Kafka, why it was created at LinkedIn, and what it's used for at Kafka Summit SF 2019's keynote. Featuring Chris Kasten, VP Walmart Cloud.
Building Event-Driven Applications with Apache Kafka & Confluent Platformconfluent
Watch this talk here: https://www.confluent.io/online-talks/building-event-driven-applications-apache-kafka-and-confluent-platform
Apache Kafka® has become the de facto technology for real-time event streaming. Confluent Platform, developed by the creators of Apache Kafka, is an event-streaming platform that enables the ingest and processing of massive amounts of data in real time.
In this session, we will cover the easiest ways to start developing event-driven applications with Apache Kafka using Confluent Platform. We will also demo a contextual event-driven application built using our ecosystem of connectors, REST proxy, and a variety of native clients.
View now to learn:
-How to create Apache Kafka topics in minutes and process event streams in real time
-Check the health of an Apache Kafka broker using Confluent Control Center
-The latest enhancements to Confluent Platform that make it easier to run Apache Kafka at scale
-How to use KSQL, streaming SQL for Apache Kafka, to process event streams in real time using simple SQL queries
More info: https://cnfl.io/cloud-native-experience-for-kafka-in-cloud | Neha Narkhede is co-founder and CTO at Confluent, a company backing the popular Apache Kafka messaging system. Prior to founding Confluent, Neha led streams infrastructure at LinkedIn, where she was responsible for LinkedIn’s streaming infrastructure built on top of Apache Kafka and Apache Samza. She is one of the initial authors of Apache Kafka and a committer and PMC member on the project.
Building Event Driven (Micro)services with Apache KafkaGuido Schmutz
What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will start with quick recap of how we created systems over the past 20 years and how different architectures evolved from it. The talk will show how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so.
Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Developing event-driven microservices with event sourcing and CQRS (phillyete)Chris Richardson
Modern, cloud-native applications typically use a microservices architecture in conjunction with NoSQL and/or sharded relational databases. However, in order to successfully use this approach you need to solve some distributed data management problems including how to maintain consistency between multiple databases without using 2PC. In this talk you will learn more about these issues and how to solve them by using an event-driven architecture. We will describe how event sourcing and Command Query Responsibility Separation (CQRS) are a great way to realize an event-driven architecture. You will learn about a simple yet powerful approach for building, modern, scalable applications.
Building microservices with Scala, functional domain models and Spring Boot (...Chris Richardson
In this talk you will learn about a modern way of designing applications that’s very different from the traditional approach of building monolithic applications that persist mutable domain objects in a relational database.We will talk about the microservice architecture, it’s benefits and drawbacks and how Spring Boot can help. You will learn about implementing business logic using functional, immutable domain models written in Scala. We will describe event sourcing and how it’s an extremely useful persistence mechanism for persisting functional domain objects in a microservices architecture.
YOW2018 - Events and Commands: Developing Asynchronous MicroservicesChris Richardson
The microservice architecture functionally decomposes an application into a set of services. Each service has its own private database that’s only accessible indirectly through the services API. Consequently, implementing queries and transactions that span multiple services is challenging.
In this presentation, you will learn how to solve these distributed data management challenges using asynchronous messaging. I describe how to implement transactions using sagas, which are sequences of local transactions, coordinated using messages. You will learn how to implement queries using Command Query Responsibility Segregation (CQRS), which uses events to maintain replicas. I describe how to use event sourcing, which is an event-centric approach to business logic and persistence, in a microservice architecture.
Building and deploying microservices with event sourcing, CQRS and Docker (QC...Chris Richardson
In this talk we share our experiences developing and deploying a microservices-based application. You will learn about the distributed data management challenges that arise in a microservices architecture. We will describe how we solved them using event sourcing to reliably publish events that drive eventually consistent workflows and pdate CQRS-based views. You will also learn how we build and deploy the application using a Jenkins-based deployment pipeline that creates Docker images that run on Amazon EC2.
Building microservices with Scala, functional domain models and Spring BootChris Richardson
In this talk you will learn about a modern way of designing applications that’s very different from the traditional approach of building monolithic applications that persist mutable domain objects in a relational database.We will talk about the microservice architecture, it’s benefits and drawbacks and how Spring Boot can help. You will learn about implementing business logic using functional, immutable domain models written in Scala. We will describe event sourcing and how it’s an extremely useful persistence mechanism for persisting functional domain objects in a microservices architecture.
SVCC Developing Asynchronous, Message-Driven Microservices Chris Richardson
The microservice architecture functionally decomposes an application into a set of services. Each service has its own private database that’s only accessible indirectly through the services API. Consequently, implementing queries and transactions that span multiple services is challenging.
In this presentation, you will learn how to solve these distributed data management challenges using asynchronous messaging. I describe how to implement transactions using sagas, which are sequences of local transactions, coordinated using messages. You will learn how to implement queries using Command Query Responsibility Segregation (CQRS), which uses events to maintain replicas. I describe the key role that messaging plays a microservice architecture.
Given at Silicon Valley Code Camp 2018
#JaxLondon: Building microservices with Scala, functional domain models and S...Chris Richardson
In this talk you will learn about a modern way of designing applications that’s very different from the traditional approach of building monolithic applications that persist mutable domain objects in a relational database.We will talk about the microservice architecture, it’s benefits and drawbacks and how Spring Boot can help. You will learn about implementing business logic using functional, immutable domain models written in Scala. We will describe event sourcing and how it’s an extremely useful persistence mechanism for persisting functional domain objects in a microservices architecture.
Building and deploying microservices with event sourcing, CQRS and Docker (Ha...Chris Richardson
In this talk we share our experiences developing and deploying a microservices-based application. You will learn about the distributed data management challenges that arise in a microservices architecture. We will describe how we solved them using event sourcing to reliably publish events that drive eventually consistent workflows and pdate CQRS-based views. You will also learn how we build and deploy the application using a Jenkins-based deployment pipeline that creates Docker images that run on Amazon EC2.
Developing Event-driven Microservices with Event Sourcing & CQRS (gotoams)Chris Richardson
Modern, cloud-native applications typically use a microservices architecture in conjunction with NoSQL and/or sharded relational databases. However, in order to successfully use this approach you need to solve some distributed data management problems including how to maintain consistency between multiple databases without using 2PC.
In this talk you will learn more about these issues and how to solve them by using an event-driven architecture. We will describe how event sourcing and Command Query Responsibility Separation (CQRS) are a great way to realize an event-driven architecture. You will learn about a simple yet powerful approach for building, modern, scalable applications.
Mucon: Not Just Events: Developing Asynchronous MicroservicesChris Richardson
The microservice architecture functionally decomposes an application into a set of services. Each service has its own private database that’s only accessible indirectly through the services API. Consequently, implementing queries and transactions that span multiple services is challenging. In this presentation, you will learn how to solve these distributed data management challenges using asynchronous messaging. Chris will share with you how to implement transactions using sagas, which are sequences of local transactions. You will learn how to coordinate sagas using either events or command messages. Chris will also explore how to implement queries using Command Query Responsibility Segregation (CQRS), which uses events to maintain easily queried replicas.
Building and deploying microservices with event sourcing, CQRS and Docker (Me...Chris Richardson
In this talk we share our experiences developing and deploying a microservices-based application. You will learn about the distributed data management challenges that arise in a microservices architecture. We will describe how we solved them using event sourcing to reliably publish events that drive eventually consistent workflows and pdate CQRS-based views. You will also learn how we build and deploy the application using a Jenkins-based deployment pipeline that creates Docker images that run on Amazon EC2.
GotoChgo 2019: Not Just Events: Developing Asynchronous MicroservicesChris Richardson
The microservice architecture functionally decomposes an application into a set of services. Each service has its own private database that’s only accessible indirectly through the services API. Consequently, implementing queries and transactions that span multiple services is challenging.
In this presentation, you will learn how to solve these distributed data management challenges using asynchronous messaging. Chris will share with you how to implement transactions using sagas, which are sequences of local transactions. You will learn how to coordinate sagas using either events or command messages. Chris will also explore how to implement queries using Command Query Responsibility Segregation (CQRS), which uses events to maintain easily queried replicas.
Developing event-driven microservices with event sourcing and CQRS (svcc, sv...Chris Richardson
Modern, cloud-native applications typically use a microservices architecture in conjunction with NoSQL and/or sharded relational databases. However, in order to successfully use this approach you need to solve some distributed data management problems including how to maintain consistency between multiple databases without using 2PC.
In this talk you will learn more about these issues and how to solve them by using an event-driven architecture. We will describe how event sourcing and Command Query Responsibility Segregation (CQRS) are a great way to realize an event-driven architecture. You will learn about a simple yet powerful approach for building, modern, scalable applications.
Explore the various options for streaming data on AWS, such as Amazon Kinesis and Amazon Managed Streaming for Kafka, and the various options for processing streams of data such as Apache Spark, Apache Flink, AWS Lambda, and Amazon Kinesis Analytics for Java. Let's explore what an architecture for processing Australia's new Open Banking data format at 60,000 transactions per second could look like.
Seamless Guest Experience with Kafka Streams (Ramaraju Indukurir and Himani A...confluent
The cruise industry has unique set of challenges when it comes to deploying products that directly interact with guests across ships. No matter what the challenges are, Applications that run on ships, always need to provide same functionality as on shore. To name few, Each ship is almost a separate datacenter and there are tens of ships. Second, the ships compute capacity is much lower than what an AWS data center can provide and hence any thing that is deployed on ship need to be lean.Third, the bandwidth, though improving, is still a big constraint in transferring data between ship and shore. Fourth, The company’s legacy systems power the core booking functionality and that information is still vital to every feature or application that happens across enterprise including ships.
Hence traditionally cruise industry shied away from real time integration and mostly resorts to a ‘rollover’ process, where in the data gets transferred at the beginning or end of the cruise. But this can’t be the solution with digital. Mobiles need to allow stream of changes real time, as guests go from ship to shore to ship or ship to ship or shore to ship. Our customer, a leading cruise company instituted a team to solve these synchronization issues and also enable event driven architecture. This presentation will address the evolution, idea and implementation of data synchronization and enabling event driven organization using Kafka and streams technologies at its core.
Andrii Dembitskyi "Events in our applications Event bus and distributed systems"Fwdays
События являются довольно сильным инструментом для приложений:
коммуникация между компонентами системы;
история действий над данными;
триггеры для операций;
интеграция с посторонними системами.
Во время доклада я расскажу о применениях событий. На какие грабли можно наступить спеша в выборе инструмента и какое место они имеют в нашей архитектуре.
Events on the outside, on the inside and at the core (jfokus jfokus2016)Chris Richardson
This is the talk I gave at JFokus 2016 on event-driven microservices.
This presentation looks at the importance of events and the role that they play in applications. We describe how events are a key application integration mechanism and how they are used by applications to communicate with the outside world. You will learn how the microservices inside a system can use events to maintain data consistency. We discuss how easy it is to implement both of these mechanisms by developing your core business logic using an event-centric approach known as event sourcing.
Developing functional domain models with event sourcing (sbtb, sbtb2015)Chris Richardson
Event sourcing persists each entity as a sequence of state changing event. An entity’s current state is derived by replaying the events. Event sourcing is a great way to implement event-driven micro services. When one service updates an entity, the new events are consumed by other services, which then update their own state. In this talk we describe how to implement business logic using a domain model that is based on event sourcing. You will learn how to write functional, immutable domain models in Scala. We will compare and contrast a hybrid OO/FP design with a purely functional approach. You will learn how Domain Driven Design concepts such as bounded contexts and aggregates fit in with event-driven microservices.
Similar to Kafka as an Event Store (Guido Schmutz, Trivadis) Kafka Summit NYC 2019 (20)
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent
In our exclusive webinar, you'll learn why event-driven architecture is the key to unlocking cost efficiency, operational effectiveness, and profitability. Gain insights on how this approach differs from API-driven methods and why it's essential for your organization's success.
Unlocking the Power of IoT: A comprehensive approach to real-time insightsconfluent
In today's data-driven world, the Internet of Things (IoT) is revolutionizing industries and unlocking new possibilities. Join Data Reply, Confluent, and Imply as we unveil a comprehensive solution for IoT that harnesses the power of real-time insights.
Workshop híbrido: Stream Processing con Flinkconfluent
El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real.
Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real.
En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...confluent
Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace.
In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms.
You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes.
Don't miss out on this opportunity to learn from industry experts and take your business to the next level.
La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.
Eventos y Microservicios - Santander TechTalkconfluent
Durante esta sesión examinaremos cómo el mundo de los eventos y los microservicios se complementan y mejoran explorando cómo los patrones basados en eventos nos permiten descomponer monolitos de manera escalable, resiliente y desacoplada.
Purpose of the session is to have a dive into Apache, Kafka, Data Streaming and Kafka in the cloud
- Dive into Apache Kafka
- Data Streaming
- Kafka in the cloud
Build real-time streaming data pipelines to AWS with Confluentconfluent
Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.
Q&A with Confluent Professional Services: Confluent Service Meshconfluent
No matter whether you are migrating your Kafka cluster to Confluent Cloud, running a cloud-hybrid environment or are in a different situation where data protection and encryption of sensitive information is required, Confluent Service Mesh allows you to transparently encrypt your data without the need to make code changes to you existing applications.
Citi Tech Talk: Event Driven Kafka Microservicesconfluent
Microservices have become a dominant architectural paradigm for building systems in the enterprise, but they are not without their tradeoffs. Learn how to build event-driven microservices with Apache Kafka
Confluent & GSI Webinars series - Session 3confluent
An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities.
It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks.
This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.
Transforming applications built with traditional messaging solutions such as TIBCO, MQ and Solace to be scalable, reliable and ready for the move to cloud
How can applications built with traditional messaging technologies like TIBCO, Solace and IBM MQ be modernised and be made cloud ready? What are the advantages to Event Streaming approaches to pub/sub vs traditional message queues? What are the strengeths and weaknesses of both approaches, and what use cases and requirements are actually a better fit for messaging than Kafka?
This session will show why the old paradigm does not work and that a new approach to the data strategy needs to be taken. It aims to show how a Data Streaming Platform is integral to the evolution of a company’s data strategy and how Confluent is not just an integration layer but the central nervous system for an organisation
Vous apprendrez également à :
• Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données
• Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience
• Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés
Confluent Partner Tech Talk with Synthesisconfluent
A discussion on the arduous planning process, and deep dive into the design/architectural decisions.
Learn more about the networking, RBAC strategies, the automation, and the deployment plan.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
The Metaverse and AI: how can decision-makers harness the Metaverse for their...Jen Stirrup
The Metaverse is popularized in science fiction, and now it is becoming closer to being a part of our daily lives through the use of social media and shopping companies. How can businesses survive in a world where Artificial Intelligence is becoming the present as well as the future of technology, and how does the Metaverse fit into business strategy when futurist ideas are developing into reality at accelerated rates? How do we do this when our data isn't up to scratch? How can we move towards success with our data so we are set up for the Metaverse when it arrives?
How can you help your company evolve, adapt, and succeed using Artificial Intelligence and the Metaverse to stay ahead of the competition? What are the potential issues, complications, and benefits that these technologies could bring to us and our organizations? In this session, Jen Stirrup will explain how to start thinking about these technologies as an organisation.
Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™UiPathCommunity
In questo evento online gratuito, organizzato dalla Community Italiana di UiPath, potrai esplorare le nuove funzionalità di Autopilot, il tool che integra l'Intelligenza Artificiale nei processi di sviluppo e utilizzo delle Automazioni.
📕 Vedremo insieme alcuni esempi dell'utilizzo di Autopilot in diversi tool della Suite UiPath:
Autopilot per Studio Web
Autopilot per Studio
Autopilot per Apps
Clipboard AI
GenAI applicata alla Document Understanding
👨🏫👨💻 Speakers:
Stefano Negro, UiPath MVPx3, RPA Tech Lead @ BSP Consultant
Flavio Martinelli, UiPath MVP 2023, Technical Account Manager @UiPath
Andrei Tasca, RPA Solutions Team Lead @NTT Data
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Epistemic Interaction - tuning interfaces to provide information for AI support
Kafka as an Event Store (Guido Schmutz, Trivadis) Kafka Summit NYC 2019
1. gschmutz
Kafka as an Event Store – is it Good
Enough?
Guido Schmutz
Kafka Summit New York – 2.4.2019
gschmutz http://guidoschmutz.wordpress.com
2. gschmutz
Agenda
1. How do we build applications traditionally?
2. CQRS & Event Sourcing
3. What exactly is an Event Store?
4. Implementing Event Store
5. Summary
Kafka as an Event Store – is it Good Enough?
3. gschmutz
Guido Schmutz
Working at Trivadis for more than 22 years
Oracle Groundbreaker Ambassador & Oracle ACE Director
Consultant, Trainer, Solution Architect for Java, Oracle, Microservices and
Big Data / Fast Data
Head of Trivadis Architecture Board
More than 30 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
151st edition
Kafka as an Event Store – is it Good Enough?
4. gschmutz
How do we build applications
traditionally?
Kafka as an Event Store – is it Good Enough?
5. gschmutz
Data Access Layer
Monolithic Applications – using Layered Architecture
User Interface
Account UI
Service Layer
{ }
Account API
Database
Customer API
REST
Customer UI
Domain
Model
Account DAO
{ }REST
Customer DAO
DB Model
Kafka as an Event Store – is it Good Enough?
6. gschmutz
Data Access Layer
Monolithic Applications – using Layered Architecture
User Interface
Account UI
Service Layer
{ }
Account API
Database
Customer API
REST
Customer UI
Domain
Model
Account DAO
{ }REST
Customer DAO
DB Model
Object/Relational
Impedance Mismatch
• Traditional approach
to persistence
• Store current state
• CRUD operations
• Coupling between
read & write
• Increased Complexity
Kafka as an Event Store – is it Good Enough?
7. gschmutz
• Traditional approach
to persistence
• Store current state
• CRUD operations
• Coupling between
read & write
• Increased Complexity
Data Access Layer
Monolithic Applications – using Layered Architecture
User Interface
Account UI
Service Layer
{ }
Account API
Database
Customer API
REST
Customer UI
Domain
Model
Account DAO
{ }REST
Customer DAO
DB Model
Object/Relational Mapping
SELECT acc.id, acc.account_number, acc.account_type.id,
acct.name, cus.first_name, cus.last_name,
addr.street, addr.city
FROM account_t acc
, account_type_t acct
, customer_t cus
, cust_adr_t cusa
, address_t addr
WHERE acc.account_type_id = acct.account_type_id
AND adc.customer_id = cus.customer_id
AND cusa.customer_id = cus.customer_id
AND cusa.address_id = addr.address_id
AND cusa.type = 'MAIN'
Kafka as an Event Store – is it Good Enough?
8. gschmutz
Microservices - using Layered Architecture
Microservices …
• are responsible for their data
• might use NoSQL instead of
RDBMS
• often still use traditional approach
to persistence
• “Data silos” do no longer support
database join
• keep synchronous
communication to a minimum
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Account Microservice
{ }
Account API
Account
Account Logic
Product Microservice
{ }
Product API
Product
Product Logic
Finance App
Finance UI UI Logic
GUI
REST
REST
REST
Kafka as an Event Store – is it Good Enough?
sync request/response
async, event pub/sub
9. gschmutz
Domain Driven Design (DDD) – Concepts
Kafka as an Event Store – is it Good Enough?
Domain Objects – hold the state of the
application
Entity – Domain Objects with an identity
Value Object – an immutable type that is
distinguishable only by the state of its
properties and has no identity
Aggregate - A cluster of domain objects that
can be treated as a single unit
Aggregate Root – one object of aggregate is
root object. Any reference from outside goes
through aggregate root
Aggregate Root
Account Aggregate
Customer
Aggregate
Aggregate Root
10. gschmutz
Events
Distribute to all handlers
strong ordering req’s
No results
Queries
Route with load balancing
Sometimes scatter-gather
Provide result
Three mechanisms through which services can interact
Kafka as an Event Store – is it Good Enough?
Commands
Route to single handler
Use consistent hashing
Provide Result
Adapted from Axon IQ
11. gschmutz
Microservices with Event-driven communication
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Account Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
REST
REST
REST
Pub /
Sub
Customer
Mat View
Kafka as an Event Store – is it Good Enough?
sync request/response
async, event pub/sub
Finance App
Finance UI UI Logic
GUI
This is Event Streaming and not
really Event Sourcing
13. gschmutz
Command Query Responsibility Segregation (CQRS)
Optimize for write and read differently
API is split between
• commands - trigger changes in state
• queries - provide read access to the state
Still using CRUD pattern, but separates
”R” from CRUD
Might involve eventual consistency
between write and read model
Data Storage
Write Model
Read Model
(read-only)
Service
Command
API
Query
API
App
UI
Projection Handler
UI Logic
Kafka as an Event Store – is it Good Enough?
publish
command
query
project
read
insert
update
delete
1
2
3
4
14. gschmutz
Event Sourcing – Persist state-changing events and not
state
Kafka as an Event Store – is it Good Enough?
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000}
3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000}
4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500}
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
MoneyDeposited
id: 123
amount: 2000
MoneyWithdrawn
id: 123
amount: 500
10 20 100 2000
15. gschmutz
Event Sourcing
persists the state of an aggregate as a
sequence of state-changing events
Each event describes a state change
that occurred to the aggregate in the
past
new event is appended to the list of
events
an aggregate’s current state is
reconstructed by replaying the events
=> a.k.a ”rehydration”
Rehydration also needed for queries
Kafka as an Event Store – is it Good Enough?
Event Store
ServiceApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
Service
Subscribe
publish
publish
apply (append)
REST
Data Storage
trigger replycommand
command
1 2
3
4
5
5
16. gschmutz
Event Sourcing - ”Rehydrate” State
Kafka as an Event Store – is it Good Enough?
1. Create an empty Aggregate
object
2. Read all events stored for
that Aggregate from event
store
3. Apply each event to the
Aggregate object in the
correct order
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
MoneyDeposited
id: 123
amount: 2000
MoneyWithdrawn
id: 123
amount: 500
Account
<empty>
Account
id: 123
accountType: Savings
balance: 0
Account
id: 123
accountType: Savings
balance: 3000
transactions: [+1000, +2000]
Account
id: 123
accountType: Savings
balance: 2500
transactions:[+1000, +2000, -500]
Account
id: 123
accountType: Savings
balance: 1000
transactions: [+1000]
applyTo
applyTo
applyTo
applyTo
17. gschmutz
Event Sourcing – Write Path CreateAccount
command
Kafka as an Event Store – is it Good Enough?
Create an event for every state change of Aggregate
Persist the stream to event store (preserving event order)
AccountCreated
id: 123
accountType: Savings
10
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
Account Aggregate
<empty>
18. gschmutz
Event Sourcing – Write Path DepositMoney
command
Kafka as an Event Store – is it Good Enough?
Create an event for every state change of Aggregate
Persist the stream to event store (preserving event order)
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000}
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
10 20
Account Aggregate
id: 123
accountType: Savings
balance: 0
19. gschmutz
Event Sourcing – Write Path DepositMoney
command
Kafka as an Event Store – is it Good Enough?
Create an event for every state change of Aggregate
Persist the stream to event store (preserving event order)
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000}
3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000}
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
MoneyDeposited
id: 123
amount: 2000
10 20 100
Account Aggregate
id: 123
accountType: Savings
balance: 1000
20. gschmutz
Event Sourcing – Write Path WithdrawMoney
command
Kafka as an Event Store – is it Good Enough?
Create an event for every state change of Aggregate
Persist the stream to event store (preserving event order)
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000}
3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000}
4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500}
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
MoneyDeposited
id: 123
amount: 2000
MoneyWithdrawn
id: 123
amount: 500
10 20 100 2000
Account Aggregate
id: 123
accountType: Savings
balance: 3000
21. gschmutz
Event Sourcing - Potential Benefits
Kafka as an Event Store – is it Good Enough?
1. Subscribe to changes from other
Aggregates
2. Examine a historical record of every
change that has ever been applied on
the model
3. Use the event store data for trend,
forcast and other business analytics
4. Consider “what if” questions by
replaying events to Aggregates which
have experimental enhancements
5. Patch errors by adding ”correction”
events (if it is legally allowed)
6. Perform “undo” and “redo”
operations by replying varying sets
of Events
22. gschmutz
Event Sourcing & CQRS
Event sourcing is commonly combined
with the CQRS pattern
Combines best of Event Sourcing and
CQRS
Project events published by Event
Store into Read Model (Materialized
Views)
Write Model and Read Model might
only support eventual consistency
Kafka as an Event Store – is it Good Enough?
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection Handler
publish
command
query read
project
1
Event Store
publish
apply (append)
trigger reply
2
3
4
5
5
6
23. gschmutz
What is an Event Store?
Kafka as an Event Store – is it Good Enough?
24. gschmutz
Event Store Capabilities
Kafka as an Event Store – is it Good Enough?
1. Append Events efficiently
2. Read aggregate’s events in order
3. Full Sequential Read (over all
aggregates)
4. Consistent writes
5. Event versioning
6. Subscribable event stream
7. Correction events (O)
8. Ingestion & event time, bi-temporal (O)
9. Adhoc-Query on event store (O)
10. Snapshot Optimization (O)
11. High-Availability and Reliability (O)
26. gschmutz
Event Store Implementations
Kafka as an Event Store – is it Good Enough?
• Event Store (https://eventstore.org/) – by Greg Young
• Axon Framework & Relational DB (https://axoniq.io/) - by Axon IQ
• Axon DB (https://axoniq.io/) - by Axon IQ
• Eventuate (https://eventuate.io/) – by Eventuate.io
• Serialized (https://serialized.io/) – by Serialized.io
• Build your own ….
• Apache Kafka ???
28. gschmutz
Kafka as an Event Store
Kafka as an Event Store – is it Good Enough?
1. One, single-partitioned Kafka topic per Aggregate
2. One, partitioned Kafka topic per Aggregate Type
3. One single, highly partitioned Kafka topic for all Aggregate Types
Should you put several Event Types in the same Kafka topic?:
https://www.confluent.io/blog/put-several-event-types-kafka-topic/
29. gschmutz
1) One, single-partitioned Kafka topic per Aggregate
Instance
Kafka as an Event Store – is it Good Enough?
This will guarantee that the events are
stored in order
Reading state of an aggregate is as
simple as reading a topic from offset 0
Not really feasible as there will be just
too many topics needed
Kafka
Customer Aggregate
Account Aggregate
30. gschmutz
2) One, partitioned Kafka topic per Aggregate Type
Kafka as an Event Store – is it Good Enough?
Required number of partitions is dependent
on number of aggregate instances
Events are produced with aggregate-id as
the key
guarantees that events are stored in order
For reading state of an aggregate, all data
of all aggregate instances have to be
scanned => slow
Possible optimization: only read the
partition where aggregate instance is stored
Kafka
Customer Aggregate
Account Aggregate
31. gschmutz
3) One single, highly partitioned Kafka topic for all
Aggregate Types
Kafka as an Event Store – is it Good Enough?
Required number of partitions is dependent
on number of aggregate types * instances
Events are produced with aggregate-id as
the key
guarantees that events are stored in order
For reading state of an aggregate, all data
of all aggregate types & instances have to
be scanned => really slow
Possible optimization: only read the
partition where aggregate instance is stored
Kafka
Customer Aggregate
Account Aggregate
32. gschmutz
Kafka as an Event Store
Kafka as an Event Store – is it Good Enough?
# Capability Kafka Broker
1 Append events efficiently yes
2 Read aggregate’s events in order not efficiently
3 Full sequential Read yes
4 Consistent Writes no
5 Event Versioning yes (if Avro is used)
6 Subscribeable Event Stream yes
7 Correction events (O) no
8 Event time & ingestion time, aka. Bi-temporal (O) no, but extra time can be passed in header
9 Snapshot Optimization (O) no
10 Ad-Hoc Query on Events (O) no
11 High-Availability and Reliability (O) yes
33. gschmutz
Event Store
Kafka is not a Database … a Database is not Kafka
We can use Kafka to run part of our own
Event Store implementation
add a database to get missing capabilities
But be careful with Dual Write!
• Would need distributed transactions
• Otherwise no guarantee for both writes
to happen
Application
{ }
API DatabaseBiz Logic
REST
Event Hub
Kafka as an Event Store – is it Good Enough?
Other App
Consumer
34. gschmutz
Event Store
Kafka is not a Database … a Database is not Kafka
We can use Kafka to run our own Event
Store implementation
adding a database to get missing
capabilities
But be careful with Dual Write!
• Would need distributed transactions
• Otherwise no guarantee for both writes
to happen
Application
{ }
API DatabaseBiz Logic
REST
Event Hub
Kafka as an Event Store – is it Good Enough?
Other App
Consumer
35. gschmutz
Event StoreEvent Store
Two solutions for avoiding «dual write»
Write Event first then consume it to write
it to database
Write through database (CDC, outbox
design pattern)
Application
{ }
API
Database
Biz Logic
REST
Kafka as an Event Store – is it Good Enough?
Event Hub
Other App
Biz Logic
Application
{ }
API
Database
REST
Biz Logic
CDC
Event Hub
CDC
Connector
Other App
Biz Logic
Publish
37. gschmutz
Axon
Kafka as an Event Store – is it Good Enough?
• Spring Boot with Axon Framework for
Application
• MongoDB for Event Store
• Kafka Broker for Event Bus
• Kafka Streams or KSQL for
Projection Handler
• Kafka Connect / Spring Boot to
persist in read model
• NoSQL and/or RDBMS for read
model
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection Handler
publish
command
query read
project
Event Store
publish
apply (append)
trigger reply
38. gschmutz
Event Sourcing with Axon
Kafka as an Event Store – is it Good Enough?
Account
Events
Account
Command
Account Aggregate
Account Command
Response
Account App
Event Store
Account
Customer
Projection
Command Handler
Event Handler
Account Query
Projection Handler
Query Handler
Account Query
Account Query
Response
Customer
Event
https://github.com/gschmutz/various-demos/tree/master/event-sourcing
39. gschmutz
Event Sourcing with Axon - Aggregate
Kafka as an Event Store – is it Good Enough?
@Aggregate
public class AccountAggregate{
@AggregateIdentifier
private String id;
private BigDecimal balance;
private String forCustomerId;
private String accountType;
@CommandHandler
...
@EventSourcingHandler
...
40. gschmutz
Event Sourcing with Axon - Command Handler
Kafka as an Event Store – is it Good Enough?
@CommandHandler
public AccountAggregate(AccountCreateCommand command) {
Assert.hasLength(command.getForCustomerId(),
"CustomerId must have a value");
Assert.hasLength(command.getAccountType(),
"AccountType must have a value");
...
apply(new AccountCreatedEvent(command.getId(),
command.getForCustomerId(),
command.getAccountType(),
new BigDecimal("0")));
}
41. gschmutz
Event Sourcing with Axon – Command Handler
Kafka as an Event Store – is it Good Enough?
@CommandHandler
public void on(WithdrawMoneyCommand command) {
Assert.isTrue(command.getAmount() > 0,
"Amount should be a positive number");
if(command.getAmount().compareTo(this.balance) > 0 ) {
throw new InsufficientBalanceException(
"Insufficient balance. Trying to withdraw:" +
command.getAmount() +
", but current balance is: " + this.balance);
}
apply(new MoneyWithdrawnEvent(command.getId(),
command.getAmount()));
}
42. gschmutz
Event Sourcing with Axon – Event Handler
Kafka as an Event Store – is it Good Enough?
@EventSourcingHandler
public void handle(AccountCreatedEvent event) {
id = event.getId();
forCustomerId = event.getForCustomerId();
accountType = event.getAccountType();
balance = event.getBalance();
}
@EventSourcingHandler
public void handle(MoneyWithdrawnEvent event) {
balance = balance.subtract(event.getAmount());
}
43. gschmutz
Event Sourcing with Axon – Projection Handler
Kafka as an Event Store – is it Good Enough?
public class AccountQueryController {
@Autowired
private AccountRepository accRepo;
@EventHandler
public void on(AccountCreatedEvent event,@Timestamp Instant instant) {
Account account = new Account(event.getId(),event.getBalance(),
event.getAccHolder(),event.getAccHolderName(),
instant.toString());
accRepo.insert(account);
}
@EventHandler
public void on(MoneyDepositedEvent event,@Timestamp Instant instant) {
Account account = accRepo.findByAccountNo(event.getId());
account.setBalance(account.getBalance().add(event.getAmount()));
account.setLastUpdated(instant.toString());
accRepo.save(account);
}
44. gschmutz
Axon Fwk with Axon DB
Kafka as an Event Store – is it Good Enough?
• Spring Boot with Axon Framework
for Application
• Axon DB for Event Store and
Event Bus
• Spring Boot for Projection Handler
• Spring Boot to persist in read
model
• NoSQL and/or RDBMS for read
model
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection Handler
publish
command
query read
project
Event Store
publish
apply (append)
trigger reply
45. gschmutz
Axon as an Event Store
Kafka as an Event Store – is it Good Enough?
# Capability Axon Framework Axon Framework & Axon DB
1 Append events efficiently yes yes
2 Read aggregate’s events in order yes yes
3 Full sequential Read yes yes
4 Consistent Writes yes yes
5 Event Versioning yes yes
6 Subscribeable Event Stream yes yes
7 Correction events (O) no no
8 Event time & ingestion time, aka. Bi-temporal (O) no no
9 Snapshot Optimization (O) yes yes
10 Ad-Hoc Query on Events (O) yes yes
11 High-Availability and Reliability (O) possible yes
47. gschmutz
Kafka & Kafka Streams
Kafka as an Event Store – is it Good Enough?
Kafka Streams with State for Event
Store
Kafka Broker for Event Bus
Kafka Streams or KSQL for Projection
Handler
No reply of events, current snapshot is
held in state store
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection Handler
publish
command
query read
project
Event Store
publish
apply (append)
trigger reply
49. gschmutz
Kafka & Kafka Streams as an Event Store
Kafka as an Event Store – is it Good Enough?
# Capability Kafka & Kafka Streams
1 Append events efficiently yes
2 Read aggregate’s events in order no (snapshot state only holds current snapshot)
3 Full sequential Read no
4 Consistent Writes yes (only one event per aggregate in flight)
5 Event Versioning yes (if Avro used)
6 Subscribeable Event Stream yes
7 Correction events (O) no
8 Event time & ingestion time, aka. Bi-temporal (O) no
9 Snapshot Optimization (O) yes (snapshot state only)
10 Ad-Hoc Query on events (O) limited (KSQL, Presto on Kafka, Drill on Kafka, …)
11 High-Availability and Reliability (O) yes
51. gschmutz
Summary
• Event Sourcing and CQRS might be more natural to business people than IT => we
are used to work with “CRUD based persistence”
• Event Sourcing provides history and logging for free
• Kafka Broker alone is really “just” Event Streaming, not Event Sourcing
• Axon Framework supports the implementation of Event Sourcing applications with
Pluggable Event Store and Event Bus implementations
• Axon DB implements an Event Store and an Event Bus
• Kafka and Kafka Streams with State Store supports event sourcing in a ”streaming
fashion”
Kafka as an Event Store – is it Good Enough?
52. gschmutz
Technology on its own won't help you.
You need to know how to use it properly.
Kafka as an Event Store – is it Good Enough?