Keep Your Cache Always Fresh With Debezium! With Gunnar Morling | Current 2022

•

0 likes•894 views

The saying goes that there are only two hard things in Computer Science: cache invalidation, and naming things. Well, turns out the first one is solved actually ;) Join us for this session to learn how to keep read views of your data in distributed caches close to your users, always kept in sync with your primary data stores change data capture. You will learn how to - Implement a low-latency data pipeline for cache updates based on Debezium, Apache Kafka, and Infinispan - Create denormalized views of your data using Kafka Streams and make them accessible via plain key look-ups from a cache cluster close by - Propagate updates between cache clusters using cross-site replication We'll also touch on some advanced concepts, such as detecting and rejecting writes to the system of record which are derived from outdated cached state, and show in a demo how all the pieces come together, of course connected via Apache Kafka.

Technology

Image © Nathalie https://flic.kr/p/21Ghf2g (CC BY 2.0)
Keep Your Cache Always Fresh With Debezium!
Gunnar Morling
Software Engineer, Red Hat
@gunnarmorling

#Debezium @gunnarmorling
The Challenge
Multi-site Application With Shared Database

#Debezium @gunnarmorling
The Challenge
● Deployments in different geographies, single DB
● 90% read requests
● Complex queries
Multi-site Application With Shared Database

#Debezium @gunnarmorling
… Multi-site application with shared system-of-record database
… With local, denormalized read views (CQRS)
... Automatically kept in sync after writes
Today’s Mission
🤔
Explore How to Build a…

#Debezium @gunnarmorling
● Software engineer at Red Hat
○ Debezium
○ Quarkus
● kcctl 🧸, JfrUnit, ModiTect, MapStruct
● Spec Lead for Bean Validation 2.0
● Java Champion
Gunnar Morling

#Debezium @gunnarmorling
The Idea
Caching to the Rescue!

https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot

#Debezium @gunnarmorling
Inﬁnispan
100% Open-source In-Memory Distributed Data Store
Interoperability
Resilient
Fault Tolerant Data
Clustered Processing Query
ACID Tx

#Debezium @gunnarmorling
App 1
Data
App 2
Data
1, Maria
2, Jenny
2, Jenny
3, Juan
App 3
Data
1, Maria
3, Juan
Inﬁnispan Deployment
Distributed Cache

#Debezium @gunnarmorling
App 1
Data
App 2
Data
1, Maria
2, Jenny
2, Jenny
3, Juan
App 3
Data
1, Maria
3, Juan
Put 4 Will Put 4 Will
Inﬁnispan Deployment
Distributed Cache

#Debezium @gunnarmorling
App 1
Data
App 2
Data
1, Maria
2, Jenny
4, Will
2, Jenny
3, Juan
4, Will
App 3
Data
1, Maria
3, Juan
Inﬁnispan Deployment
Distributed Cache

#Debezium @gunnarmorling
Inﬁnispan
Client/Server
Service 1
Service 2
Binary
(hot rod)
REST
...
Service 3
Data
Infinispan Cluster
Data
Data

#Debezium @gunnarmorling
Inﬁnispan Cross-Site Replication
AWS (LON)
GCP (NYC)
Load Balancer
APP
APP
Service
APP
APP
Service
Shared
State
Shared
State
Shared
State
Shared
State
Data
Data
Data NYC
Data LON
RELAY2

#Debezium @gunnarmorling
The Question
How To Keep The Cache In Sync?

#Debezium @gunnarmorling
Dual Writes
Easy to Get Wrong

https://flic.kr/p/PFDvkY Public Domain, Angelo Br

#Debezium @gunnarmorling
Debezium in a Nutshell
Open-Source Change Data Capture
● A CDC Platform
○ Based on transaction logs
○ Snapshotting, filtering, etc.
○ Outbox support
○ Web-based UI
● Fully open-source, very active
community
● Large production deployments

#Debezium @gunnarmorling
Debezium
Becoming the De-Facto CDC Standard
https://debezium.io/blog/2021/09/22/deep-dive-into-a-debezium-community-connector-scylla-cdc-source-connector/

#Debezium @gunnarmorling
Debezium: Deployment Alternatives
Embedded Engine and Debezium Server

#Debezium @gunnarmorling
Solution Overview
Capturing Changes From the Database

https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot
Demo

#Debezium @gunnarmorling
● Fast start-up, low memory consumption
● Developer joy
● Imperative and Reactive
● Best-of-breed libraries
● Run via HotSpot and GraalVM native binaries
Quarkus - Supersonic Subatomic Java
A Stack for Building Cloud-native Apps

#Debezium @gunnarmorling
Eventual Consistency
Dealing With Stale Data
● Cached views updated asynchronously
● How to ensure read your own writes semantics?
● How to prevent taking action based on stale data?
🤔

#Debezium @gunnarmorling
Eventual Consistency
Read Your Own Writes

#Debezium @gunnarmorling
Eventual Consistency
Detecting Writes Derived From Stale State

#Debezium @gunnarmorling
Discussion
What Have We Gained?
● Pro:
○ Lower latencies
○ Reduced load on primary database
○ Increased availability
● Con:
○ Increased complexity

#Debezium @gunnarmorling
● Incremental snapshotting
● Postgres logical decoding messages
● Multi-DB support (SQL Server)
● Debezium Server sinks
● MongoDB change streams support
● Debezium UI
● Debezium 2.0
What’s New in Debezium?

#Debezium @gunnarmorling
● Infinispan: @infinispan | https://infinispan.org/
● Debezium: @debezium | https://debezium.io/
● Demo: https://github.com/debezium/debezium-examples/ →
distributed-caching
● kcctl 🧸: https://github.com/kcctl/kcctl/
Learn More

#Debezium @gunnarmorling
Q & A
gunnar@hibernate.org
@gunnarmorling
📧
Thank You!

Keep Your Cache Always Fresh With Debezium! With Gunnar Morling | Current 2022

This document discusses how Quarkus brings serverless capabilities to Java developers. It begins with an introduction of the speaker and outlines the developer journey from monolith applications to microservices and serverless functions. It then discusses challenges with existing serverless offerings and how Kubernetes and Knative address these. Quarkus is presented as a solution for serverless Java by providing benefits like developer productivity, fast and lightweight applications, and unifying imperative and reactive programming. The talk concludes by highlighting some of Quarkus' supported frameworks and providing additional resources.

Collaborative environment with data science notebook

Moon Soo Lee

This document discusses how to build an efficient data science toolchain around notebook technologies. It describes how notebooks can be used for interactive analytics and collaboration. It recommends sharing notebooks and data to maximize their potential. Methods for sharing include GitHub, nbviewer, Apache Zeppelin, and commercial services. It also discusses enabling multi-user environments through JupyterHub and Zeppelin and building data catalogs for managing and sharing datasets.

Cloud Native CI/CD with Jenkins X and Knative Pipelines

C4Media

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2Pc3H50. Christie Wilson and James Rawlings explain the CI/CD challenges in a cloud native landscape, and show how Jenkins X rises to them by leveraging open source cloud native technologies like Knative Pipelines. They demo a GitOps based Jenkins X workflow, showing how simple Jenkins X makes it for developers to stage and deploy changes on demand. Filmed at qconlondon.com. Christie Wilson is a software engineer at Google, currently leading the knative build-pipeline project. Over the past ten years she has worked in the mobile, financial and video game industries. James Rawlings is a co-creator of the open source project Jenkins X and works for CloudBees, where he aims to help developers and teams move to the cloud.

Polyglot Persistence - Two Great Tastes That Taste Great Together

John Wood

The days of the relational database being a one-stop-shop for all of your persistence needs are over. Although NoSQL databases address some issues that can’t be addressed by relational databases, the opposite is true as well. The relational database offers an unparalleled feature set and rock solid stability. One cannot underestimate the importance of using the right tool for the job, and for some jobs, one tool is not enough. This talk focuses on the strength and weaknesses of both relational and NoSQL databases, the benefits and challenges of polyglot persistence, and examples of polyglot persistence in the wild. These slides were presented at WindyCityDB 2010.

Multiplier Effect: Case Studies in Distributions for Publishers

Jon Peck

Join members from both Four Kitchens and Meredith Agrimedia as they discuss the experience of migration and relaunch of the digital presence of two magazines: Successful Farming at Agriculture.com and WOOD Magazine at woodmagazine.com. We'll start by discussing the scope of the projects, delve into the commonalities and differences, explore their common advertising and analytics implementation, and analyze the unified distribution that supports both brands. By developing the infrastructure simultaneously, brand-agnostic functionality became a priority which in turn created a more modular and flexible system that facilitated open-sourcing and cross-organizational sharing. Thanks to the codebase approach and experience, the first site took about 6 months and the second took less than 6 weeks.

Enabling Cognitive Workloads on the Cloud: GPUs with Mesos, Docker and Marath...

Indrajit Poddar

This document provides an overview of enabling cognitive workloads on the cloud using GPUs with Mesos, Docker, and Marathon on IBM's POWER systems. It discusses requirements for GPUs in the cloud like exposing GPUs to containers and supporting multiple GPUs per node. It also summarizes Mesos and Kubernetes support for GPUs, and demonstrates running a deep learning workload on OpenPOWER hardware to identify dog breeds using Docker containers and GPUs.

Lets look at an example of what a performant website can look like. This discuss what concepts should we be considering when looking at website performance. Next we will go over two areas pertaining to website performance: 1) website performance tweaks that you as a web developer can directly make 2) website performance tweaks that you may have to work with your hosting provider or IT department to achieve

Introduction to Modern DevOps Technologies

Kriangkrai Chaonithi

This document provides an introduction to modern DevOps technologies. It discusses DevOps concepts like source code management using Git, different methods of deploying programs including using bare metal servers, virtualization, containers, and cloud functions. Specific container and container orchestration technologies like Docker and Kubernetes are explained. Continuous integration and continuous delivery (CI/CD) practices are also introduced. The presentation includes an agenda with slides on these topics and ends with a question and answer section and announcement of a Docker workshop to deploy an HTTP server container.

Building Web Mobile App that don’t suck - FITC Web Unleashed - 2014-09-18

Frédéric Harper

Mobility is everywhere. It’s even more important to think about smaller devices like smartphones when building application or game using web technology. Everybody wants to have a mobile application, but it's not sufficient: you need to create an awesome experience for your users, your customers. You need to be future proof, and think about different markets as users needs: you never know when your application will be the next Flappy Bird. This presentation is about concrete tips, tricks and guidelines for web developers using HTML, CSS, and JavaScript based on experience that will help you make a success of your next exciting mobile application or game idea.

Monolith vs Microservices with Golang at practice - Ivan Kutuzov

Kuberton

Improving Workflows With Grunt.js - Big D Design 2014 - Dallas Texas

Preston McCauley

The document discusses using Grunt.js to automate and improve workflows when building web sites and applications. It describes how Grunt.js can be used to speed up repetitive tasks like minification, compilation of CoffeeScript to JavaScript, compilation of LESS to CSS, running a web server, and automatically refreshing the browser on changes. The presenters then demonstrate setting up a Gruntfile to automate common tasks like compiling CoffeeScript and LESS, separating pre-compiled from production code, and watching for changes.

Conquer Architectural Challenges with End-to-End JavaScript - enterJS 2014

Alexandre Morgaut

CodiLime Tech Talk - Dawid Trzebiatowski i Wojciech Urbański: Opening the Flo...

CodiLime

Implementing data and databases on K8s within the Dutch government

DoKC

A small walkthrough of projects within the dutch government running Data(bases) on OpenShift. This talk shares success stories, provides a proven recipe to `get it done` and debunks some of the FUD. About Sebastiaan: I have always been a weird DBA, trying to combine Databases with out-of-the-box thinking and a DevOps mindset. Around 2016 I fell in love with both Postgres and Kubernetes, and I then committed my life to enabling Dutch organisations with running their Database workloads CloudNative. Over the last few years I worked as a private contractor for 2 large government agencies doing exactly that, and I want to share my and others (success stories) hoping to enable and inspire Data on Kubernetes adoption.

Why run Postgres in Kubernetes?

DoKC

Postgres should run inside your Kubernetes cluster. Yes, inside, not outside Kubernetes. After all, a database should be seen as an application, a special type of application - for which it is legitimate to require an additional level of care and attention. However, the small price you pay for this is worth the return on investment that your organization receives by running microservice applications, plus database combos, inside your Cloud Native infrastructure. In this session, I will cover why it is important to make such a decision. Discover the challenges and the opportunities that running Postgres inside Kubernetes presents. I will discuss what we all expect from a good Postgres operator, including self-healing, high availability, scalability, backup and recovery, performance, and - last but not least - security. I will then go beyond technical aspects to spark conversations about the holistic improvements that running a Cloud Native database brings to your organization. This talk was given by Gabriele Bartolini for DoK Day Europe @ KubeCon 2022.

Why run Postgres in Kubernetes?

DoKC

Link: https://youtu.be/XRD5-V6907A https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Postgres should run inside your Kubernetes cluster. Yes, inside, not outside Kubernetes. After all, a database should be seen as an application, a special type of application - for which it is legitimate to require an additional level of care and attention. However, the small price you pay for this is worth the return on investment that your organization receives by running microservice applications, plus database combos, inside your Cloud Native infrastructure. In this session, I will cover why it is important to make such a decision. Discover the challenges and the opportunities that running Postgres inside Kubernetes presents. I will discuss what we all expect from a good Postgres operator, including self-healing, high availability, scalability, backup and recovery, performance, and - last but not least - security. I will then go beyond technical aspects to spark conversations about the holistic improvements that running a Cloud Native database brings to your organization. ----- A long time open source programmer and entrepreneur, Gabriele has a degree in Statistics from the University of Florence. After having consistently contributed to the growth of 2ndQuadrant and its members through nurturing a lean and devops culture, he is now leading the Cloud Native initiative at EDB. Gabriele lives in Prato, a small but vibrant city located in the northern part of Tuscany, Italy - known for having hosted the first European PostgreSQL conferences. His second home is Melbourne, Australia, where he studied at Monash University and worked in the ICT sector. He loves playing the Blues with his Fender Stratocaster, but his major passions are called Elisabeth and Charlotte!

Modern Webapps

Dimitris Tsironis

The document discusses strategies for developing modern web applications. It recommends developing apps using a modular approach that decouples components into services that communicate over REST APIs. This allows front-end code like JavaScript apps to interface with back-end services like Rails or Django through a consistent JSON interface. It also emphasizes testing strategies like test-driven development, behavior-driven development, and continuous integration to deploy apps with confidence.

JHipster presentation by Gaetan Bloch

Gaëtan Bloch

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics

carlostorres15106

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics

Andrey Dotsenko

Scalable Clusters On Demand

Bogdan Kyryliuk

At Opendoor, we do a lot of big data processing, and use Spark and Dask clusters for the computations. Our machine learning platform is written in Dask and we are actively moving data ingestion pipelines and geo computations to PySpark. The biggest challenge is that jobs vary in memory, cpu needs, and the load in not evenly distributed over time, which causes our workers and clusters to be over-provisioned. In addition to this, we need to enable data scientists and engineers run their code without having to upgrade the cluster for every request and deal with the dependency hell. To solve all of these problems, we introduce a lightweight integration across some popular tools like Kubernetes, Docker, Airflow and Spark. Using a combination of these tools, we are able to spin up on-demand Spark and Dask clusters for our computing jobs, bring down the cost using autoscaling and spot pricing, unify DAGs across many teams with different stacks on the single Airflow instance, and all of it at minimal cost.

Lunch & Learn BigQuery & Firebase from other Google Cloud customers

Daniel Zivkovic

1) Migrating your on-prem #Enterprise #Data #Warehouse into the #Cloud? Here is what you need to learn (and unlearn) when designing a modern Cloud #DataWarehouse in #BigQuery! 2) Launching a #Startup? See how to supercharge your idea with #Firebase! Watch the recording at https://youtu.be/zezhXNqD0rs and more forward-looking talks on #Cloud #Architectures & #DataEngineering join http://ServerlessToronto.org User Group.

Cloud native - CI/CD

Elad Hirsch

The “rise” of the containers created very interesting opportunities for running and deploying micro-services and distributed software in general, like any good thing, it comes with a price. Building a Cloud-Native CI/CD infrastructure utilizing the advantages of containers is quite challenging. In this session, we will introduce the challenges of CI/CD in the cloud native world including building our CI/CD infrastructure as code and working with dynamic workers. We will explore popular projects aiming to help us with these challenges: Bloody Jenkins, Jenkins X and more.

Multimedia support in WebKitGTK and WPE, current status and plans (GStreamer ...

Igalia

By Philippe Normand. This talk is about multimedia support in the WPE and GTK+ WebKit ports. I will give a status update about the HTML5 features currently supported by our GStreamer backend, such as WebRTC, MSE, MediaCapabilities support. The talk would also include a brief case study about using WPE and its Cog browser on IMX6 platforms. (c) GStreamer Conference 2018 CC-BY-SA 3.0 https://gstreamer.freedesktop.org/conference/2018/ https://gstconf.ubicast.tv/videos/multimedia-support-in-webkitgtk-and-wpe-current-status-and-plans/

jhipster-geekle-gbloch

Gaëtan Bloch

Chrome Dev Summit Summary 2013 part 1 - what’s hot ?

Sacha Leprêtre

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

HostedbyConfluent

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way. Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks. Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case. Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

Renaming a Kafka Topic | Kafka Summit London

HostedbyConfluent

"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics. Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"

Similar to Keep Your Cache Always Fresh With Debezium! With Gunnar Morling | Current 2022

20 tips for website performance

Andrew Siemer

Introduction to Modern DevOps Technologies

Kriangkrai Chaonithi

Building Web Mobile App that don’t suck - FITC Web Unleashed - 2014-09-18

Frédéric Harper

Monolith vs Microservices with Golang at practice - Ivan Kutuzov

Kuberton

Improving Workflows With Grunt.js - Big D Design 2014 - Dallas Texas

Preston McCauley

Conquer Architectural Challenges with End-to-End JavaScript - enterJS 2014

Alexandre Morgaut

CodiLime Tech Talk - Dawid Trzebiatowski i Wojciech Urbański: Opening the Flo...

CodiLime

Implementing data and databases on K8s within the Dutch government

DoKC

Why run Postgres in Kubernetes?

DoKC

Why run Postgres in Kubernetes?

DoKC

Modern Webapps

Dimitris Tsironis

JHipster presentation by Gaetan Bloch

Gaëtan Bloch

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics

carlostorres15106

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics

Andrey Dotsenko

Scalable Clusters On Demand

Bogdan Kyryliuk

Lunch & Learn BigQuery & Firebase from other Google Cloud customers

Daniel Zivkovic

Cloud native - CI/CD

Elad Hirsch

Multimedia support in WebKitGTK and WPE, current status and plans (GStreamer ...

Igalia

jhipster-geekle-gbloch

Gaëtan Bloch

Chrome Dev Summit Summary 2013 part 1 - what’s hot ?

Sacha Leprêtre

Similar to Keep Your Cache Always Fresh With Debezium! With Gunnar Morling | Current 2022 (20)

20 tips for website performance

Introduction to Modern DevOps Technologies

Building Web Mobile App that don’t suck - FITC Web Unleashed - 2014-09-18

Monolith vs Microservices with Golang at practice - Ivan Kutuzov

Improving Workflows With Grunt.js - Big D Design 2014 - Dallas Texas

Conquer Architectural Challenges with End-to-End JavaScript - enterJS 2014

CodiLime Tech Talk - Dawid Trzebiatowski i Wojciech Urbański: Opening the Flo...

Implementing data and databases on K8s within the Dutch government

Why run Postgres in Kubernetes?

Modern Webapps

JHipster presentation by Gaetan Bloch

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics

Scalable Clusters On Demand

Lunch & Learn BigQuery & Firebase from other Google Cloud customers

Cloud native - CI/CD

Multimedia support in WebKitGTK and WPE, current status and plans (GStreamer ...

jhipster-geekle-gbloch

Chrome Dev Summit Summary 2013 part 1 - what’s hot ?

More from HostedbyConfluent

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

HostedbyConfluent

Renaming a Kafka Topic | Kafka Summit London

HostedbyConfluent

Evolution of NRT Data Ingestion Pipeline at Trendyol

HostedbyConfluent

"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights. We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string. Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring. The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

HostedbyConfluent

"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service. While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications. The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection. Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."

Exactly-once Stream Processing with Arroyo and Kafka

HostedbyConfluent

"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output. Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing. In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."

Fish Plays Pokemon | Kafka Summit London

HostedbyConfluent

"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium. This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption. By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."

Tiered Storage 101 | Kafla Summit London

HostedbyConfluent

Building a Self-Service Stream Processing Portal: How And Why

HostedbyConfluent

"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work. However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business. You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want. Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware. What you’ll take away from our talk: - What were the pain points in the previous environment - How we transitioned to Confluent without service downtime - Creating a self-service stream processing portal built on top of Connect and ksqlDB - Use case of stream process portal"

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

HostedbyConfluent

"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics. Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

HostedbyConfluent

"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application. At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."

Navigating Private Network Connectivity Options for Kafka Clusters

HostedbyConfluent

"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option! In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

HostedbyConfluent

"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met. I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines. Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"

Explaining How Real-Time GenAI Works in a Noisy Pub

HostedbyConfluent

"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many. LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data. Large language models for your streaming application - explained with a little maths and a lot of pub stories"

TL;DR Kafka Metrics | Kafka Summit London

HostedbyConfluent

"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them. This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."

A Window Into Your Kafka Streams Tasks | KSL

HostedbyConfluent

Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

HostedbyConfluent

"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance. - Introduction to Kafka producer internal and workflow - Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance - Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient. - Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees. - Q&A session with attendees to address specific questions and concerns."

Data Contracts Management: Schema Registry and Beyond

HostedbyConfluent

"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element. In this talk, we’ll: 1. see why data contracts are so important but also difficult to implement; 2. identify the characteristics of a well-designed data contract: discuss the anatomy of a data contract, its main elements and, how to formally describe them; 3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."

Code-First Approach: Crafting Efficient Flink Apps

HostedbyConfluent

"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications. We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications. Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."

Debezium vs. the World: An Overview of the CDC Ecosystem

HostedbyConfluent

"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you! [1] https://debezium.io/"

Beyond Tiered Storage: Serverless Kafka with No Local Disks

HostedbyConfluent

"Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."

More from HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Renaming a Kafka Topic | Kafka Summit London

Evolution of NRT Data Ingestion Pipeline at Trendyol

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Exactly-once Stream Processing with Arroyo and Kafka

Fish Plays Pokemon | Kafka Summit London

Tiered Storage 101 | Kafla Summit London

Building a Self-Service Stream Processing Portal: How And Why

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Navigating Private Network Connectivity Options for Kafka Clusters

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Explaining How Real-Time GenAI Works in a Noisy Pub

TL;DR Kafka Metrics | Kafka Summit London

A Window Into Your Kafka Streams Tasks | KSL

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Data Contracts Management: Schema Registry and Beyond

Code-First Approach: Crafting Efficient Flink Apps

Debezium vs. the World: An Overview of the CDC Ecosystem

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Recently uploaded

WeTestAthens: Postman's AI & Automation Techniques

Postman

Letter and Document Automation for Bonterra Impact Management (fka Social Sol...

Jeffrey Haguewood

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on automated letter generation for Bonterra Impact Management using Google Workspace or Microsoft 365. Interested in deploying letter generation automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...

Jeffrey Haguewood

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on integration of Salesforce with Bonterra Impact Management. Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Programming Foundation Models with DSPy - Meetup Slides

Zilliz

GenAI Pilot Implementation in the organizations

kumardaparthi1024

Driving Business Innovation: Latest Generative AI Advancements & Success Story

Safe Software

Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency. During the hour, we’ll take you through: Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board. Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes. Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI. We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI. This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!

Generating privacy-protected synthetic data using Secludy and Milvus

Zilliz

During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.

Choosing The Best AWS Service For Your Website + API.pptx

Brandon Minnick, MBA

Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API? Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose? Which one is cheapest? Which one is fastest? Which one will scale to meet our needs? Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!

Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...

saastr

Operating System Used by Users in day-to-day life.pptx

Pravash Chandra Das

Dive into the realm of operating systems (OS) with Pravash Chandra Das, a seasoned Digital Forensic Analyst, as your guide. 🚀 This comprehensive presentation illuminates the core concepts, types, and evolution of OS, essential for understanding modern computing landscapes. Beginning with the foundational definition, Das clarifies the pivotal role of OS as system software orchestrating hardware resources, software applications, and user interactions. Through succinct descriptions, he delineates the diverse types of OS, from single-user, single-task environments like early MS-DOS iterations, to multi-user, multi-tasking systems exemplified by modern Linux distributions. Crucial components like the kernel and shell are dissected, highlighting their indispensable functions in resource management and user interface interaction. Das elucidates how the kernel acts as the central nervous system, orchestrating process scheduling, memory allocation, and device management. Meanwhile, the shell serves as the gateway for user commands, bridging the gap between human input and machine execution. 💻 The narrative then shifts to a captivating exploration of prominent desktop OSs, Windows, macOS, and Linux. Windows, with its globally ubiquitous presence and user-friendly interface, emerges as a cornerstone in personal computing history. macOS, lauded for its sleek design and seamless integration with Apple's ecosystem, stands as a beacon of stability and creativity. Linux, an open-source marvel, offers unparalleled flexibility and security, revolutionizing the computing landscape. 🖥️ Moving to the realm of mobile devices, Das unravels the dominance of Android and iOS. Android's open-source ethos fosters a vibrant ecosystem of customization and innovation, while iOS boasts a seamless user experience and robust security infrastructure. Meanwhile, discontinued platforms like Symbian and Palm OS evoke nostalgia for their pioneering roles in the smartphone revolution. The journey concludes with a reflection on the ever-evolving landscape of OS, underscored by the emergence of real-time operating systems (RTOS) and the persistent quest for innovation and efficiency. As technology continues to shape our world, understanding the foundations and evolution of operating systems remains paramount. Join Pravash Chandra Das on this illuminating journey through the heart of computing. 🌟

leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...

alexjohnson7307

June Patch Tuesday

Ivanti

Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.

Serial Arm Control in Real Time Presentation

tolgahangng

Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf

flufftailshop

When it comes to unit testing in the .NET ecosystem, developers have a wide range of options available. Among the most popular choices are NUnit, XUnit, and MSTest. These unit testing frameworks provide essential tools and features to help ensure the quality and reliability of code. However, understanding the differences between these frameworks is crucial for selecting the most suitable one for your projects.

Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr

saastr

TrustArc Webinar - 2024 Global Privacy Survey

TrustArc

How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024? In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores. See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe. This webinar will review: - The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey - The top challenges for privacy leaders, practitioners, and organizations in 2024 - Key themes to consider in developing and maintaining your privacy program

Introduction of Cybersecurity with OSS at Code Europe 2024

Hiroshi SHIBATA

I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems. The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS. Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application. I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.

Artificial Intelligence for XMLDevelopment

Octavian Nadolu

In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject. We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup. Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved. The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring. The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise. By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.

Nordic Marketo Engage User Group_June 13_ 2024.pptx

MichaelKnudsen27

Monitoring and Managing Anomaly Detection on OpenShift.pdf

Tosin Akinosho

Monitoring and Managing Anomaly Detection on OpenShift Overview Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices. Key Topics Covered 1. Introduction to Anomaly Detection - Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems. 2. Understanding Edge (IoT) - Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source. 3. What is ArgoCD? - Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices. 4. Deployment Using ArgoCD for Edge Devices - Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD. 5. Introduction to Apache Kafka and S3 - Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions. 6. Viewing Kafka Messages in the Data Lake - Learn how to view and analyze Kafka messages stored in a data lake for better insights. 7. What is Prometheus? - Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices. 8. Monitoring Application Metrics with Prometheus - Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system. 9. What is Camel K? - Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes. 10. Configuring Camel K Integrations for Data Pipelines - Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow. 11. What is a Jupyter Notebook? - Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text. 12. Jupyter Notebooks with Code Examples - Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.

Recently uploaded (20)

WeTestAthens: Postman's AI & Automation Techniques

Letter and Document Automation for Bonterra Impact Management (fka Social Sol...

Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...

Programming Foundation Models with DSPy - Meetup Slides

GenAI Pilot Implementation in the organizations

Driving Business Innovation: Latest Generative AI Advancements & Success Story

Generating privacy-protected synthetic data using Secludy and Milvus

Choosing The Best AWS Service For Your Website + API.pptx

Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...

Operating System Used by Users in day-to-day life.pptx

leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...

June Patch Tuesday

Serial Arm Control in Real Time Presentation

Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf

Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr

TrustArc Webinar - 2024 Global Privacy Survey

Introduction of Cybersecurity with OSS at Code Europe 2024

Artificial Intelligence for XMLDevelopment

Nordic Marketo Engage User Group_June 13_ 2024.pptx

Monitoring and Managing Anomaly Detection on OpenShift.pdf

Keep Your Cache Always Fresh With Debezium! With Gunnar Morling | Current 2022

1. Image © Nathalie https://flic.kr/p/21Ghf2g (CC BY 2.0) Keep Your Cache Always Fresh With Debezium! Gunnar Morling Software Engineer, Red Hat @gunnarmorling

2. #Debezium @gunnarmorling The Challenge Multi-site Application With Shared Database

3. #Debezium @gunnarmorling The Challenge Multi-site Application With Shared Database

4. #Debezium @gunnarmorling The Challenge Multi-site Application With Shared Database

5. #Debezium @gunnarmorling The Challenge ● Deployments in different geographies, single DB ● 90% read requests ● Complex queries Multi-site Application With Shared Database

6. #Debezium @gunnarmorling … Multi-site application with shared system-of-record database … With local, denormalized read views (CQRS) ... Automatically kept in sync after writes Today’s Mission 🤔 Explore How to Build a…

7. #Debezium @gunnarmorling ● Software engineer at Red Hat ○ Debezium ○ Quarkus ● kcctl 🧸, JfrUnit, ModiTect, MapStruct ● Spec Lead for Bean Validation 2.0 ● Java Champion Gunnar Morling

8. #Debezium @gunnarmorling The Idea Caching to the Rescue!

9. #Debezium @gunnarmorling The Idea Caching to the Rescue!

10. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot

11. #Debezium @gunnarmorling Inﬁnispan 100% Open-source In-Memory Distributed Data Store Interoperability Resilient Fault Tolerant Data Clustered Processing Query ACID Tx

12. #Debezium @gunnarmorling App 1 Data App 2 Data 1, Maria 2, Jenny 2, Jenny 3, Juan App 3 Data 1, Maria 3, Juan Inﬁnispan Deployment Distributed Cache

13. #Debezium @gunnarmorling App 1 Data App 2 Data 1, Maria 2, Jenny 2, Jenny 3, Juan App 3 Data 1, Maria 3, Juan Put 4 Will Put 4 Will Inﬁnispan Deployment Distributed Cache

14. #Debezium @gunnarmorling App 1 Data App 2 Data 1, Maria 2, Jenny 4, Will 2, Jenny 3, Juan 4, Will App 3 Data 1, Maria 3, Juan Inﬁnispan Deployment Distributed Cache

15. #Debezium @gunnarmorling

16. #Debezium @gunnarmorling Inﬁnispan Client/Server Service 1 Service 2 Binary (hot rod) REST ... Service 3 Data Infinispan Cluster Data Data

17. #Debezium @gunnarmorling Inﬁnispan Cross-Site Replication AWS (LON) GCP (NYC) Load Balancer APP APP Service APP APP Service Shared State Shared State Shared State Shared State Data Data Data NYC Data LON RELAY2

18. #Debezium @gunnarmorling The Question How To Keep The Cache In Sync?

19. #Debezium @gunnarmorling Dual Writes Easy to Get Wrong

20. https://flic.kr/p/PFDvkY Public Domain, Angelo Br

21. #Debezium @gunnarmorling Debezium in a Nutshell Open-Source Change Data Capture ● A CDC Platform ○ Based on transaction logs ○ Snapshotting, filtering, etc. ○ Outbox support ○ Web-based UI ● Fully open-source, very active community ● Large production deployments

22. #Debezium @gunnarmorling Debezium Becoming the De-Facto CDC Standard https://debezium.io/blog/2021/09/22/deep-dive-into-a-debezium-community-connector-scylla-cdc-source-connector/

23. #Debezium @gunnarmorling Debezium: Deployment Alternatives Embedded Engine and Debezium Server

24. #Debezium @gunnarmorling Solution Overview Capturing Changes From the Database

25. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot Demo

26.

27. #Debezium @gunnarmorling ● Fast start-up, low memory consumption ● Developer joy ● Imperative and Reactive ● Best-of-breed libraries ● Run via HotSpot and GraalVM native binaries Quarkus - Supersonic Subatomic Java A Stack for Building Cloud-native Apps

28. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot Demo

29. #Debezium @gunnarmorling Eventual Consistency Dealing With Stale Data ● Cached views updated asynchronously ● How to ensure read your own writes semantics? ● How to prevent taking action based on stale data? 🤔

30. #Debezium @gunnarmorling Eventual Consistency Read Your Own Writes

31. #Debezium @gunnarmorling Eventual Consistency Read Your Own Writes

32. #Debezium @gunnarmorling Eventual Consistency Read Your Own Writes

33. #Debezium @gunnarmorling Eventual Consistency Read Your Own Writes

34. #Debezium @gunnarmorling Eventual Consistency Detecting Writes Derived From Stale State

35. Wrap-Up

36. #Debezium @gunnarmorling … Multi-site application with shared system-of-record database ✅ … With local, denormalized read views (CQRS) ✅ ... Automatically kept in sync after writes ✅ Today’s Mission 🤩 Explore How to Build a…

37. #Debezium @gunnarmorling Discussion What Have We Gained? ● Pro: ○ Lower latencies ○ Reduced load on primary database ○ Increased availability ● Con: ○ Increased complexity

38. #Debezium @gunnarmorling ● Incremental snapshotting ● Postgres logical decoding messages ● Multi-DB support (SQL Server) ● Debezium Server sinks ● MongoDB change streams support ● Debezium UI ● Debezium 2.0 What’s New in Debezium?

39. #Debezium @gunnarmorling ● Infinispan: @infinispan | https://infinispan.org/ ● Debezium: @debezium | https://debezium.io/ ● Demo: https://github.com/debezium/debezium-examples/ → distributed-caching ● kcctl 🧸: https://github.com/kcctl/kcctl/ Learn More

40. #Debezium @gunnarmorling Q & A gunnar@hibernate.org @gunnarmorling 📧 Thank You!

Keep Your Cache Always Fresh With Debezium! With Gunnar Morling | Current 2022

Recommended

Recommended

More Related Content

Similar to Keep Your Cache Always Fresh With Debezium! With Gunnar Morling | Current 2022

Similar to Keep Your Cache Always Fresh With Debezium! With Gunnar Morling | Current 2022 (20)

More from HostedbyConfluent

More from HostedbyConfluent (20)

Recently uploaded

Recently uploaded (20)

Keep Your Cache Always Fresh With Debezium! With Gunnar Morling | Current 2022