Kafka: Legacy microservices
•Download as PPTX, PDF•
1 like•416 views
Kafka can be used to smoothly migrate a legacy monolithic application to a new microservices-oriented architecture. The migration occurs gradually with no big bang or production stop. Features are gradually swapped from the legacy application to new microservices. The legacy application and microservices run side by side and can concurrently modify the same data conflictlessly. The migration involves two steps - first pushing changes from the legacy application to the microservices using Kafka streams, and then pushing changes from the microservices back to the legacy application. This allows for an ability to rollback and derisks the data migration process.
Report
Share
Report
Share
Recommended
Clustered Architecture Patterns Delivering Scalability And Availability
The document discusses different architecture patterns for delivering scalability and availability in clustered systems. It covers load-balanced and partitioned scale-out patterns, and how to balance simplicity, scalability, and availability. JVM-level clustering is presented as an approach that can address these patterns by sharing memory across JVMs in a transparent way.
Two way data sync between legacy and your brand new micro-service architecture
This document discusses strategies for synchronizing data between a legacy system and a new microservices architecture. It proposes using change data capture (CDC) to log all database changes in the legacy system and streaming them to the microservices using Apache Kafka. This allows the two systems to run concurrently while keeping the data eventually consistent as the services are migrated gradually using a strangler pattern.
How to build an event driven architecture with kafka and kafka connect
How to build an event driven architecture with Kafka & Kafka Connect?
Bài talk chia sẻ về quá trình 2 năm ứng dụng Kafka, Kafka Connect để chuyển đổi mô hình hệ thống của Vexere từ Monolithic thành Microservice, event driven
- Event driven architecture là gì?
- Làm thế nào để xây dựng 1 hệ thống event driven architecture một cách hiệu qủa bằng Kafka và Kafka Connect
onnect
- Các usercase hữu ích với Kafka & Kafka Connect
- Kinh nghiệm thực tế và các bài học rút ra
Use ksqlDB to migrate core-banking processing from batch to streaming | Mark ...
Core banking systems are batch oriented: typically with heavy overnight batch cycles before business opens each morning. In this talk I will explain some of the common interface points between core-banking infrastructure and event streaming systems. Then I will focus on how to do stream processing using ksqlDB for core-banking shaped data: showing how to do common operation using various ksqlDB functions. The key features are avro-record keys and multi-key joins (ksqlDB 0.15), schema management and state store planning.
Flink Forward Berlin 2018: Ravi Suhag & Sumanth Nakshatrithaya - "Managing Fl...
Managing Flink operations at GO-JEK involves resource provisioning, isolation, data quality, monitoring, and failovers across multiple clusters and data centers. GO-JEK uses Flink for use cases like surge pricing, API health monitoring, and fraud detection. They developed tools to automate cluster provisioning reducing time by 90%, isolate resources for security and performance, manage data quality with schema changes, monitor multiple clusters, and enable kafka input stream and cluster failovers for resiliency. Chaos engineering experiments also help test systems through disaster simulations and load testing.
O'Reilly Media Webcast: Building Real-Time Data Pipelines
As our customers tap into new sources of data or modify to existing data pipelines, we are often asked questions like: What technologies should we consider? Where can we reduce data latency? How can we simplify our data architecture?
To eliminate the guesswork, we teamed up with Ben Lorica, Chief Data Scientist at O’Reilly Media to host a webcast centered around building real-time data pipelines.
INTRODUCING: CREATE PIPELINE
This document introduces MemSQL Pipelines, which allow for exactly-once data ingestion semantics when streaming data from Kafka into MemSQL. MemSQL Pipelines provide a native way to extract, transform, and load external data into MemSQL tables. They offer a scalable and highly performant ETL process across a distributed cluster. The document explains streaming semantics like at least once and exactly once delivery, and how MemSQL Pipelines coordinate with Kafka to enable exactly-once ingestion through offset tracking. It presents the architecture of MemSQL Pipelines and demonstrates their use through a live demo.
How to use Standard SQL over Kafka: From the basics to advanced use cases | F...
Several different frameworks have been developed to draw data from Kafka and maintain standard SQL over continually changing data. This provides an easy way to query and transform data - now accessible by orders of magnitude more users.
At the same time, using Standard SQL against changing data is a new pattern for many engineers and analysts. While the language hasn’t changed, we’re still in the early stages of understanding the power of SQL over Kafka - and in some interesting ways, this new pattern introduces some exciting new idioms.
In this session, we’ll start with some basic use cases of how Standard SQL can be effectively used over events in Kafka- including how these SQL engines can help teams that are brand new to streaming data get started. From there, we’ll cover a series of more advanced functions and their implications, including:
- WHERE clauses that contain time change the validity intervals of your data; you can programmatically introduce and retract records based on their payloads!
- LATERAL joins turn streams of query arguments into query results; they will automatically share their query plans and resources!
- GROUP BY aggregations can be applied to ever-growing data collections; reduce data that wouldn't even fit in a database in the first place.
We'll review in-production examples where each of these cases make unmodified Standard SQL, run and maintain over data streams in Kafka, and provide the functionality of bespoke stream processors.
Recommended
Clustered Architecture Patterns Delivering Scalability And Availability
The document discusses different architecture patterns for delivering scalability and availability in clustered systems. It covers load-balanced and partitioned scale-out patterns, and how to balance simplicity, scalability, and availability. JVM-level clustering is presented as an approach that can address these patterns by sharing memory across JVMs in a transparent way.
Two way data sync between legacy and your brand new micro-service architecture
This document discusses strategies for synchronizing data between a legacy system and a new microservices architecture. It proposes using change data capture (CDC) to log all database changes in the legacy system and streaming them to the microservices using Apache Kafka. This allows the two systems to run concurrently while keeping the data eventually consistent as the services are migrated gradually using a strangler pattern.
How to build an event driven architecture with kafka and kafka connect
How to build an event driven architecture with Kafka & Kafka Connect?
Bài talk chia sẻ về quá trình 2 năm ứng dụng Kafka, Kafka Connect để chuyển đổi mô hình hệ thống của Vexere từ Monolithic thành Microservice, event driven
- Event driven architecture là gì?
- Làm thế nào để xây dựng 1 hệ thống event driven architecture một cách hiệu qủa bằng Kafka và Kafka Connect
onnect
- Các usercase hữu ích với Kafka & Kafka Connect
- Kinh nghiệm thực tế và các bài học rút ra
Use ksqlDB to migrate core-banking processing from batch to streaming | Mark ...
Core banking systems are batch oriented: typically with heavy overnight batch cycles before business opens each morning. In this talk I will explain some of the common interface points between core-banking infrastructure and event streaming systems. Then I will focus on how to do stream processing using ksqlDB for core-banking shaped data: showing how to do common operation using various ksqlDB functions. The key features are avro-record keys and multi-key joins (ksqlDB 0.15), schema management and state store planning.
Flink Forward Berlin 2018: Ravi Suhag & Sumanth Nakshatrithaya - "Managing Fl...
Managing Flink operations at GO-JEK involves resource provisioning, isolation, data quality, monitoring, and failovers across multiple clusters and data centers. GO-JEK uses Flink for use cases like surge pricing, API health monitoring, and fraud detection. They developed tools to automate cluster provisioning reducing time by 90%, isolate resources for security and performance, manage data quality with schema changes, monitor multiple clusters, and enable kafka input stream and cluster failovers for resiliency. Chaos engineering experiments also help test systems through disaster simulations and load testing.
O'Reilly Media Webcast: Building Real-Time Data Pipelines
As our customers tap into new sources of data or modify to existing data pipelines, we are often asked questions like: What technologies should we consider? Where can we reduce data latency? How can we simplify our data architecture?
To eliminate the guesswork, we teamed up with Ben Lorica, Chief Data Scientist at O’Reilly Media to host a webcast centered around building real-time data pipelines.
INTRODUCING: CREATE PIPELINE
This document introduces MemSQL Pipelines, which allow for exactly-once data ingestion semantics when streaming data from Kafka into MemSQL. MemSQL Pipelines provide a native way to extract, transform, and load external data into MemSQL tables. They offer a scalable and highly performant ETL process across a distributed cluster. The document explains streaming semantics like at least once and exactly once delivery, and how MemSQL Pipelines coordinate with Kafka to enable exactly-once ingestion through offset tracking. It presents the architecture of MemSQL Pipelines and demonstrates their use through a live demo.
How to use Standard SQL over Kafka: From the basics to advanced use cases | F...
Several different frameworks have been developed to draw data from Kafka and maintain standard SQL over continually changing data. This provides an easy way to query and transform data - now accessible by orders of magnitude more users.
At the same time, using Standard SQL against changing data is a new pattern for many engineers and analysts. While the language hasn’t changed, we’re still in the early stages of understanding the power of SQL over Kafka - and in some interesting ways, this new pattern introduces some exciting new idioms.
In this session, we’ll start with some basic use cases of how Standard SQL can be effectively used over events in Kafka- including how these SQL engines can help teams that are brand new to streaming data get started. From there, we’ll cover a series of more advanced functions and their implications, including:
- WHERE clauses that contain time change the validity intervals of your data; you can programmatically introduce and retract records based on their payloads!
- LATERAL joins turn streams of query arguments into query results; they will automatically share their query plans and resources!
- GROUP BY aggregations can be applied to ever-growing data collections; reduce data that wouldn't even fit in a database in the first place.
We'll review in-production examples where each of these cases make unmodified Standard SQL, run and maintain over data streams in Kafka, and provide the functionality of bespoke stream processors.
Change Data Capture - Scale by the Bay 2019
The document discusses change data capture (CDC) in distributed systems. CDC allows capturing changes to data and notifying actors so they can react accordingly. Some considerations for CDC include reliability, scalability, performance, consistency, and fault tolerance. Common strategies involve using timestamps, triggers, diffs, or log scraping. CDC has various use cases such as caching, search indexing, offline processing, and simplifying monolithic applications. Popular frameworks for implementing CDC include pg2k4j, Maxwell, AWS Database Migration Service, SpinalTap, DataBus, and Debezium.
Storing State Forever: Why It Can Be Good For Your Analytics
State is an essential part of the modern streaming pipelines: it enables a variety of foundational capabilities like windowing, aggregation, enrichment, etc. But usually, the state is either transient, so we only keep it until the window is closed, or it's fairly small and doesn't grow much. But what if we treat the state differently? The keyed state in Flink can be scaled vertically and horizontally, it's reliable and fault-tolerant... so is scaling a stateful Flink application that different from scaling any data store like Kafka or MySQL?
At Shopify, we've worked on a massive analytical data pipeline that's needed to support complex streaming joins and correctly handle arbitrarily late-arriving data. We came up with an idea to never clear state and support joins this way. We've made a successful proof of concept, ingested all historical transactional Shopify data and ended up storing more than 10 TB of Flink state. In the end, it allowed us to achieve 100% data correctness.
All Streams Ahead! ksqlDB Workshop ANZ
Watch the available webcast: https://videos.confluent.io/watch/Jbb3PUbgDFck6jLcCkVvur?
Speaker: David Peterson
Kafka Summit SF 2017 - Fast Data in Supply Chain Planning
This document discusses using fast data and stream processing with Kafka to improve supply chain planning. It describes problems with traditional sequential and batch-oriented systems and proposes using Kafka streams to process continuous data in real-time. Examples are given of using Kafka streams for message translation, splitting messages, aggregation, and integrating data from multiple topics to generate reports. Challenges with testing integration points and data quality are also mentioned.
From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous M...
In this talk we’ll describe the evolution of stream processing at Tesla and the challenges that are specific to our needs, such as large skews in message-processing latencies. We’ll describe how we built a reliable and performant ingestion platform that allows us to take an idea from a whiteboard to production in just a matter of hours. We’ll also discuss the design principles, tools, and incident response processes that have enabled a small team to support Kafka and downstream services in highly-available and multi-tenant environments at scale.
How to mutate your immutable log | Andrey Falko, Stripe
Have you ever had your upstream producers write poisoned data that breaks your downstream consumers? Did Personal Identifiable Information (PII) land in a Kafka topic that wasn’t supposed to have it? Is your data pipeline under development and you simply want to iterate quickly? Immutability is one of the key and desirable features of Kafka. However, when mistakes happen and you are paged at night you sometimes wish there was an “easy button” to change the log.
This session first dives into some of the errors we have seen that caused outages for considerable durations of time. Recovery from the errors required late night code changes on consumers or simply waiting things out.
The next part of the session proposes a topic versioning scheme that allows us to recover from the examples that we mention. It segues into what it would take to build a control plane to manage and lifecycle these versioned topics. We’ll cover the benefits and pitfalls of our proposed solution.
Scaling a Core Banking Engine Using Apache Kafka | Peter Dudbridge, Thought M...
Core banking is one of the last bastions for the mainframe. As many other industries have moved to the cloud, why are most of the world’s banks yet to follow?
The answer lies in a bank's conflicting needs: correctness and scale - historically achievable using a monolithic application running on a large mainframe. The clock is ticking for the banks as we approach an inflection point where the mainframes become too expensive, and aren’t flexible enough to meet the modern banking consumers needs
A simple lift and shift onto the cloud does not work. As we distribute our core processing we spend an increasing amount of time on the network, and race conditions lurk that threaten ‘correctness’
This session explores how Thought Machine’s core banking system ‘Vault’ was built in a cloud first manner, leveraging Kafka to enable asynchronous and parallel processing at scale, specifically focusing on the architectural patterns we have used to ensure ‘correctness’ in such an environment
CDC patterns in Apache Kafka®
This document discusses using Apache Kafka as a data hub to capture changes from various data sources using change data capture (CDC). It outlines several common CDC patterns like using modification dates, database triggers, or log files to identify changes. It then discusses using Kafka Connect to integrate various data sources like MongoDB, PostgreSQL and replicate changes. The document provides examples of open source CDC connectors and concludes with suggestions for getting involved in the Apache Kafka community.
Lessons Learned from Building and Operating Scuba
This document provides an overview of Scuba, Facebook's real-time analytics database. It summarizes Scuba's key features including real-time data ingestion and querying capabilities with simple rollup queries and flexible schemas. It also describes Scuba's architecture with distributed data storage and demand control. Finally, it discusses lessons learned from building and operating Scuba, including common issues and reasons for its success filling a specific niche for analytics.
Flink Forward Berlin 2017: Gyula Fora - Building and operating large-scale st...
This document discusses King's streaming platform for processing 40 billion events per day from its mobile games. It describes how King uses Apache Flink to power a streaming application called RBea that allows live scripting on event streams using windows, aggregates and stateful computations. RBea has been in stable production since 2016, processing data for over 20 games. The document also covers challenges of building and operating the streaming platform at scale.
Creating an Elastic Platform Using Kafka and Microservices in OpenShift
(Pradeep Chintam, American Express Global Business Travel) Kafka Summit SF 2018
When a new project, Global Trip Record was launched at American Express GBT and we were looking for a robust, scalable and fault-tolerant middleware to handle all the orchestration and connectivity needs of the project.
The existing solution was monolithic, and we wanted to convert that to a microservices framework, but the biggest challenge was managing the increasing number of external applications that are connected to the platform. Any slow external application or partner system connected to the platform was slowing down the entire platform. There is always a need for partner systems to go offline or a need to resend the entire day’s data, especially with a system like our data lake where the data volumes are huge.
After evaluating multiple solutions, we settled on Apache Kafka, and started with a simple implementation of around 100,000 messages to just decouple one partner system and the core platform.
Today, we are running our microservices (Docker) running in OpenShift (Kubernetes) processing Kafka Streams, running real-time anomaly detection using Kafka Streams, powering our data lake through Kafka, feeding our distributed caching layer (Apache Ignite) and connecting all internal and external systems using Kafka. With a total of more than 10 million messages per day, i.e., 1.5TB of data with just a small three-node cluster, we are one happy platform for over a year now. With the kind of stability, flexibility and success in our project, a lot of other teams started and will soon be in production with Kafka Steams. The powerful combination of Kafka and OpenShift has proven to be an easily scalable model with great elasticity to the entire platform.
Via Varejo taking data from legacy to a new world at Brazil Black Friday (Mar...
"Use of techniques to services decomposition into a set of stages allowing code modularity and reuse. Good practices for dealing with DeadLetter, Monitoring, CorrelationID, Log, Base classes to control all software development best practices, Buffer Control in Apache Kafka and aspects related to Apache Kafka scalability and fault tolerance. Processing and management of high messages streaming on Black Friday (~ 25.4 million / day)
After a retrospective of how our structure behaved during the last Black Friday, we learned a few lessons and decided to adopt a new approach to address some specific scenarios which have millions of messages, ensuring resilience, uptime of at least 99.9%, monitoring and alerts for each module. We decided to adopt the SEDA architecture standard to traffic these millions of messages as closely as possible and deliver the desired quality to the target systems with scalability and reliability. By separating the pipeline processing modules, we were able to scale each of these modules horizontally, increasing the number of PODs (Openshift) and partitions of Kafka topics in order to process a given pipeline step faster. In addition, we also need to apply tunnings to Apache Kafka, one of which concerns the guarantee of delivery of the message. The focus of this presentation is to show the solution designed and how we use Apache Kafka and the SEDA architecture standard to orchestrate this massive stream of data we face."
ksqlDB Workshop
Watch the webcast here: https://videos.confluent.io/watch/woobbdupdrwpJCHZKEwF5E?
Speaker: Patrick Druley
Flink Forward Berlin 2018: Timo Walther - "Flink SQL in Action"
This document discusses Flink SQL and its capabilities for stream and batch processing. Flink SQL provides a unified SQL interface that allows the same queries to run over both bounded batch data and unbounded streaming data. It achieves this through dynamic tables that are continuously updated as streaming data arrives. The document outlines Flink SQL's features, how queries are translated and executed, and examples of how SQL can be used to build data pipelines, real-time analytics, and live dashboards. It also introduces the new SQL client for interacting with Flink without writing code.
Abstractions for managed stream processing platform (Arya Ketan - Flipkart)
Speaker: Arya Ketan (https://www.linkedin.com/in/aryaketan/)
Talk presented during Bangalore Kafka group's stream processing meetup at Walmart
https://www.meetup.com/Bangalore-Apache-Kafka-Group/events/266777028/
Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020
In this talk, Confluent co-founder and CEO, Jay Kreps will cover the rise of two trends:
1. The rise of Apache Kafka and event streams
2. The rise of the public cloud and cloud-native data systems
... and the problems we need to solve as these two trends come together.
Modeling the Smart and Connected City of the Future with Kafka and Spark
- Modeling the Smart and Connected City of the Future with Kafka and Spark discusses using Kafka, Spark, and MemSQL to build a real-time data pipeline for a hypothetical "MemCity" that captures data from 1.4 million households.
- The document outlines the components of the "Real-Time Trinity" - Kafka for a high-throughput message queue, Spark for data transformation, and MemSQL for real-time data serving and analytics.
- It also introduces MemSQL Streamliner, which is designed to simplify the creation of real-time data pipelines through a graphical interface and one-click deployment of integrated Apache Spark clusters.
Using Kafka to scale database replication
LinkedIn used Kafka to unify and scale their database infrastructure. They replaced their MySQL replication with a Kafka-based approach to allow for more flexible shard placement, easier cluster expansion, and higher availability. Using Kafka eliminated the need for a separate data replication system and provided significant cost savings compared to the previous architecture.
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
Serverless architectures allow automatic scaling of compute resources and pay-per-use pricing models. While scalability is often a key benefit, serverless functions have limitations regarding state management. Event-driven serverless platforms can be viewed as a form of stream processing. To handle stateful operations, the data model should be designed based on how data will be queried. An "inside out database" approach treats each event as data and action, and uses the database primarily for data storage rather than querying. This mirrors an CQRS pattern and allows scaling to infinite event streams.
Apache Kafka Streams Use Case
#ApacheKafkaTLV: Building distributed, fault-tolerant processing apps with Kafka Streams - use case
The second part of the #2 Meetup, delivered by Anatoly Tichonov - Mentory. Hosted by WeWork Sarona TLV,
HOP! Airlines Jets to Real Time
(Joseph deBuzna + Zulfikar Quereshi, HVR) Kafka Summit SF 2018
This presentation is a customer story about France-based regional airline HOP! and their need to make better use of data that was contained in various applications. They also needed this information to be available in real time. As one can imagine, airlines manage a wide variety of information such as weather, customer information, flight plans, sensor data from planes and much more.
In this presentation, Joe will discuss how HOP! was delivering their data before and the limitations associated with delivering this data. Joe will then talk about HOP!’s selection of Kafka and HVR as a solution to enabling data availability and real-time information for analysis and action.
In this session, attendees will learn:
-How Kafka was selected and chosen as a solution for HOP!’s complex challenges
-Architecture and capabilities implemented that enabled data feeding from multiple sources to Kafka
-Considerations and challenges with this approach
-Business results and future plans
Building Event-Driven Services with Apache Kafka
Should you use REST to sew services together? Is it better to use a richer, brokered protocol? This practical talk will dig into how we piece services together in event driven systems, how we we use a distributed log to create a central, persistent narrative and what benefits we reap from doing so.
More Related Content
What's hot
Change Data Capture - Scale by the Bay 2019
The document discusses change data capture (CDC) in distributed systems. CDC allows capturing changes to data and notifying actors so they can react accordingly. Some considerations for CDC include reliability, scalability, performance, consistency, and fault tolerance. Common strategies involve using timestamps, triggers, diffs, or log scraping. CDC has various use cases such as caching, search indexing, offline processing, and simplifying monolithic applications. Popular frameworks for implementing CDC include pg2k4j, Maxwell, AWS Database Migration Service, SpinalTap, DataBus, and Debezium.
Storing State Forever: Why It Can Be Good For Your Analytics
State is an essential part of the modern streaming pipelines: it enables a variety of foundational capabilities like windowing, aggregation, enrichment, etc. But usually, the state is either transient, so we only keep it until the window is closed, or it's fairly small and doesn't grow much. But what if we treat the state differently? The keyed state in Flink can be scaled vertically and horizontally, it's reliable and fault-tolerant... so is scaling a stateful Flink application that different from scaling any data store like Kafka or MySQL?
At Shopify, we've worked on a massive analytical data pipeline that's needed to support complex streaming joins and correctly handle arbitrarily late-arriving data. We came up with an idea to never clear state and support joins this way. We've made a successful proof of concept, ingested all historical transactional Shopify data and ended up storing more than 10 TB of Flink state. In the end, it allowed us to achieve 100% data correctness.
All Streams Ahead! ksqlDB Workshop ANZ
Watch the available webcast: https://videos.confluent.io/watch/Jbb3PUbgDFck6jLcCkVvur?
Speaker: David Peterson
Kafka Summit SF 2017 - Fast Data in Supply Chain Planning
This document discusses using fast data and stream processing with Kafka to improve supply chain planning. It describes problems with traditional sequential and batch-oriented systems and proposes using Kafka streams to process continuous data in real-time. Examples are given of using Kafka streams for message translation, splitting messages, aggregation, and integrating data from multiple topics to generate reports. Challenges with testing integration points and data quality are also mentioned.
From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous M...
In this talk we’ll describe the evolution of stream processing at Tesla and the challenges that are specific to our needs, such as large skews in message-processing latencies. We’ll describe how we built a reliable and performant ingestion platform that allows us to take an idea from a whiteboard to production in just a matter of hours. We’ll also discuss the design principles, tools, and incident response processes that have enabled a small team to support Kafka and downstream services in highly-available and multi-tenant environments at scale.
How to mutate your immutable log | Andrey Falko, Stripe
Have you ever had your upstream producers write poisoned data that breaks your downstream consumers? Did Personal Identifiable Information (PII) land in a Kafka topic that wasn’t supposed to have it? Is your data pipeline under development and you simply want to iterate quickly? Immutability is one of the key and desirable features of Kafka. However, when mistakes happen and you are paged at night you sometimes wish there was an “easy button” to change the log.
This session first dives into some of the errors we have seen that caused outages for considerable durations of time. Recovery from the errors required late night code changes on consumers or simply waiting things out.
The next part of the session proposes a topic versioning scheme that allows us to recover from the examples that we mention. It segues into what it would take to build a control plane to manage and lifecycle these versioned topics. We’ll cover the benefits and pitfalls of our proposed solution.
Scaling a Core Banking Engine Using Apache Kafka | Peter Dudbridge, Thought M...
Core banking is one of the last bastions for the mainframe. As many other industries have moved to the cloud, why are most of the world’s banks yet to follow?
The answer lies in a bank's conflicting needs: correctness and scale - historically achievable using a monolithic application running on a large mainframe. The clock is ticking for the banks as we approach an inflection point where the mainframes become too expensive, and aren’t flexible enough to meet the modern banking consumers needs
A simple lift and shift onto the cloud does not work. As we distribute our core processing we spend an increasing amount of time on the network, and race conditions lurk that threaten ‘correctness’
This session explores how Thought Machine’s core banking system ‘Vault’ was built in a cloud first manner, leveraging Kafka to enable asynchronous and parallel processing at scale, specifically focusing on the architectural patterns we have used to ensure ‘correctness’ in such an environment
CDC patterns in Apache Kafka®
This document discusses using Apache Kafka as a data hub to capture changes from various data sources using change data capture (CDC). It outlines several common CDC patterns like using modification dates, database triggers, or log files to identify changes. It then discusses using Kafka Connect to integrate various data sources like MongoDB, PostgreSQL and replicate changes. The document provides examples of open source CDC connectors and concludes with suggestions for getting involved in the Apache Kafka community.
Lessons Learned from Building and Operating Scuba
This document provides an overview of Scuba, Facebook's real-time analytics database. It summarizes Scuba's key features including real-time data ingestion and querying capabilities with simple rollup queries and flexible schemas. It also describes Scuba's architecture with distributed data storage and demand control. Finally, it discusses lessons learned from building and operating Scuba, including common issues and reasons for its success filling a specific niche for analytics.
Flink Forward Berlin 2017: Gyula Fora - Building and operating large-scale st...
This document discusses King's streaming platform for processing 40 billion events per day from its mobile games. It describes how King uses Apache Flink to power a streaming application called RBea that allows live scripting on event streams using windows, aggregates and stateful computations. RBea has been in stable production since 2016, processing data for over 20 games. The document also covers challenges of building and operating the streaming platform at scale.
Creating an Elastic Platform Using Kafka and Microservices in OpenShift
(Pradeep Chintam, American Express Global Business Travel) Kafka Summit SF 2018
When a new project, Global Trip Record was launched at American Express GBT and we were looking for a robust, scalable and fault-tolerant middleware to handle all the orchestration and connectivity needs of the project.
The existing solution was monolithic, and we wanted to convert that to a microservices framework, but the biggest challenge was managing the increasing number of external applications that are connected to the platform. Any slow external application or partner system connected to the platform was slowing down the entire platform. There is always a need for partner systems to go offline or a need to resend the entire day’s data, especially with a system like our data lake where the data volumes are huge.
After evaluating multiple solutions, we settled on Apache Kafka, and started with a simple implementation of around 100,000 messages to just decouple one partner system and the core platform.
Today, we are running our microservices (Docker) running in OpenShift (Kubernetes) processing Kafka Streams, running real-time anomaly detection using Kafka Streams, powering our data lake through Kafka, feeding our distributed caching layer (Apache Ignite) and connecting all internal and external systems using Kafka. With a total of more than 10 million messages per day, i.e., 1.5TB of data with just a small three-node cluster, we are one happy platform for over a year now. With the kind of stability, flexibility and success in our project, a lot of other teams started and will soon be in production with Kafka Steams. The powerful combination of Kafka and OpenShift has proven to be an easily scalable model with great elasticity to the entire platform.
Via Varejo taking data from legacy to a new world at Brazil Black Friday (Mar...
"Use of techniques to services decomposition into a set of stages allowing code modularity and reuse. Good practices for dealing with DeadLetter, Monitoring, CorrelationID, Log, Base classes to control all software development best practices, Buffer Control in Apache Kafka and aspects related to Apache Kafka scalability and fault tolerance. Processing and management of high messages streaming on Black Friday (~ 25.4 million / day)
After a retrospective of how our structure behaved during the last Black Friday, we learned a few lessons and decided to adopt a new approach to address some specific scenarios which have millions of messages, ensuring resilience, uptime of at least 99.9%, monitoring and alerts for each module. We decided to adopt the SEDA architecture standard to traffic these millions of messages as closely as possible and deliver the desired quality to the target systems with scalability and reliability. By separating the pipeline processing modules, we were able to scale each of these modules horizontally, increasing the number of PODs (Openshift) and partitions of Kafka topics in order to process a given pipeline step faster. In addition, we also need to apply tunnings to Apache Kafka, one of which concerns the guarantee of delivery of the message. The focus of this presentation is to show the solution designed and how we use Apache Kafka and the SEDA architecture standard to orchestrate this massive stream of data we face."
ksqlDB Workshop
Watch the webcast here: https://videos.confluent.io/watch/woobbdupdrwpJCHZKEwF5E?
Speaker: Patrick Druley
Flink Forward Berlin 2018: Timo Walther - "Flink SQL in Action"
This document discusses Flink SQL and its capabilities for stream and batch processing. Flink SQL provides a unified SQL interface that allows the same queries to run over both bounded batch data and unbounded streaming data. It achieves this through dynamic tables that are continuously updated as streaming data arrives. The document outlines Flink SQL's features, how queries are translated and executed, and examples of how SQL can be used to build data pipelines, real-time analytics, and live dashboards. It also introduces the new SQL client for interacting with Flink without writing code.
Abstractions for managed stream processing platform (Arya Ketan - Flipkart)
Speaker: Arya Ketan (https://www.linkedin.com/in/aryaketan/)
Talk presented during Bangalore Kafka group's stream processing meetup at Walmart
https://www.meetup.com/Bangalore-Apache-Kafka-Group/events/266777028/
Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020
In this talk, Confluent co-founder and CEO, Jay Kreps will cover the rise of two trends:
1. The rise of Apache Kafka and event streams
2. The rise of the public cloud and cloud-native data systems
... and the problems we need to solve as these two trends come together.
Modeling the Smart and Connected City of the Future with Kafka and Spark
- Modeling the Smart and Connected City of the Future with Kafka and Spark discusses using Kafka, Spark, and MemSQL to build a real-time data pipeline for a hypothetical "MemCity" that captures data from 1.4 million households.
- The document outlines the components of the "Real-Time Trinity" - Kafka for a high-throughput message queue, Spark for data transformation, and MemSQL for real-time data serving and analytics.
- It also introduces MemSQL Streamliner, which is designed to simplify the creation of real-time data pipelines through a graphical interface and one-click deployment of integrated Apache Spark clusters.
Using Kafka to scale database replication
LinkedIn used Kafka to unify and scale their database infrastructure. They replaced their MySQL replication with a Kafka-based approach to allow for more flexible shard placement, easier cluster expansion, and higher availability. Using Kafka eliminated the need for a separate data replication system and provided significant cost savings compared to the previous architecture.
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
Serverless architectures allow automatic scaling of compute resources and pay-per-use pricing models. While scalability is often a key benefit, serverless functions have limitations regarding state management. Event-driven serverless platforms can be viewed as a form of stream processing. To handle stateful operations, the data model should be designed based on how data will be queried. An "inside out database" approach treats each event as data and action, and uses the database primarily for data storage rather than querying. This mirrors an CQRS pattern and allows scaling to infinite event streams.
Apache Kafka Streams Use Case
#ApacheKafkaTLV: Building distributed, fault-tolerant processing apps with Kafka Streams - use case
The second part of the #2 Meetup, delivered by Anatoly Tichonov - Mentory. Hosted by WeWork Sarona TLV,
What's hot (20)
Storing State Forever: Why It Can Be Good For Your Analytics
Storing State Forever: Why It Can Be Good For Your Analytics
Kafka Summit SF 2017 - Fast Data in Supply Chain Planning
Kafka Summit SF 2017 - Fast Data in Supply Chain Planning
From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous M...
From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous M...
How to mutate your immutable log | Andrey Falko, Stripe
How to mutate your immutable log | Andrey Falko, Stripe
Scaling a Core Banking Engine Using Apache Kafka | Peter Dudbridge, Thought M...
Scaling a Core Banking Engine Using Apache Kafka | Peter Dudbridge, Thought M...
Flink Forward Berlin 2017: Gyula Fora - Building and operating large-scale st...
Flink Forward Berlin 2017: Gyula Fora - Building and operating large-scale st...
Creating an Elastic Platform Using Kafka and Microservices in OpenShift
Creating an Elastic Platform Using Kafka and Microservices in OpenShift
Via Varejo taking data from legacy to a new world at Brazil Black Friday (Mar...
Via Varejo taking data from legacy to a new world at Brazil Black Friday (Mar...
Flink Forward Berlin 2018: Timo Walther - "Flink SQL in Action"
Flink Forward Berlin 2018: Timo Walther - "Flink SQL in Action"
Abstractions for managed stream processing platform (Arya Ketan - Flipkart)
Abstractions for managed stream processing platform (Arya Ketan - Flipkart)
Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020
Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020
Modeling the Smart and Connected City of the Future with Kafka and Spark
Modeling the Smart and Connected City of the Future with Kafka and Spark
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
Similar to Kafka: Legacy microservices
HOP! Airlines Jets to Real Time
(Joseph deBuzna + Zulfikar Quereshi, HVR) Kafka Summit SF 2018
This presentation is a customer story about France-based regional airline HOP! and their need to make better use of data that was contained in various applications. They also needed this information to be available in real time. As one can imagine, airlines manage a wide variety of information such as weather, customer information, flight plans, sensor data from planes and much more.
In this presentation, Joe will discuss how HOP! was delivering their data before and the limitations associated with delivering this data. Joe will then talk about HOP!’s selection of Kafka and HVR as a solution to enabling data availability and real-time information for analysis and action.
In this session, attendees will learn:
-How Kafka was selected and chosen as a solution for HOP!’s complex challenges
-Architecture and capabilities implemented that enabled data feeding from multiple sources to Kafka
-Considerations and challenges with this approach
-Business results and future plans
Building Event-Driven Services with Apache Kafka
Should you use REST to sew services together? Is it better to use a richer, brokered protocol? This practical talk will dig into how we piece services together in event driven systems, how we we use a distributed log to create a central, persistent narrative and what benefits we reap from doing so.
Event Driven Services Part 2: Building Event-Driven Services with Apache Kafka
The second online talk in the Confluent Event Driven Services series covers how we build a shared narrative, using an event driven architecture, to conflate communication protocol and state transfer. We investigate how this leads to CQRS and discuss patterns for attaching Read-centric views to our canonical set of streams.
How did we move the mountain? - Migrating 1 trillion+ messages per day across...
Have you ever migrated Kafka clusters from one data center to another being completely transparent to client applications?
At PayPal, as part of a massive datacenter migration initiative, Kafka team successfully moved all PayPal Kafka traffic across data centers. This initiative involved migrating 20+ Kafka clusters (1000+ broker and zookeeper nodes), as well as 60+ mirrormaker groups which seamlessly handle Kafka traffic volumes as high as 1 trillion messages per day. Throughout the course of this migration, applications required no modification, encountered 0% service outage, 0% message loss and duplicated messages. The whole migration process was fully transparent to Kafka applications.
In this session, you will learn the strategies, techniques and tools the PayPal Kafka team has utilized for managing the migration process. You will also learn the lessons and pitfalls they experienced during this exercise, as well as the secret sauce of making the migration successful.
Unleashing Apache Kafka and TensorFlow in the Cloud
How can you leverage the flexibility and extreme scale in the public cloud combined with your Apache Kafka ecosystem to build scalable, mission-critical machine learning infrastructures, which span multiple public clouds or bridge your on-premise data centre to cloud?
This talk will discuss and demo how you can leverage machine learning technologies such as TensorFlow with your Kafka deployments in public cloud to build a scalable, mission-critical machine learning infrastructure for data preprocessing and ingestion, and model training, deployment and monitoring.
The discussed architecture includes capabilities like scalable data preprocessing for training and predictions, combination of different Deep Learning frameworks, data replication between data centres, intelligent real time microservices running on Kubernetes, and local deployment of analytic models for offline predictions.
Deep Learning UDF for KSQL for Streaming Anomaly Detection of MQTT IoT Sensor Data.:
I built a KSQL UDF for sensor analytics. It leverages the new API features of KSQL to build UDF / UDAF functions easily with Java to do continuous stream processing on incoming events.
Use Case: Connected Cars - Real Time Streaming Analytics using Deep Learning
Continuously process millions of events from connected devices (sensors of cars in this example).
From Monoliths to Microservices - A Journey With Confluent With Gayathri Veal...
Indeed is consciously transforming our monolith applications to microservices. Moving monoliths from on-premise to a hybrid architecture is a non-trivial endeavor. It is as we know a marathon and never never a race when we refactor not all of our applications but, incrementally progress onward to resilience with cloud.
By partnering with Confluent we were able to procedurally migrate many of our workloads both critical and non-critical primarily using Kafka by adopting a data domain driven approach. In this talk, you will learn,
1. How to piece complex puzzles when you have bits of information
2. What questions to ask to prioritize feature improvements
3. How to enumerate impact
4. How to let your vendor know what is valuable
With over 20 years of experience working with various databases and datastores, I will share real examples of success and failures and lessons we learned when working with Confluent Cloud by:
- Implementing strategies
- Addressing short and long term value - for both technical and business
- The very methodical methods to form roadmaps
If you’re in discussions surrounding engineering platforms at your organization then this talk is for you. If you are a data driven engineering organization with solid leadership with sound decisions behind it, join us for this talk and let’s have a discussion.
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Learn the differences between an event-driven streaming platform and middleware like MQ, ETL and ESBs – including best practices and anti-patterns, but also how these concepts and tools complement each other in an enterprise architecture.
Extract-Transform-Load (ETL) is still a widely-used pattern to move data between different systems via batch processing. Due to its challenges in today’s world where real time is the new standard, an Enterprise Service Bus (ESB) is used in many enterprises as integration backbone between any kind of microservice, legacy application or cloud service to move data via SOAP / REST Web Services or other technologies. Stream Processing is often added as its own component in the enterprise architecture for correlation of different events to implement contextual rules and stateful analytics. Using all these components introduces challenges and complexities in development and operations.
This session discusses how teams in different industries solve these challenges by building a native streaming platform from the ground up instead of using ETL and ESB tools in their architecture. This allows to build and deploy independent, mission-critical streaming real time application and microservices. The architecture leverages distributed processing and fault-tolerance with fast failover, no-downtime rolling deployments and the ability to reprocess events, so you can recalculate output when your code changes. Integration and Stream Processing are still key functionality but can be realized in real time natively instead of using additional ETL, ESB or Stream Processing tools.
Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...
Converting production databases into live data streams for Apache Kafka can be labor intensive and costly. As Kafka architectures grow, complexity also rises as data teams begin to configure clusters for redundancy, partitions for performance, as well as for consumer groups for correlated analytics processing. In this breakout session, you’ll hear data streaming success stories from Generali and Skechers that leverage Qlik Data Integration and Confluent. You’ll discover how Qlik’s data integration platform lets organizations automatically produce real-time transaction streams into Kafka, Confluent Platform, or Confluent Cloud, deliver faster business insights from data, enable streaming analytics, as well as streaming ingestion for modern analytics. Learn how these customer use Qlik and Confluent to: - Turn databases into live data feeds - Simplify and automate the real-time data streaming process - Accelerate data delivery to enable real-time analytics Learn how Skechers and Generali breathe new life into data in the cloud, stay ahead of changing demands, while lowering over-reliance on resources, production time and costs.
J-Spring 2018 - A journey from Java EE to Cloud Native microservices
About a decade ago, when the world was still all about J2EE and WebSphere application servers, Rabobank introduced a new Online platform with a custom, intentionally self-built, portal framework. Since then our world has changed in many ways. Nobody does portlets anymore; instead we build rich front-end applications backed by services spitting out JSON. Like most of the rest of the world Rabobank moved from waterfall to Agile to DevOps. Microservices came along. Java is no longer the only preferred language on the server. Today, Rabobank is migrating to a new platform based on Pivotal Cloud Foundry, that intends to cope with all these changes. In this talk Vincent Oostindië will talk you through a decade of IT at Rabobank Online, of course focusing on the new platform and the underlying technologies used there, like Spring Boot and Spring Cloud Services.
Deploying Large Spark Models to production and model scoring in near real time
This document discusses challenges with deploying large machine learning models for scoring in near real-time at scale. It describes an initial solution using AWS Lambda that had limitations around model size and retraining. A better solution called Mleap is proposed, which allows exporting PySpark models as serialized JSON or protobuf formats. This facilitates loading models into a scalable REST API for fast scoring in production without code changes. Mleap supports most common Spark transformers and classifiers for both serialization and deployment.
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka can act as both an enemy and a friend to traditional middleware like message queues, ETL tools, and enterprise service buses. As an enemy, Kafka replaces many of the individual components and limitations of traditional middleware with a single, scalable event streaming platform. However, Kafka can also integrate with traditional middleware as a friend through connectors and client APIs, using traditional tools for specific integrations while relying on Kafka for scalable event collection and processing. In complex environments with both new and legacy systems, Kafka acts as a "frenemy" by facilitating a gradual migration from old middleware to a modern event streaming architecture centered around Kafka.
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka can act as both an enemy and a friend to traditional middleware like message queues, ETL tools, and enterprise service buses. As an enemy, Kafka replaces many of the individual components and provides a single scalable platform for messaging, storage, and processing. However, Kafka can also integrate with traditional middleware as a friend through connectors and client APIs, allowing certain use cases to still leverage existing tools. In complex environments with both new and legacy systems, Kafka acts as a "frenemy" - replacing some functions but integrating with other existing technologies to provide a bridge to new architectures.
Microservices on top of kafka
1) The document discusses how a company gradually transitioned from a monolithic architecture to microservices using Apache Kafka as the backbone.
2) It outlines the steps taken, including defining service responsibilities and data models, using event sourcing and CQRS patterns, designing Kafka topics, and validating data.
3) The document emphasizes that Kafka should be the single source of truth for critical data and applications should be able to reprocess historical data from Kafka topics.
Deploying spark ml models
This document discusses challenges with deploying large machine learning models at scale for near real-time scoring. It describes an initial solution using AWS Lambda that had limitations. A better solution called Mleap is proposed, which allows exporting trained Spark ML models into a serialized format and loading them into a Scala pipeline for fast serving. A demo is shown building a model in PySpark, exporting it with Mleap, and loading it into a Docker container for real-time scoring in 50ms compared to Spark's 1.5 seconds.
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
Namely is a late-stage startup that builds HR, Payroll and Benefits software for mid-sized businesses. Over the years, we've ended up with a number of monolithic and legacy applications covering overlapping domain concepts, which has limited our ability to deliver new and innovative features to our customers. We need a way to get our data out of the monoliths to decouple our systems and increase our velocity. We've chosen Kafka as our way to liberate our data in a reliable, scalable and maintainable way. This talk covers specific examples of successes and missteps in our move to Kafka as the backbone of our architecture. It then looks to the future - where we are trying to go, and how we plan on getting, both from the short term and long term perspectives. Key Takeaways: - Successful and unsuccessful approaches to gradually introducing Kafka to a large organization in a way that meets the short and long term needs of the business. - Successful and unsuccessful patterns for using Kafka. - Pragmaticism versus purisim: Building Kafka-first systems, and migrating legacy systems to Kafka with Debezium. - Combining event driven systems with RPC based systems. Observability, alerting and testing. - Actionable steps that you can take to your organization to help drive adoption.
Performance Metrics Driven CI/CD - Introduction to Continuous Innovation and ...
Deck used for my talk at the 2016 Spring User Conference in Toronto. Deck was followed up by a walkthrough of a Jenkins workflow that deployed to Cloud Foundry based on jmeter test results
How Yelp Leapt to Microservices with More than a Message Queue
Without seeing what’s wrong with today’s messaging queues, it can be initially confusing to view Apache Kafka as more. By adding additional functionality, true storage, and guarantees it opens opportunities to take full advantage of a publish/subscribe model.
Joined by Yelp’s Justin Cunningham we’ll see how their infrastructure has quickly evolved. Powered by Kafka, Yelp has made the leap to microservices and is seeing the benefits of efficiency and performance.
Speakers:
Justin Cunningham
Technical Lead, Software Engineer, Yelp
Gehrig Kunz
Technical Product Marketing Manager, Confluent
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
(Bob Lehmann, Bayer) Kafka Summit SF 2018
You’ve built your streaming data platform. The early adopters are “all in” and have developed producers, consumers and stream processing apps for a number of use cases. A large percentage of the enterprise, however, has expressed interest but hasn’t made the leap. Why?
In 2014, Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform.
In this talk, Bob Lehmann will describe the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob will also talk about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.
Building Microservices with Apache Kafka
Building distributed systems is challenging. Luckily, Apache Kafka provides a powerful toolkit for putting together big services as a set of scalable, decoupled components. In this talk, I'll describe some of the design tradeoffs when building microservices, and how Kafka's powerful abstractions can help. I'll also talk a little bit about what the community has been up to with Kafka Streams, Kafka Connect, and exactly-once semantics.
Presentation by Colin McCabe, Confluent, Big Data Day LA
James Watters Kafka Summit NYC 2019 Keynote
The document discusses how Spring Boot and Kafka can form the basis of a new enterprise application platform that enables continuous delivery and efficient scaling through microservices and event-driven architecture. It provides examples of companies like Netflix and T-Mobile that have successfully adopted this approach. The document advocates an "event-first" design and argues this platform approach allows for arbitrary scaling, multi-cloud deployment, and increased developer autonomy and agility.
Similar to Kafka: Legacy microservices (20)
Event Driven Services Part 2: Building Event-Driven Services with Apache Kafka
Event Driven Services Part 2: Building Event-Driven Services with Apache Kafka
How did we move the mountain? - Migrating 1 trillion+ messages per day across...
How did we move the mountain? - Migrating 1 trillion+ messages per day across...
Unleashing Apache Kafka and TensorFlow in the Cloud
Unleashing Apache Kafka and TensorFlow in the Cloud
From Monoliths to Microservices - A Journey With Confluent With Gayathri Veal...
From Monoliths to Microservices - A Journey With Confluent With Gayathri Veal...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...
Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...
J-Spring 2018 - A journey from Java EE to Cloud Native microservices
J-Spring 2018 - A journey from Java EE to Cloud Native microservices
Deploying Large Spark Models to production and model scoring in near real time
Deploying Large Spark Models to production and model scoring in near real time
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
Performance Metrics Driven CI/CD - Introduction to Continuous Innovation and ...
Performance Metrics Driven CI/CD - Introduction to Continuous Innovation and ...
How Yelp Leapt to Microservices with More than a Message Queue
How Yelp Leapt to Microservices with More than a Message Queue
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
Recently uploaded
Webinar On-Demand: Using Flutter for Embedded
Flutter is a popular open source, cross-platform framework developed by Google. In this webinar we'll explore Flutter and its architecture, delve into the Flutter Embedder and Flutter’s Dart language, discover how to leverage Flutter for embedded device development, learn about Automotive Grade Linux (AGL) and its consortium and understand the rationale behind AGL's choice of Flutter for next-gen IVI systems. Don’t miss this opportunity to discover whether Flutter is right for your project.
Unveiling the Advantages of Agile Software Development.pdf
Learn about Agile Software Development's advantages. Simplify your workflow to spur quicker innovation. Jump right in! We have also discussed the advantages.
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing non-technical founders before engaging a tech agency is crucial for the success of their projects. It starts with clearly defining their vision and goals, conducting thorough market research, and gaining a basic understanding of relevant technologies. Setting realistic expectations and preparing a detailed project brief are essential steps. Founders should select a tech agency with a proven track record and establish clear communication channels. Additionally, addressing legal and contractual considerations and planning for post-launch support are vital to ensure a smooth and successful collaboration. This preparation empowers non-technical founders to effectively communicate their needs and work seamlessly with their chosen tech agency.Visit our site to get more details about this. Contact us today www.ishtechnologies.com.au
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM 🐰
一比一原版(USF毕业证)旧金山大学毕业证如何办理
USF硕士毕业证成绩单【微信95270640】一比一伪造旧金山大学文凭@假冒USF毕业证成绩单+Q微信95270640办理USF学位证书@仿造USF毕业文凭证书@购买旧金山大学毕业证成绩单USF真实使馆认证/真实留信认证回国人员证明
#一整套旧金山大学文凭证件办理#—包含旧金山大学旧金山大学本科毕业证成绩单学历认证|使馆认证|归国人员证明|教育部认证|留信网认证永远存档教育部学历学位认证查询办理国外文凭国外学历学位认证#我们提供全套办理服务。
一整套留学文凭证件服务：
一：旧金山大学旧金山大学本科毕业证成绩单毕业证 #成绩单等全套材料从防伪到印刷水印底纹到钢印烫金
二：真实使馆认证（留学人员回国证明）使馆存档
三：真实教育部认证教育部存档教育部留服网站永久可查
四：留信认证留学生信息网站永久可查
国外毕业证学位证成绩单办理方法：
1客户提供办理旧金山大学旧金山大学本科毕业证成绩单信息：姓名生日专业学位毕业时间等（如信息不确定可以咨询顾问：我们有专业老师帮你查询）；
2开始安排制作毕业证成绩单电子图；
3毕业证成绩单电子版做好以后发送给您确认；
4毕业证成绩单电子版您确认信息无误之后安排制作成品；
5成品做好拍照或者视频给您确认；
6快递给客户（国内顺丰国外DHLUPS等快读邮寄）。
教育部文凭学历认证认证的用途：
如果您计划在国内发展那么办理国内教育部认证是必不可少的。事业性用人单位如银行国企公务员在您应聘时都会需要您提供这个认证。其他私营 #外企企业无需提供！办理教育部认证所需资料众多且烦琐所有材料您都必须提供原件我们凭借丰富的经验帮您快速整合材料让您少走弯路。
实体公司专业为您服务如有需要请联系我: 微信95270640奈一次次令他失望山娃今年岁上五年级识得很多字从走出小屋开始山娃就知道父亲的家和工地共有一个很动听的名字——天河工地的底层空空荡荡很宽阔很凉爽在地上铺上报纸和水泥袋父亲和工人们中午全睡在地上地面坑坑洼洼山娃曾多次绊倒过也曾有长铁钉穿透凉鞋刺在脚板上但山娃不怕工地上也常有五六个从乡下来的小学生他们的父母亲也是高楼上的建筑工人小伙伴来自不同省份都操着带有浓重口音的普通话可不知为啥山娃不仅很快与他们熟识了
Malibou Pitch Deck For Its €3M Seed Round
French start-up Malibou raised a €3 million Seed Round to develop its payroll and human resources
management platform for VSEs and SMEs. The financing round was led by investors Breega, Y Combinator, and FCVC.
UI5con 2024 - Bring Your Own Design System
How do you combine the OpenUI5/SAPUI5 programming model with a design system that makes its controls available as Web Components? Since OpenUI5/SAPUI5 1.120, the framework supports the integration of any Web Components. This makes it possible, for example, to natively embed own Web Components of your design system which are created with Stencil. The integration embeds the Web Components in a way that they can be used naturally in XMLViews, like with standard UI5 controls, and can be bound with data binding. Learn how you can also make use of the Web Components base class in OpenUI5/SAPUI5 to also integrate your Web Components and get inspired by the solution to generate a custom UI5 library providing the Web Components control wrappers for the native ones.
What next after learning python programming basics
What next after learning python programming basics
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
Presentation from Ghazal Aakel and Eric Jochum at the GSD Community Stage Meetup on June 6th, 2024
How Can Hiring A Mobile App Development Company Help Your Business Grow?
ToXSL Technologies is an award-winning Mobile App Development Company in Dubai that helps businesses reshape their digital possibilities with custom app services. As a top app development company in Dubai, we offer highly engaging iOS & Android app solutions. https://rb.gy/necdnt
E-commerce Development Services- Hornet Dynamics
For any business hoping to succeed in the digital age, having a strong online presence is crucial. We offer Ecommerce Development Services that are customized according to your business requirements and client preferences, enabling you to create a dynamic, safe, and user-friendly online store.
WWDC 2024 Keynote Review: For CocoaCoders Austin
Overview of WWDC 2024 Keynote Address.
Covers: Apple Intelligence, iOS18, macOS Sequoia, iPadOS, watchOS, visionOS, and Apple TV+.
Understandable dialogue on Apple TV+
On-device app controlling AI.
Access to ChatGPT with a guest appearance by Chief Data Thief Sam Altman!
App Locking! iPhone Mirroring! And a Calculator!!
Oracle 23c New Features For DBAs and Developers.pptx
Most important New features of Oracle 23c for DBAs and Developers. You can get more idea from my youtube channel video from https://youtu.be/XvL5WtaC20A
Measures in SQL (SIGMOD 2024, Santiago, Chile)
SQL has attained widespread adoption, but Business Intelligence tools still use their own higher level languages based upon a multidimensional paradigm. Composable calculations are what is missing from SQL, and we propose a new kind of column, called a measure, that attaches a calculation to a table. Like regular tables, tables with measures are composable and closed when used in queries.
SQL-with-measures has the power, conciseness and reusability of multidimensional languages but retains SQL semantics. Measure invocations can be expanded in place to simple, clear SQL.
To define the evaluation semantics for measures, we introduce context-sensitive expressions (a way to evaluate multidimensional expressions that is consistent with existing SQL semantics), a concept called evaluation context, and several operations for setting and modifying the evaluation context.
A talk at SIGMOD, June 9–15, 2024, Santiago, Chile
Authors: Julian Hyde (Google) and John Fremlin (Google)
https://doi.org/10.1145/3626246.3653374
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
UMN硕士毕业证成绩单【微信95270640】购买（明尼苏达大学毕业证成绩单硕士学历）Q微信95270640代办UMN学历认证留信网伪造明尼苏达大学学位证书精仿明尼苏达大学本科/硕士文凭证书补办明尼苏达大学 diplomaoffer,Transcript购买明尼苏达大学毕业证成绩单购买UMN假毕业证学位证书购买伪造明尼苏达大学文凭证书学位证书,专业办理雅思、托福成绩单，学生ID卡，在读证明，海外各大学offer录取通知书，毕业证书，成绩单，文凭等材料:1:1完美还原毕业证、offer录取通知书、学生卡等各种在读或毕业材料的防伪工艺（包括 烫金、烫银、钢印、底纹、凹凸版、水印、防伪光标、热敏防伪、文字图案浮雕，激光镭射，紫外荧光，温感光标）学校原版上有的工艺我们一样不会少，不论是老版本还是最新版本，都能保证最高程度还原，力争完美以求让所有同学都能享受到完美的品质服务。
#毕业证成绩单 #毕业証 #成绩单 #學生卡 #OFFER录取通知书 #雅思#托福等……
国外大学明尼苏达大学明尼苏达大学毕业证offer制作方法（一对一专业服务）
1客户提供办理信息：姓名生日专业学位毕业时间等（如信息不确定可以咨询顾问：我们有专业老师帮你查询）；
2开始安排制作毕业证成绩单电子图；
3毕业证成绩单电子版做好以后发送给您确认；
4毕业证成绩单电子版您确认信息无误之后安排制作成品；
5成品做好拍照或者视频给您确认；
6快递给客户（国内顺丰国外DHLUPS等快读邮寄）
— — 制作工艺 【高仿真】— —
凭借多年的制作经验本公司制作明尼苏达大学明尼苏达大学毕业证offer《激光》《水印》《钢印》《烫金》《紫外线》凹凸版uv版等防伪技术一流高精仿度几乎跟学校100%相同！让您绝对满意。
— — -公司理念 【诚信为主】— — —
我們以質量求生存.以服务求发展有雄厚的实力专业的团队咨询顾问为您细心解答可详谈是真是假眼见为实让您真正放心平凡人生,尽我所能助您一臂之力让我們携手圆您梦想!
此贴长年有效【贴心专线/微-信: 95270640】敬请保留此联系方式以备用！如有不在线请给我们留言！我们将在第一时间给您回复!上散发着一抹抹的光晕而这每处自然形成的细节融合在一起浑然天成的美实在令人心生愉悦小道的周边无秩序的生长着几株艳丽的野花红的粉的紫的虽混乱无章却给这幅美景更增添一份性感夹杂着一份纯洁的妖娆毫无违和感实在给人带来一份悠然幸福的心情如果说现在的审美已经断然拒绝了无声的话那么在树林间飞掠而过的小鸟叽叽咋咋的叫声是否就是这最后的点睛之笔悠然走在林间的小路上宁静与清香一丝丝的盛夏气息吸入身体昔日生活里的繁忙多
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Consistent toolbox talks are critical for maintaining workplace safety, as they provide regular opportunities to address specific hazards and reinforce safe practices.
These brief, focused sessions ensure that safety is a continual conversation rather than a one-time event, which helps keep safety protocols fresh in employees' minds. Studies have shown that shorter, more frequent training sessions are more effective for retention and behavior change compared to longer, infrequent sessions.
Engaging workers regularly, toolbox talks promote a culture of safety, empower employees to voice concerns, and ultimately reduce the likelihood of accidents and injuries on site.
The traditional method of conducting safety talks with paper documents and lengthy meetings is not only time-consuming but also less effective. Manual tracking of attendance and compliance is prone to errors and inconsistencies, leading to gaps in safety communication and potential non-compliance with OSHA regulations. Switching to a digital solution like Safelyio offers significant advantages.
Safelyio automates the delivery and documentation of safety talks, ensuring consistency and accessibility. The microlearning approach breaks down complex safety protocols into manageable, bite-sized pieces, making it easier for employees to absorb and retain information.
This method minimizes disruptions to work schedules, eliminates the hassle of paperwork, and ensures that all safety communications are tracked and recorded accurately. Ultimately, using a digital platform like Safelyio enhances engagement, compliance, and overall safety performance on site. https://safelyio.com/
Energy consumption of Database Management - Florina Jonuzi
Presentation from Florina Jonuzi at the GSD Community Stage Meetup on June 06, 2024
Recently uploaded (20)
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing Non - Technical Founders for Engaging a Tech Agency
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
What next after learning python programming basics
What next after learning python programming basics
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Energy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina Jonuzi
Kafka: Legacy microservices
- 1. 1 Kafka: Smoothly migrate your legacy monolith to your new micro-services oriented architecture. Brice LEPORINI - brice@confluent.io @blep
- 2. 2 Me, myself and I... Brice LEPORINI, 44 Systems Engineer @Confluent Formerly developper : Java, Scala and JS (under threat only)
- 3. 33 Once upon a time, in the retailing business...
- 4. 4 FCC is a fictive retailing company, operating physical points of sales as well as an online platform FCC is currently running its platform since more than 10 years ● Desktop web application ● Mobile APIs ● Back-office applications Its architecture is based on a monolithic application running a RDBMS Key figures $1 B revenue for 2018 40k+ orders per week 1k+ point of sales Aggressive revenue growth target challenges are expected by FCC’s top management for the next 3 years
- 5. 5
- 6. 6 Who said big bang?
- 7. 77 Changing the engine on a plane while in flight
- 8. 88 Gradual migration No big bang Strangler application pattern Features are gradually swapped with new MS Canary release
- 9. 9 The legacy and the micro-services have to run side by side. They can modify concurrently same data. Conflictless!
- 10. 10 Step 1 : push changes from legacy to MS @blep
- 11. 11 A sample from a broader physical model @blep
- 12. 12 Step 1 : push changes from legacy to MS @blep
- 13. 1313 Demo time!
- 14. 14 Why Kafka? • Scalability • Retention • Durability • Replication • Security • Resiliency • Throughput • Ordering • Exactly Once Semantic • Transaction • Idempotency • Immutability • … @blep
- 15. 15 Step 1 : push changes from legacy to MS @blep
- 16. 16 @blep
- 17. 17 Step 1 : push changes from legacy to MS @blep
- 18. 18 A few words about duality of Streams and Tables @blep
- 19. 19 Step 1 : push changes from legacy to MS @blep
- 20. 20 Why Kafka Streams? Higher level API: transformation, filtering Stateful features: ● Aggregations ● Joins ● Windowing All are resilient and able to scale out! @blep
- 21. 2121 Demo time!
- 22. 22 @blep
- 23. 23 Step 2: push changes from MS to legacy @blep
- 24. 24 Step 2: push changes from MS to legacy @blep
- 25. 25 Entity state in MS environment = Last entity found in the snapshot + Events without validation
- 26. 26 Yes it is a kind of CQRS + ES usage CQRS = Command Query Responsability Segregation ES = Event Sourcing
- 27. 27 @blep
- 28. 28 @blep
- 29. 29 I tried to sell you a dream! CDC highly needs your attention in different ways depending on the DB you use JSON usage makes data governance at risk Needs more code to support updates This code for presentation purpose ! There’s still a lot of corner cases to cover! @blep
- 30. 3030 But when you cover all cases, this kind of architecture rocks!
- 31. 31 In conclusion No big bang ✅ Gradual migration ✅ Ability to rollback ✅ No production stop ✅ Derisked data migration ✅
Editor's Notes
- I’m going to make the case that databases and software architectures are only 50% complete, that there is a fundamental missing ingredient that we don’t represent and that capturing this will be a major leap forward in how we think about the architecture of data systems and of companies and that Kafka is they key building block spurring this change. This is a little bit of an outrageous claim. After all there is nothing really new in computer science and we’ve been building systems for a while, if there was something this fundamental we’d know about it, right? Well in a way we do. What I’m talking about is of course....