Vous apprendrez également à :
• Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données
• Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience
• Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés
The Top 5 Apache Kafka Use Cases and Architectures in 2022Kai Wähner
I see the following topics coming up more regularly in conversations with customers, prospects, and the broader Kafka community across the globe:
Kappa Architecture: Kappa goes mainstream to replace Lambda and Batch pipelines (that does not mean that there is no batch processing anymore). Examples: Kafka-powered Kappa architectures from Uber, Disney, Shopify, and Twitter.
Hyper-personalized Omnichannel: Retail and customer communication across online and offline channels becomes the new black, including context-specific upselling, recommendations, and location-based services. Examples: Omnichannel Retail and Customer 360 in Real-Time with Apache Kafka.
Multi-Cloud Deployments: Business units and IT infrastructures span across regions, continents, and cloud providers. Linking clusters for bi-directional replication of data in real-time becomes crucial for many business models. Examples: Global Kafka deployments.
Edge Analytics: Low latency requirements, cost efficiency, or security requirements enforce the deployment of (some) event streaming use cases at the far edge (i.e., outside a data center), for instance, for predictive maintenance and quality assurance on the shop floor level in smart factories. Examples: Edge analytics with Kafka.
Real-time Cybersecurity: Situational awareness and threat intelligence need to process massive data in real-time to defend against cyberattacks successfully. The many successful ransomware attacks across the globe in 2021 were a warning for most CIOs. Examples: Cybersecurity for situational awareness and threat intelligence in real-time.
Real-Life Use Cases & Architectures for Event Streaming with Apache KafkaKai Wähner
Streaming all over the World: Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka.
Learn about various case studies for event streaming with Apache Kafka across industries. The talk explores architectures for real-world deployments from Audi, BMW, Disney, Generali, Paypal, Tesla, Unity, Walmart, William Hill, and more. Use cases include fraud detection, mainframe offloading, predictive maintenance, cybersecurity, edge computing, track&trace, live betting, and much more.
Meetup: Streaming Data Pipeline DevelopmentTimothy Spann
Meetup: Streaming Data Pipeline Development
In this interactive session, Tim will lead participants through how to best build streaming data pipelines. He will cover how to build applications from some common use cases and highlight tips, tricks, best practices and patterns.
He will show how to build the easy way and then dive deep into the underlying open source technologies including Apache NiFi, Apache Flink, Apache Kafka and Apache Iceberg.
If you wish to follow along, please download open source projects beforehand. You can also download this helpful streaming platform: https://docs.cloudera.com/csp-ce/latest/installation/topics/csp-ce-installing-ce.html
All source code and slides will be shared for those interested in building their own FLaNK Apps. https://www.flankstack.dev/
You can join the meeting virtually here:
https://cloudera.zoom.us/j/91603330726
Speaker - Tim Spann
Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science.
Apache Kafka is the de facto standard for data streaming to process data in motion. With its significant adoption growth across all industries, I get a very valid question every week: When NOT to use Apache Kafka? What limitations does the event streaming platform have? When does Kafka simply not provide the needed capabilities? How to qualify Kafka out as it is not the right tool for the job?
This session explores the DOs and DONTs. Separate sections explain when to use Kafka, when NOT to use Kafka, and when to MAYBE use Kafka.
No matter if you think about open source Apache Kafka, a cloud service like Confluent Cloud, or another technology using the Kafka protocol like Redpanda or Pulsar, check out this slide deck.
A detailed article about this topic:
https://www.kai-waehner.de/blog/2022/01/04/when-not-to-use-apache-kafka/
An Introduction to Confluent Cloud: Apache Kafka as a Serviceconfluent
Business breakout during Confluent’s streaming event in Munich, presented by Hans Jespersen, VP WW Systems Engineering at Confluent. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.
The Top 5 Apache Kafka Use Cases and Architectures in 2022Kai Wähner
I see the following topics coming up more regularly in conversations with customers, prospects, and the broader Kafka community across the globe:
Kappa Architecture: Kappa goes mainstream to replace Lambda and Batch pipelines (that does not mean that there is no batch processing anymore). Examples: Kafka-powered Kappa architectures from Uber, Disney, Shopify, and Twitter.
Hyper-personalized Omnichannel: Retail and customer communication across online and offline channels becomes the new black, including context-specific upselling, recommendations, and location-based services. Examples: Omnichannel Retail and Customer 360 in Real-Time with Apache Kafka.
Multi-Cloud Deployments: Business units and IT infrastructures span across regions, continents, and cloud providers. Linking clusters for bi-directional replication of data in real-time becomes crucial for many business models. Examples: Global Kafka deployments.
Edge Analytics: Low latency requirements, cost efficiency, or security requirements enforce the deployment of (some) event streaming use cases at the far edge (i.e., outside a data center), for instance, for predictive maintenance and quality assurance on the shop floor level in smart factories. Examples: Edge analytics with Kafka.
Real-time Cybersecurity: Situational awareness and threat intelligence need to process massive data in real-time to defend against cyberattacks successfully. The many successful ransomware attacks across the globe in 2021 were a warning for most CIOs. Examples: Cybersecurity for situational awareness and threat intelligence in real-time.
Real-Life Use Cases & Architectures for Event Streaming with Apache KafkaKai Wähner
Streaming all over the World: Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka.
Learn about various case studies for event streaming with Apache Kafka across industries. The talk explores architectures for real-world deployments from Audi, BMW, Disney, Generali, Paypal, Tesla, Unity, Walmart, William Hill, and more. Use cases include fraud detection, mainframe offloading, predictive maintenance, cybersecurity, edge computing, track&trace, live betting, and much more.
Meetup: Streaming Data Pipeline DevelopmentTimothy Spann
Meetup: Streaming Data Pipeline Development
In this interactive session, Tim will lead participants through how to best build streaming data pipelines. He will cover how to build applications from some common use cases and highlight tips, tricks, best practices and patterns.
He will show how to build the easy way and then dive deep into the underlying open source technologies including Apache NiFi, Apache Flink, Apache Kafka and Apache Iceberg.
If you wish to follow along, please download open source projects beforehand. You can also download this helpful streaming platform: https://docs.cloudera.com/csp-ce/latest/installation/topics/csp-ce-installing-ce.html
All source code and slides will be shared for those interested in building their own FLaNK Apps. https://www.flankstack.dev/
You can join the meeting virtually here:
https://cloudera.zoom.us/j/91603330726
Speaker - Tim Spann
Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science.
Apache Kafka is the de facto standard for data streaming to process data in motion. With its significant adoption growth across all industries, I get a very valid question every week: When NOT to use Apache Kafka? What limitations does the event streaming platform have? When does Kafka simply not provide the needed capabilities? How to qualify Kafka out as it is not the right tool for the job?
This session explores the DOs and DONTs. Separate sections explain when to use Kafka, when NOT to use Kafka, and when to MAYBE use Kafka.
No matter if you think about open source Apache Kafka, a cloud service like Confluent Cloud, or another technology using the Kafka protocol like Redpanda or Pulsar, check out this slide deck.
A detailed article about this topic:
https://www.kai-waehner.de/blog/2022/01/04/when-not-to-use-apache-kafka/
An Introduction to Confluent Cloud: Apache Kafka as a Serviceconfluent
Business breakout during Confluent’s streaming event in Munich, presented by Hans Jespersen, VP WW Systems Engineering at Confluent. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.
Apache Kafka Streams + Machine Learning / Deep LearningKai Wähner
Machine Learning and Deep Learning Applied to Real Time with Apache Kafka Streams...
Big Data and Machine Learning are key for innovation in many industries today. Large amounts of historical data are stored and analyzed in Hadoop, Spark or other clusters to find patterns and insights, e.g. for predictive maintenance, fraud detection or cross-selling.
This first part of the session explains how to build analytic models with R, Python and Scala leveraging open source machine learning / deep learning frameworks like Apache Spark, TensorFlow or H2O.ai. The second part discusses how to leverage these built analytic models in your own streaming applications or microservices; leveraging the Apache Kafka cluster and Kafka Streams instead of building an own stream processing cluster. The session focuses on live demos and teaches lessons learned for executing analytic models in a highly scalable and performant way.
The last part explains how Apache Kafka can help to move from a manual build and deployment of analytic models to continuous online model improvement in real time.
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and LinkerdKai Wähner
Microservice architectures are not free lunch! Microservices need to be decoupled, flexible, operationally transparent, data aware and elastic. Most material from last years only discusses point-to-point architectures with inflexible and non-scalable technologies like REST / HTTP. This video takes a look at cutting edge technologies like Apache Kafka, Kubernetes, Envoy, Linkerd and Istio to implement a cloud-native service mesh to solve these challenges and bring microservices to the next level of scale, speed and efficiency.
Key takeaways:
- Apache Kafka decouples services, including event streams and request-response
- Kubernetes provides a cloud-native infrastructure for the Kafka ecosystem
- Service Mesh helps with security and observability at ecosystem / organization scale
- Envoy and Istio sit in the layer above Kafka and are orthogonal to the goals Kafka addresses
Blog post: http://www.kai-waehner.de/blog/2019/09/24/cloud-native-apache-kafka-kubernetes-envoy-istio-linkerd-service-mesh
Video recording of this slide deck: https://youtu.be/Us_C4RFOUrA
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureKai Wähner
Apache Kafka in conjunction with Apache Spark became the de facto standard for processing and analyzing data. Both frameworks are open, flexible, and scalable.
Unfortunately, the latter makes operations a challenge for many teams. Ideally, teams can use serverless SaaS offerings to focus on business logic. However, hybrid and multi-cloud scenarios require a cloud-native platform that provides automated and elastic tooling to reduce the operations burden.
This session explores different architectures to build serverless Apache Kafka and Apache Spark multi-cloud architectures across regions and continents.
We start from the analytics perspective of a data lake and explore its relation to a fully integrated data streaming layer with Kafka to build a modern data Data Lakehouse.
Real-world use cases show the joint value and explore the benefit of the "delta lake" integration.
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...Kai Wähner
Architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments
Multi-cluster and cross-data center deployments of Apache Kafka have become the norm rather than an exception. This session gives an overview of several scenarios that may require multi-cluster solutions and discusses real-world examples with their specific requirements and trade-offs, including disaster recovery, aggregation for analytics, cloud migration, mission-critical stretched deployments and global Kafka.
Key takeaways:
In many scenarios, one Kafka cluster is not enough. Understand different architectures and alternatives for multi-cluster deployments.
Zero data loss and high availability are two key requirements. Understand how to realize this, including trade-offs.
Learn about features and limitations of Kafka for multi cluster deployments
Global Kafka and mission-critical multi-cluster deployments with zero data loss and high availability became the normal, not an exception.
Kafka and Confluent are nice, but what about the integration with public clouds like Azure. Or even better, to integrate Kafka and Confluent with a managed API management like Azure API Gateway.
In this talk I will show you how it is possible to integrate an event streaming platform like Confluent into an enterprise API Management and different other services to build up a lambda based data platform architecture.
Kafka's basic terminologies, its architecture, its protocol and how it works.
Kafka at scale, its caveats, guarantees and use cases offered by it.
How we use it @ZaprMediaLabs.
Kafka for Real-Time Replication between Edge and Hybrid CloudKai Wähner
Not all workloads allow cloud computing. Low latency, cybersecurity, and cost-efficiency require a suitable combination of edge computing and cloud integration.
This session explores architectures and design patterns for software and hardware considerations to deploy hybrid data streaming with Apache Kafka anywhere. A live demo shows data synchronization from the edge to the public cloud across continents with Kafka on Hivecell and Confluent Cloud.
Watch this talk here: https://www.confluent.io/online-talks/apache-kafka-architecture-and-fundamentals-explained-on-demand
This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput.
This talk provides a comprehensive overview of Kafka architecture and internal functions, including:
-Topics, partitions and segments
-The commit log and streams
-Brokers and broker replication
-Producer basics
-Consumers, consumer groups and offsets
This session is part 2 of 4 in our Fundamentals for Apache Kafka series.
Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will dig into how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to
hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Apache Camel v3, Camel K and Camel QuarkusClaus Ibsen
In this session, we will explore key challenges with function interactions and coordination, addressing these problems using Enterprise Integration Patterns (EIP) and modern approaches with the latest innovations from the Apache Camel community:
Apache Camel is the Swiss army knife of integration, and the most powerful integration framework. In this session you will hear about the latest features in the brand new 3rd generation.
Camel K, is a lightweight integration platform that enables Enterprise Integration Patterns to be used natively on any Kubernetes cluster. When used in combination with Knative, a framework that adds serverless building blocks to Kubernetes, and the subatomic execution environment of Quarkus, Camel K can mix serverless features such as auto-scaling, scaling to zero, and event-based communication with the outstanding integration capabilities of Apache Camel.
- Apache Camel 3
- Camel K
- Camel Quarkus
We will show how Camel K works. We’ll also use examples to demonstrate how Camel K makes it easier to connect to cloud services or enterprise applications using some of the 300 components that Camel provides.
Benefits of Stream Processing and Apache Kafka Use Casesconfluent
Watch this talk here: https://www.confluent.io/online-talks/benefits-of-stream-processing-and-apache-kafka-use-cases-on-demand
This talk explains how companies are using event-driven architecture to transform their business and how Apache Kafka serves as the foundation for streaming data applications.
Learn how major players in the market are using Kafka in a wide range of use cases such as microservices, IoT and edge computing, core banking and fraud detection, cyber data collection and dissemination, ESB replacement, data pipelining, ecommerce, mainframe offloading and more.
Also discussed in this talk are the differences between Apache Kafka and Confluent Platform.
This session is part 1 of 4 in our Fundamentals for Apache Kafka series.
ksqlDB: A Stream-Relational Database Systemconfluent
Speaker: Matthias J. Sax, Software Engineer, Confluent
ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka®, a distributed event streaming platform. In this talk, we discuss ksqlDB’s architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries while achieving fault tolerance and high vailability. Furthermore, we explore ksqlDB’s streaming SQL dialect and the different types of supported queries.
Matthias J. Sax is a software engineer at Confluent working on ksqlDB. He mainly contributes to Kafka Streams, Apache Kafka's stream processing library, which serves as ksqlDB's execution engine. Furthermore, he helps evolve ksqlDB's "streaming SQL" language. In the past, Matthias also contributed to Apache Flink and Apache Storm and he is an Apache committer and PMC member. Matthias holds a Ph.D. from Humboldt University of Berlin, where he studied distributed data stream processing systems.
https://db.cs.cmu.edu/events/quarantine-db-talk-2020-confluent-ksqldb-a-stream-relational-database-system/
Top 5 Event Streaming Use Cases for 2021 with Apache KafkaKai Wähner
Apache Kafka and Event Streaming are two of the most relevant buzzwords in tech these days. Ever wonder what the predicted TOP 5 Event Streaming Architectures and Use Cases for 2021 are? Check out the following presentation. Learn about edge deployments, hybrid and multi-cloud architectures, service mesh-based microservices, streaming machine learning, and cybersecurity.
On-demand video recording: https://videos.confluent.io/watch/XAjxV3j8hzwCcEKoZVErUJ
Best Practices for Streaming IoT Data with MQTT and Apache KafkaKai Wähner
Organizations today are looking to stream IoT data to Apache Kafka. However, connecting tens of thousands or even millions of devices over unreliable networks can create some architecture challenges. In this session, we will identify and demo some best practices for implementing a large scale IoT system that can stream MQTT messages to Apache Kafka.
We use HiveMQ as open source MQTT broker to ingest data from IoT devices, ingest the data in real time into an Apache Kafka cluster for preprocessing (using Kafka Streams / KSQL), and model training + inference (using TensorFlow 2.0 and its TensorFlow I/O Kafka plugin).
We leverage additional enterprise components from HiveMQ and Confluent to allow easy operations, scalability and monitoring.
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Flink Forward
Flink Forward San Francisco 2022.
Flink consumers read from Kafka as a scalable, high throughput, and low latency data source. However, there are challenges in scaling out data streams where migration and multiple Kafka clusters are required. Thus, we introduced a new Kafka source to read sharded data across multiple Kafka clusters in a way that conforms well with elastic, dynamic, and reliable infrastructure. In this presentation, we will present the source design and how the solution increases application availability while reducing maintenance toil. Furthermore, we will describe how we extended the existing KafkaSource to provide mechanisms to read logical streams located on multiple clusters, to dynamically adapt to infrastructure changes, and to perform transparent cluster migrations and failover.
by
Mason Chen
Confluent Partner Tech Talk with Synthesisconfluent
A discussion on the arduous planning process, and deep dive into the design/architectural decisions.
Learn more about the networking, RBAC strategies, the automation, and the deployment plan.
Apache Kafka Streams + Machine Learning / Deep LearningKai Wähner
Machine Learning and Deep Learning Applied to Real Time with Apache Kafka Streams...
Big Data and Machine Learning are key for innovation in many industries today. Large amounts of historical data are stored and analyzed in Hadoop, Spark or other clusters to find patterns and insights, e.g. for predictive maintenance, fraud detection or cross-selling.
This first part of the session explains how to build analytic models with R, Python and Scala leveraging open source machine learning / deep learning frameworks like Apache Spark, TensorFlow or H2O.ai. The second part discusses how to leverage these built analytic models in your own streaming applications or microservices; leveraging the Apache Kafka cluster and Kafka Streams instead of building an own stream processing cluster. The session focuses on live demos and teaches lessons learned for executing analytic models in a highly scalable and performant way.
The last part explains how Apache Kafka can help to move from a manual build and deployment of analytic models to continuous online model improvement in real time.
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and LinkerdKai Wähner
Microservice architectures are not free lunch! Microservices need to be decoupled, flexible, operationally transparent, data aware and elastic. Most material from last years only discusses point-to-point architectures with inflexible and non-scalable technologies like REST / HTTP. This video takes a look at cutting edge technologies like Apache Kafka, Kubernetes, Envoy, Linkerd and Istio to implement a cloud-native service mesh to solve these challenges and bring microservices to the next level of scale, speed and efficiency.
Key takeaways:
- Apache Kafka decouples services, including event streams and request-response
- Kubernetes provides a cloud-native infrastructure for the Kafka ecosystem
- Service Mesh helps with security and observability at ecosystem / organization scale
- Envoy and Istio sit in the layer above Kafka and are orthogonal to the goals Kafka addresses
Blog post: http://www.kai-waehner.de/blog/2019/09/24/cloud-native-apache-kafka-kubernetes-envoy-istio-linkerd-service-mesh
Video recording of this slide deck: https://youtu.be/Us_C4RFOUrA
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureKai Wähner
Apache Kafka in conjunction with Apache Spark became the de facto standard for processing and analyzing data. Both frameworks are open, flexible, and scalable.
Unfortunately, the latter makes operations a challenge for many teams. Ideally, teams can use serverless SaaS offerings to focus on business logic. However, hybrid and multi-cloud scenarios require a cloud-native platform that provides automated and elastic tooling to reduce the operations burden.
This session explores different architectures to build serverless Apache Kafka and Apache Spark multi-cloud architectures across regions and continents.
We start from the analytics perspective of a data lake and explore its relation to a fully integrated data streaming layer with Kafka to build a modern data Data Lakehouse.
Real-world use cases show the joint value and explore the benefit of the "delta lake" integration.
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...Kai Wähner
Architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments
Multi-cluster and cross-data center deployments of Apache Kafka have become the norm rather than an exception. This session gives an overview of several scenarios that may require multi-cluster solutions and discusses real-world examples with their specific requirements and trade-offs, including disaster recovery, aggregation for analytics, cloud migration, mission-critical stretched deployments and global Kafka.
Key takeaways:
In many scenarios, one Kafka cluster is not enough. Understand different architectures and alternatives for multi-cluster deployments.
Zero data loss and high availability are two key requirements. Understand how to realize this, including trade-offs.
Learn about features and limitations of Kafka for multi cluster deployments
Global Kafka and mission-critical multi-cluster deployments with zero data loss and high availability became the normal, not an exception.
Kafka and Confluent are nice, but what about the integration with public clouds like Azure. Or even better, to integrate Kafka and Confluent with a managed API management like Azure API Gateway.
In this talk I will show you how it is possible to integrate an event streaming platform like Confluent into an enterprise API Management and different other services to build up a lambda based data platform architecture.
Kafka's basic terminologies, its architecture, its protocol and how it works.
Kafka at scale, its caveats, guarantees and use cases offered by it.
How we use it @ZaprMediaLabs.
Kafka for Real-Time Replication between Edge and Hybrid CloudKai Wähner
Not all workloads allow cloud computing. Low latency, cybersecurity, and cost-efficiency require a suitable combination of edge computing and cloud integration.
This session explores architectures and design patterns for software and hardware considerations to deploy hybrid data streaming with Apache Kafka anywhere. A live demo shows data synchronization from the edge to the public cloud across continents with Kafka on Hivecell and Confluent Cloud.
Watch this talk here: https://www.confluent.io/online-talks/apache-kafka-architecture-and-fundamentals-explained-on-demand
This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput.
This talk provides a comprehensive overview of Kafka architecture and internal functions, including:
-Topics, partitions and segments
-The commit log and streams
-Brokers and broker replication
-Producer basics
-Consumers, consumer groups and offsets
This session is part 2 of 4 in our Fundamentals for Apache Kafka series.
Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will dig into how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to
hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Apache Camel v3, Camel K and Camel QuarkusClaus Ibsen
In this session, we will explore key challenges with function interactions and coordination, addressing these problems using Enterprise Integration Patterns (EIP) and modern approaches with the latest innovations from the Apache Camel community:
Apache Camel is the Swiss army knife of integration, and the most powerful integration framework. In this session you will hear about the latest features in the brand new 3rd generation.
Camel K, is a lightweight integration platform that enables Enterprise Integration Patterns to be used natively on any Kubernetes cluster. When used in combination with Knative, a framework that adds serverless building blocks to Kubernetes, and the subatomic execution environment of Quarkus, Camel K can mix serverless features such as auto-scaling, scaling to zero, and event-based communication with the outstanding integration capabilities of Apache Camel.
- Apache Camel 3
- Camel K
- Camel Quarkus
We will show how Camel K works. We’ll also use examples to demonstrate how Camel K makes it easier to connect to cloud services or enterprise applications using some of the 300 components that Camel provides.
Benefits of Stream Processing and Apache Kafka Use Casesconfluent
Watch this talk here: https://www.confluent.io/online-talks/benefits-of-stream-processing-and-apache-kafka-use-cases-on-demand
This talk explains how companies are using event-driven architecture to transform their business and how Apache Kafka serves as the foundation for streaming data applications.
Learn how major players in the market are using Kafka in a wide range of use cases such as microservices, IoT and edge computing, core banking and fraud detection, cyber data collection and dissemination, ESB replacement, data pipelining, ecommerce, mainframe offloading and more.
Also discussed in this talk are the differences between Apache Kafka and Confluent Platform.
This session is part 1 of 4 in our Fundamentals for Apache Kafka series.
ksqlDB: A Stream-Relational Database Systemconfluent
Speaker: Matthias J. Sax, Software Engineer, Confluent
ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka®, a distributed event streaming platform. In this talk, we discuss ksqlDB’s architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries while achieving fault tolerance and high vailability. Furthermore, we explore ksqlDB’s streaming SQL dialect and the different types of supported queries.
Matthias J. Sax is a software engineer at Confluent working on ksqlDB. He mainly contributes to Kafka Streams, Apache Kafka's stream processing library, which serves as ksqlDB's execution engine. Furthermore, he helps evolve ksqlDB's "streaming SQL" language. In the past, Matthias also contributed to Apache Flink and Apache Storm and he is an Apache committer and PMC member. Matthias holds a Ph.D. from Humboldt University of Berlin, where he studied distributed data stream processing systems.
https://db.cs.cmu.edu/events/quarantine-db-talk-2020-confluent-ksqldb-a-stream-relational-database-system/
Top 5 Event Streaming Use Cases for 2021 with Apache KafkaKai Wähner
Apache Kafka and Event Streaming are two of the most relevant buzzwords in tech these days. Ever wonder what the predicted TOP 5 Event Streaming Architectures and Use Cases for 2021 are? Check out the following presentation. Learn about edge deployments, hybrid and multi-cloud architectures, service mesh-based microservices, streaming machine learning, and cybersecurity.
On-demand video recording: https://videos.confluent.io/watch/XAjxV3j8hzwCcEKoZVErUJ
Best Practices for Streaming IoT Data with MQTT and Apache KafkaKai Wähner
Organizations today are looking to stream IoT data to Apache Kafka. However, connecting tens of thousands or even millions of devices over unreliable networks can create some architecture challenges. In this session, we will identify and demo some best practices for implementing a large scale IoT system that can stream MQTT messages to Apache Kafka.
We use HiveMQ as open source MQTT broker to ingest data from IoT devices, ingest the data in real time into an Apache Kafka cluster for preprocessing (using Kafka Streams / KSQL), and model training + inference (using TensorFlow 2.0 and its TensorFlow I/O Kafka plugin).
We leverage additional enterprise components from HiveMQ and Confluent to allow easy operations, scalability and monitoring.
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Flink Forward
Flink Forward San Francisco 2022.
Flink consumers read from Kafka as a scalable, high throughput, and low latency data source. However, there are challenges in scaling out data streams where migration and multiple Kafka clusters are required. Thus, we introduced a new Kafka source to read sharded data across multiple Kafka clusters in a way that conforms well with elastic, dynamic, and reliable infrastructure. In this presentation, we will present the source design and how the solution increases application availability while reducing maintenance toil. Furthermore, we will describe how we extended the existing KafkaSource to provide mechanisms to read logical streams located on multiple clusters, to dynamically adapt to infrastructure changes, and to perform transparent cluster migrations and failover.
by
Mason Chen
Confluent Partner Tech Talk with Synthesisconfluent
A discussion on the arduous planning process, and deep dive into the design/architectural decisions.
Learn more about the networking, RBAC strategies, the automation, and the deployment plan.
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022HostedbyConfluent
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
Modern streaming use cases are generating massive amounts of data - much of it needs to be organized and queried over time. The sheer amount and complexity of this problem presents new challenges for data engineers and developers alike.
To solve this problem Apache Kafka and MongoDB Time Series collections are a powerful combination. In this talk, Kenny Gorman and Elena Cuevas will present how Apache Kafka on Confluent Cloud can stream massive amounts of data to Time Series Collections via the MongoDB Connector for Apache Kafka. Elena and Kenny will discuss the required configuration details and critical components of Confluent Cloud and MongoDB Atlas as well as some tips, tricks and best practices.
You will leave armed with the knowledge of how Confluent Cloud, Apache Kafka, MongoDB Atlas, and Time Series collections fit into your event-driven architecture.
La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.
Bridge to Cloud: Using Apache Kafka to Migrate to AWSconfluent
Watch this talk here: https://www.confluent.io/online-talks/bridge-to-cloud-apache-kafka-migrate-aws
Speakers: Priya Shivakumar, Director of Product, Confluent + Konstantine Karantasis, Software Engineer, Confluent + Rohit Pujari, Partner Solutions Architect, AWS
Most companies start their cloud journey with a new use case, or a new application. Sometimes these applications can run independently in the cloud, but often times they need data from the on premises datacenter. Existing applications will slowly migrate, but will need a strategy and the technology to enable a multi-year migration.
In this session, we will share how companies around the world are using Confluent Cloud, a fully managed Apache Kafka service, to migrate to AWS. By implementing a central-pipeline architecture using Apache Kafka to sync on-prem and cloud deployments, companies can accelerate migration times and reduce costs.
In this online talk we will cover:
•How to take the first step in migrating to AWS
•How to reliably sync your on premises applications using a persistent bridge to cloud
•Learn how Confluent Cloud can make this daunting task simple, reliable and performant
•See a demo of the hybrid-cloud and multi-region deployment of Apache Kafka
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersenconfluent
Best Practices for building Hybrid-Cloud Architectures - Hans Jespersen
Afternoon opening presentation during Confluent’s streaming event in Paris, presented by Hans Jespersen, VP WW Systems Engineering at Confluent.
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...HostedbyConfluent
Apache Kafka users who want to leverage Google Cloud Platform's (GCPs) data analytics platform and open source hosting capabilities can bridge their existing Kafka infrastructure on-premise or in other clouds to GCP using Confluent's replicator tool and managed Kafka service on GCP. Using actual customer examples and a reference architecture, we'll showcase how existing Kafka users can stream data to GCP and use it in popular tools like Apache Beam on Dataflow, BigQuery, Google Cloud Storage (GCS), Spark on Dataproc, and Tensorflow for data warehousing, data processing, data storage, and advanced analytics using AI and ML.
App modernization on AWS with Apache Kafka and Confluent CloudKai Wähner
Presentation from AWS ReInvent 2020.
Learn how you can accelerate application modernization and benefit from the open-source Apache Kafka ecosystem by connecting your legacy, on-premises systems to the cloud. In this session, hear real customer stories about timely insights gained from event-driven applications built on an event streaming platform from Confluent Cloud running on AWS, which stores and processes historical data and real-time data streams. Confluent makes Apache Kafka enterprise-ready using infinite Kafka storage with Amazon S3 and multiple private networking options including AWS PrivateLink, along with self-managed encryption keys for storage volume encryption with AWS Key Management Service (AWS KMS).
The Top 5 Event Streaming Use Cases & Architectures in 2021confluent
Learn how companies will leverage event streaming, Apache Kafka, and Confluent to meet the demand of a real-time market, rising regulations, and customer expectations, and much more in 2021
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...HostedbyConfluent
Event-driven application architectures are becoming increasingly common as a large number of users demand more interactive, real-time, and intelligent responses. Yet it can be challenging to decide how to capture and perform real-time data analysis and deliver differentiating experiences. Join experts from Confluent and AWS to learn how to build Apache Kafka®-based streaming applications backed by machine learning models. Adopting the recommendations will help you establish repeatable patterns for high performing event-based apps.
Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...confluent
Watch this talk here: https://www.confluent.io/online-talks/using-apache-kafka-to-optimize-real-time-analytics-financial-services-iot-applications
When it comes to the fast-paced nature of capital markets and IoT, the ability to analyze data in real time is critical to gaining an edge. It’s not just about the quantity of data you can analyze at once, it’s about the speed, scale, and quality of the data you have at your fingertips.
Modern streaming data technologies like Apache Kafka and the broader Confluent platform can help detect opportunities and threats in real time. They can improve profitability, yield, and performance. Combining Kafka with Panopticon visual analytics provides a powerful foundation for optimizing your operations.
Use cases in capital markets include transaction cost analysis (TCA), risk monitoring, surveillance of trading and trader activity, compliance, and optimizing profitability of electronic trading operations. Use cases in IoT include monitoring manufacturing processes, logistics, and connected vehicle telemetry and geospatial data.
This online talk will include in depth practical demonstrations of how Confluent and Panopticon together support several key applications. You will learn:
-Why Apache Kafka is widely used to improve performance of complex operational systems
-How Confluent and Panopticon open new opportunities to analyze operational data in real time
-How to quickly identify and react immediately to fast-emerging trends, clusters, and anomalies
-How to scale data ingestion and data processing
-Build new analytics dashboards in minutes
Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Curren...HostedbyConfluent
Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Current 2022
We see data and the way organizations using it to create unique customer experiences is being completely reimagined. Every time a customer clicks, types or swipes data is in constant motion spanning systems, environments and applications. This in turn requires business’ manage complex integrations, synchronizations and processing of data spread across cloud and on-prem environments.
To accelerate time to value, data needs to be seamlessly ingested, integrated and/or replicated to a cloud environment to take advantage of its analytical, BI and AI use cases. Google Cloud delivers a simplified approach for all your Data Movement needs through a highly differentiated product portfolio.
In this session, learn how organizations can unlock data value using best-in-class, cloud native products on Google Cloud and its partners such as Confluent.
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesKai Wähner
This session introduces Apache Kafka, an event-driven open source streaming platform. Apache Kafka goes far beyond scalable, high volume messaging. In addition, you can leverage Kafka Connect for integration and the Kafka Streams API for building lightweight stream processing microservices in autonomous teams. The Confluent Platform adds further components such as a Schema Registry, REST Proxy, KSQL, Clients for different programming languages and Connectors for different technologies.
The session discusses how tech giants like LinkedIn, Ebay or Airbnb leverage Apache Kafka as event streaming platform to solve various different business problems and how to create a scalable, flexible microservice architecture. A live demo shows how you can easily process and analyze streams of events using Apache Kafka and KSQL.
Budapest Data/ML - Building Modern Data Streaming Apps with NiFi, Flink and K...Timothy Spann
Budapest Data/ML - Building Modern Data Streaming Apps with NiFi, Flink and Kafka
Apache NiFi, Apache Flink, Apache Kafka
Timothy Spann
Principal Developer Advocate
Cloudera
Data in Motion
https://budapestdata.hu/2023/en/speakers/timothy-spann/
Timothy Spann
Principal Developer Advocate
Cloudera (US)
LinkedIn · GitHub · datainmotion.dev
June 8 · Online · English talk
Building Modern Data Streaming Apps with NiFi, Flink and Kafka
In my session, I will show you some best practices I have discovered over the last 7 years in building data streaming applications including IoT, CDC, Logs, and more.
In my modern approach, we utilize several open-source frameworks to maximize the best features of all. We often start with Apache NiFi as the orchestrator of streams flowing into Apache Kafka. From there we build streaming ETL with Apache Flink SQL. We will stream data into Apache Iceberg.
We use the best streaming tools for the current applications with FLaNK. flankstack.dev
BIO
Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming.
Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science.
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it YourselfDATAVERSITY
With your most talented teams bogged down managing a massive Kafka deployment, it can be challenging to move the dial on projects that drive real value for your business. For example, launching your next major feature, fueling more best-in-breed services like AI/ML on your cloud provider platform, or developing your first use cases for real-time data movement across clouds. By shifting to a fully managed, cloud-native service for Kafka you can unlock your teams to work on the projects that make the best use of your data in motion.
In this webinar you will learn about:
• The increasing value of data in motion to your business
• Challenges and costs of self-managing a large-scale Kafka deployment
• Benefits of managed cloud services for non-core activities like data storage, data warehousing, and messaging
• Optimizing time usage for value-generating activity like new product launches
• Potential cost savings for your business with a cloud-native service for Kafka
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent
In our exclusive webinar, you'll learn why event-driven architecture is the key to unlocking cost efficiency, operational effectiveness, and profitability. Gain insights on how this approach differs from API-driven methods and why it's essential for your organization's success.
Unlocking the Power of IoT: A comprehensive approach to real-time insightsconfluent
In today's data-driven world, the Internet of Things (IoT) is revolutionizing industries and unlocking new possibilities. Join Data Reply, Confluent, and Imply as we unveil a comprehensive solution for IoT that harnesses the power of real-time insights.
Workshop híbrido: Stream Processing con Flinkconfluent
El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real.
Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real.
En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...confluent
Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace.
In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms.
You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes.
Don't miss out on this opportunity to learn from industry experts and take your business to the next level.
Eventos y Microservicios - Santander TechTalkconfluent
Durante esta sesión examinaremos cómo el mundo de los eventos y los microservicios se complementan y mejoran explorando cómo los patrones basados en eventos nos permiten descomponer monolitos de manera escalable, resiliente y desacoplada.
Purpose of the session is to have a dive into Apache, Kafka, Data Streaming and Kafka in the cloud
- Dive into Apache Kafka
- Data Streaming
- Kafka in the cloud
Build real-time streaming data pipelines to AWS with Confluentconfluent
Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.
Q&A with Confluent Professional Services: Confluent Service Meshconfluent
No matter whether you are migrating your Kafka cluster to Confluent Cloud, running a cloud-hybrid environment or are in a different situation where data protection and encryption of sensitive information is required, Confluent Service Mesh allows you to transparently encrypt your data without the need to make code changes to you existing applications.
Citi Tech Talk: Event Driven Kafka Microservicesconfluent
Microservices have become a dominant architectural paradigm for building systems in the enterprise, but they are not without their tradeoffs. Learn how to build event-driven microservices with Apache Kafka
Confluent & GSI Webinars series - Session 3confluent
An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities.
It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks.
This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.
Transforming applications built with traditional messaging solutions such as TIBCO, MQ and Solace to be scalable, reliable and ready for the move to cloud
How can applications built with traditional messaging technologies like TIBCO, Solace and IBM MQ be modernised and be made cloud ready? What are the advantages to Event Streaming approaches to pub/sub vs traditional message queues? What are the strengeths and weaknesses of both approaches, and what use cases and requirements are actually a better fit for messaging than Kafka?
This session will show why the old paradigm does not work and that a new approach to the data strategy needs to be taken. It aims to show how a Data Streaming Platform is integral to the evolution of a company’s data strategy and how Confluent is not just an integration layer but the central nervous system for an organisation
Citi Tech Talk: Monitoring and Performanceconfluent
The objective of the engagement is for Citi to have an understanding and path forward to monitor their Confluent Platform and
- Platform Monitoring
- Maintenance and Upgrade
Strategies for Successful Data Migration Tools.pptxvarshanayak241
Data migration is a complex but essential task for organizations aiming to modernize their IT infrastructure and leverage new technologies. By understanding common challenges and implementing these strategies, businesses can achieve a successful migration with minimal disruption. Data Migration Tool like Ask On Data play a pivotal role in this journey, offering features that streamline the process, ensure data integrity, and maintain security. With the right approach and tools, organizations can turn the challenge of data migration into an opportunity for growth and innovation.
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
Cyaniclab : Software Development Agency Portfolio.pdfCyanic lab
CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...informapgpstrackings
Keep tabs on your field staff effortlessly with Informap Technology Centre LLC. Real-time tracking, task assignment, and smart features for efficient management. Request a live demo today!
For more details, visit us : https://informapuae.com/field-staff-tracking/
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?XfilesPro
Worried about document security while sharing them in Salesforce? Fret no more! Here are the top-notch security standards XfilesPro upholds to ensure strong security for your Salesforce documents while sharing with internal or external people.
To learn more, read the blog: https://www.xfilespro.com/how-does-xfilespro-make-document-sharing-secure-and-seamless-in-salesforce/
Into the Box Keynote Day 2: Unveiling amazing updates and announcements for modern CFML developers! Get ready for exciting releases and updates on Ortus tools and products. Stay tuned for cutting-edge innovations designed to boost your productivity.
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Hivelance Technology
Cryptocurrency trading bots are computer programs designed to automate buying, selling, and managing cryptocurrency transactions. These bots utilize advanced algorithms and machine learning techniques to analyze market data, identify trading opportunities, and execute trades on behalf of their users. By automating the decision-making process, crypto trading bots can react to market changes faster than human traders
Hivelance, a leading provider of cryptocurrency trading bot development services, stands out as the premier choice for crypto traders and developers. Hivelance boasts a team of seasoned cryptocurrency experts and software engineers who deeply understand the crypto market and the latest trends in automated trading, Hivelance leverages the latest technologies and tools in the industry, including advanced AI and machine learning algorithms, to create highly efficient and adaptable crypto trading bots
Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have.
For more Tendenci AMS events, check out www.tendenci.com/events
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
Check out the webinar slides to learn more about how XfilesPro transforms Salesforce document management by leveraging its world-class applications. For more details, please connect with sales@xfilespro.com
If you want to watch the on-demand webinar, please click here: https://www.xfilespro.com/webinars/salesforce-document-management-2-0-smarter-faster-better/
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Your Digital Assistant.
Making complex approach simple. Straightforward process saves time. No more waiting to connect with people that matter to you. Safety first is not a cliché - Securely protect information in cloud storage to prevent any third party from accessing data.
Would you rather make your visitors feel burdened by making them wait? Or choose VizMan for a stress-free experience? VizMan is an automated visitor management system that works for any industries not limited to factories, societies, government institutes, and warehouses. A new age contactless way of logging information of visitors, employees, packages, and vehicles. VizMan is a digital logbook so it deters unnecessary use of paper or space since there is no requirement of bundles of registers that is left to collect dust in a corner of a room. Visitor’s essential details, helps in scheduling meetings for visitors and employees, and assists in supervising the attendance of the employees. With VizMan, visitors don’t need to wait for hours in long queues. VizMan handles visitors with the value they deserve because we know time is important to you.
Feasible Features
One Subscription, Four Modules – Admin, Employee, Receptionist, and Gatekeeper ensures confidentiality and prevents data from being manipulated
User Friendly – can be easily used on Android, iOS, and Web Interface
Multiple Accessibility – Log in through any device from any place at any time
One app for all industries – a Visitor Management System that works for any organisation.
Stress-free Sign-up
Visitor is registered and checked-in by the Receptionist
Host gets a notification, where they opt to Approve the meeting
Host notifies the Receptionist of the end of the meeting
Visitor is checked-out by the Receptionist
Host enters notes and remarks of the meeting
Customizable Components
Scheduling Meetings – Host can invite visitors for meetings and also approve, reject and reschedule meetings
Single/Bulk invites – Invitations can be sent individually to a visitor or collectively to many visitors
VIP Visitors – Additional security of data for VIP visitors to avoid misuse of information
Courier Management – Keeps a check on deliveries like commodities being delivered in and out of establishments
Alerts & Notifications – Get notified on SMS, email, and application
Parking Management – Manage availability of parking space
Individual log-in – Every user has their own log-in id
Visitor/Meeting Analytics – Evaluate notes and remarks of the meeting stored in the system
Visitor Management System is a secure and user friendly database manager that records, filters, tracks the visitors to your organization.
"Secure Your Premises with VizMan (VMS) – Get It Now"
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
4. Evaluez votre niveau de maturité dans le
streaming de données
● Scannez le QR code
● Répondez aux questions
● Découvrez votre niveau de maturité
Data in Motion
5. Agenda
Plénière
Horaire SESSION
09:30 Keynote: Reinventing Kafka in the Data Streaming Era
10:05 Adéo : Construire une plateforme de données sur-mesure
10:40 CA GIP - CA PS : Publication métier des évènements du système d’autorisation émetteur
11:15 Pause café - Networking
11:45 Lactalis : le bilan
12:20 L’Oréal : L’Oréal Beauty Tech empowered by event-driven architecture
12:55 Cocktail déjeunatoire - Networking
14:00 CDC Informatique : Scaling with Kafka
14:30 Keynote : Stream processing with Apache Flink
15:00 Europcar : De Kafka open-source à une stratégie multi-cloud avec Confluent Cloud
15:30 Everysens : How Everysens made its product pivot a success with confluent cloud
16:00 AWS : Building Modern Streaming Analytics with Confluent on AWS
6. Agenda
Breakout
Horaire SESSION - Auditorium
16:30
Confluent et Flink: le mariage parfait à l'ère des données en temps
réel
17:00
Comment gouverner une plateforme Confluent - un équilibre à
trouver entre anarchisme et autoritarisme
17:30 Cocktail - Networking- Clap de fin
H SESSION - Auditorium
Imply : Building an Event Analytics Pipeline with Confluent Cloud and
Imply Polaris
Tinybird : Speed Wins: From Kafka to APIs in Minutes
7. Keynote : Reinventing Kafka in the
Data Streaming Era
Dan Rosanova
Head of Product
Confluent Cloud Platform and Growth
8. Loyalty Rewards
Curbside Pickup
Trending Now
Popular on Netflix
Top Picks for Joshua
Created by the founders of
Confluent while at LinkedIn
Apache Kafka has ushered in the
data streaming era…
>70%
of the Fortune 500
>100,000+
Organizations
>41,000
Kafka Meetup
Attendees
>200
Global Meetup
Groups
>750
Kafka Improvement Proposals
(KIPs)
>12,000
Jiras for Apache Kafka
>32,000
Stack Overflow
Questions
Real-time Trades
Ride ETA
Personalized Recommendations
9. The need for a cloud-native, data streaming platform
Connecting all your apps, systems and data into a central nervous system
10. Self-managing Kafka comes with cost and complexity…
Infrastructure and
Operations
Development
Resources
Security &
Governance
Global
Availability
11. “Hosted Streaming Services” didn’t solve all our
problems…
How can I connect
to all of my source
and sink systems?
How do I govern
my data for quality
and compliance?
How do I deploy
across multi and
hybrid cloud
environments?
How can I control my
networking
costs?
How can I ensure
low-latency,
while maintaining a
resilient service
How can I meet each
use case with
stream
processing?
12. What is this costing your business?
Unpacking the direct and indirect costs of self-managing and hosted streaming services
FTE Costs
It’s hard because... Which results in...
Costly time & resources (~$3-5M/year) managing Kafka,
connectors, governance, security, etc.
Delayed Time-to-Value
Infra Spend
$$$$ on underutilized infra for storage, compute and
networking
Increased Total Cost
of Ownership
Business Risk
Potential downtime and security breaches means
diverting resources
Unplanned Downtime
and Breaches
It’s expensive because... Which results in...
13. The world is moving towards fully-managed services…
Data Warehousing Databases
Self-managed
hardware and
software
Fully managed
services
Snowflake
“By 2025, at least 75% of organizations will depend on managed
services.”
— Globe Newswire
Hosted cloud
services CloudSQL
Data Streaming
Hosted Streaming
Vendors
15. A Cloud-Native Kafka Service Can
Eliminate Operational and Infrastructure
Burden…
Compute and
Storage Decoupling
Networking and
Global Replication
Elastic and
Automated
Multi-tenancy
and Serverless
… But Putting Kafka in the Cloud Isn’t Just Putting Kafka in the Cloud
16. We Transformed Kafka for the Cloud, Ground Up!
Resilient
with automated
operations to ensure
high availability and
reliability
Performant
with networking
service decoupling
and replication
optimization
Elastic
to seamlessly
expand and shrink
based on customer
demands
KORA ENGINE
The Apache Kafka® Engine Built for the Cloud
Cost efficient
with multi-tenancy,
data tiering, cloud
optimizations and
hands-off operations
17. We Invested 5M Engineering Hours to Reachitect Every
Layer of Kafka and Built a Truly Cloud-Native Engine
NETWORK
COMPUTE
AZ AZ AZ
Cells
Cells
Cells
OBJECT
STORAGE
CUSTOMERS
Multi-Cloud Networking & Routing Tier
Metadata
Durability Audits
METRICS & OBSERVABILITY
CONNECT
PROCESSING
GOVERNANCE
Data Balancing
Health Checks
Real-
time
feedback
data
Other Confluent Cloud Services
GLOBAL CONTROL PLANE
18. 50
40
30
20
10
0
Hours required to scale 3 brokers to 4, replication factor of 3,
30-day retention, 100 MBps throughput, 10GBps network
30X
ELASTICITY
Scale to handle GBps+
workloads and peak
customer demands
30x faster without
operational burden
30X
Confluent
Cloud
OSS
Kafka
Hours
19. 10
8
6
4
2
0
Other
Kafka Service
Confluent
Cloud
Minimum downtime commitment
by Kafka service based on SLA
10X
RESILIENCY
Ensure high availability
and offload Kafka ops
with 99.99% uptime
SLA, multi-AZ clusters,
and no-touch Kafka
patches
10X
8.76 hrs
0.876 hrs
99.99%
99.9%
Minimum
downtime
commitment
(hrs/year)
20. Infinite Storage
AWS GA
Infinite Storage
GCP GA
Infinite Storage
Azure GA
Time
X
∞
Average Storage Used per Cluster
by Cloud Providers
AWS GCP AZURE
STORAGE
Never worry about
Kafka storage again
with Intelligent Tiered
Storage and Infinite
Retention
AVG Storage
per Cluster
21. $2.57M
Total savings
Operate 60%+ more efficiently with reduced
infrastructure costs, maintenance demands
and overhead, and downtime risk
257%
3-year ROI
Launch in months rather than years by
reducing the burden on your teams
with our fully managed cloud service
Our Customers Save on Costs and Increase Their ROI
Total Economic Impact of using Confluent • Forrester, March 2022
“Confluent Cloud made it possible for us to meet our tight launch deadline with limited resources.
With event streaming as a managed service, we had no costly hires to maintain our clusters and
no worries about 24x7 reliability.”
22. Cloud-native data streaming platform built by the founders of Apache Kafka®
KORA: THE APACHE KAFKA ENGINE, BUILT FOR THE CLOUD
STREAM
Fully managed
service, available
Everywhere
The 10x, Cloud-native
Kafka service powered
by Kora Engine
A Complete,
enterprise-grade
Data Streaming
Platform
CONNECT GOVERN PROCESS
Confluent is so much more than Apache Kafka
24. Tom
Architect Lead
Anne
Architect Lead
Legacy apps
Real-time
apps
Cloud-native
apps
Cloud-based
data systems
Both Tom and Anne are tasked with…
● Maintaining OSS Kafka across all distributed systems, apps,
etc.
● Ensuring the web application is performant and resilient
● Building the new digital experiences for mobile, tablets, and
etc.
Legacy data
systems
Mainframes
PIVOT
INC.
FOSTER
OPS
25. …This is the result!
Without a fully
managed Kafka
service, Tom is
struggling…
PIVOT
INC.
…His “vendor” doesn’t
help connect, process,
or govern data
Self-managing Kafka was costly
and complex…
26. Creates, maintains and scales Kafka clusters
Onboards teams to use Kafka in a secure way
Connect to source and sink systems, while maintaining
governance
1
2
3
Build projects and distribute time between new tasks and
Kafka management
4
In this example, you will see how Anne…
Anne is going to
try with
Confluent
Cloud!
FOSTER
OPS
37. Tom and Anne have
very different budgets
and delivery timelines
Tom has exponentially rising
TCO, and can’t deliver for 12
months!
Anne has reduced TCO of
by up to 60%, and can
deliver in 3 months!
*App development time for example purposes only, actual time varies based on use
case
Cost to operate
Kafka environment
Time to market
~6-9 months to build production grade Kafka platform
~3 months on app
development*
~3 months on app
development*
Start
in 1
wee
k
Go to market in ~12 months
⬇60%
Cloud
Infrastructure
Operational
(FTE)
Downtime
Impact
Support & other
3rd party spend
Total self-
managed
Confluent
Cloud
Go to market in ~3 months
38. Who would you rather
be?
Anne at Foster Ops
with Confluent Cloud
Fully managed, cloud-native data
streaming solution
Complete data streaming platform with
connectors, governance and security
Flexible deployments across clouds and
on-premises
Anne has reduced TCO by up to 60% while
delivering to market 3x faster, and is in line for
that promotion real soon!
Significant effort self-managing
and maintaining Kafka
Custom coded connectors, governance
and security
Manually replicate clusters across
environments
Tom at Pivot Inc. with
OSS Kafka
Tom has exponentially rising infra costs and
spends 80% of his time self-managing Kafka,
thus is constantly getting pestered by
leadership!
39. As a result, Tom
isn’t very popular
right now…
PIVOT
INC.
41. $2.57M
Total savings
Operate 60%+ more efficiently with reduced
infrastructure costs, maintenance demands
and overhead, and downtime risk
257%
3-year ROI
Launch in months rather than years by
reducing the burden on your teams
with our fully managed cloud service
Our Customers Save on Costs and Increase Their ROI
Total Economic Impact of using Confluent • Forrester, March 2022
“Confluent Cloud made it possible for us to meet our tight launch deadline with limited resources.
With event streaming as a managed service, we had no costly hires to maintain our clusters and
no worries about 24x7 reliability.”
43. Program Details/Benefits
- Grand prize of up to $500K
- 2 runner-up awards of up to $250K
- Oppty to pitch to Benchmark, Sequoia,
Index
Target Profile
- Founded within last 5 years
- <$10M in venture funding
- Must use Confluent in submission
9/12 to 12/31 → Application window open
1/22 → Top 10 Announced
2/15 → Top 3 announced
3/19 → Grand prize announced at KSL
Sign up now!
44. Scan to get started
Start your free trial of Confluent
Cloud & get $500 in credits
Get started with Confluent Cloud!
$400 to spend immediately, plus an
additional $100 credit voucher
Code: DIMT2023 confluent.io/get-started/
54. Product Manager
Data Exchange & Processing
Dad of a little boy
I have a passion for technology and how to make it
useful
Data, Apache Kafka, Api management, ESB, REST,
Java, GO ...
Mustapha Benosmane
55. Collaborateurs ADEO
Habitants
Professionnels de
l’amélioration de l’habitat
Ecosystèmes
Fournisseurs, Partenaires
Marchands
Construire, Rénover
Aménager, Décorer
Produire, Délivrer
Agir, Impacter
Maison, appartement
Quartier, ville
Environnement
Planète
Endroit sain, sécurisé
responsable, durable
économe et confortable
Vie
Bien-être
Accomplissement
59
61. Centralizing skills ensures strong governance
Centralizing skills can help mutualize costs.
Centralizing skills reduces training and support costs
Centralizing skills reduces iteration capacity.
Centralizing the platform disengages users.
Centralizing skills and platforms reduces autonomy
and innovation.
Lessons learned
62. How can we provide
a service that
enables autonomy
and innovation,
while maintaining a
high level of
governance?
63. 67
Data Streaming Platform
Topic As A
Service
Technology
Governance
Self-Service
1. Enable developers to search, find,
understand and use Topics.
2. Enable teams to subscribe and agree on a
defined Interface agreement.
3. Enable developers to create and manage
the life-cycle of Topics and Schemas
4. Within a defined framework.
Automatically enforced.
5. Provide visibility of links between
applications.
6. Enable the product teams to control costs.
64. 1. Kafka for its properties
2. A managed offering -> No added
value in operating a Kafka cluster
3. Performance and resilience
4. A high level of security
5. A controlled cost
Technologie
65. 69
Kafka as a service
Serverless
● Elastic scaling up &
down from 0 to GBps
● Auto capacity mgmt,
load balancing, and
upgrades
High Availability
● 99.99% SLA
● Multi-region / AZ availability
across cloud providers
● Patches deployed in
Confluent Cloud before
Apache Kafka
Infinite Storage
● Store data cost-effectively
at any scale without
growing compute
DevOps Automation
● API-driven and/or point-
and-click ops
● Service portability &
consistency across cloud
providers and on-prem
Network Flexibility
● Public, VPC, and Private Link
● Seamlessly link across clouds
and on-prem with Cluster
Linking
66. 1. Respect best practices.
2. Maintain visibility and control over
interdependencies.
3. Provide and enforce interface
contracts.
4. Resource segmentation
5. Control access and authorizations
Governance
79. Great
responsiveness from
the team in the Run
channel
Very fast
OnBoarding for
newcomers
Extremely high
user autonomy
Rich and clear
documentation
A pleasure to work
with DSP
Glad to have a knowledgeable
team at Adeo with this level of
maturity
83. Do you have any questions?
Mustapha.benosmane@adeo.com
THANKS!
84. Publication métier des événements
du système d’autorisation émetteur
Julien Legrand
Product owner data
Crédit Agricole Gip
Camille Facque
Chef de projet
Crédit Agricole Gip
85.
86.
87. « Construire une offre de service, c’est industrialiser le déploiement d’une
solution technique complexe en y ajoutant un ensemble d’outils et
d’expertises permettant de rendre autonome l’utilisateur final. »
90. Les activités et l’expertise paiement pour le compte du Crédit Agricole
Gestion des cartes bancaires,
de l’émission de la carte
jusqu’au paiement
Monétique porteurs
Encaissement des paiements
par carte ou par chèque en proximité
ou en VAD
Monétique commerçants
Paiements SEPA & internationaux
Echanges et Flux
Gestion et mise à disposition de billets
et de pièces sur les différents marchés
(particuliers, professionnels, entreprises).
Fiduciaire
Garantir aux clients la sécurité des
transactions et des systèmes
d’information notamment via la DATA
Science et des outils d’IA
Authentification, Sécurité
& DATA
Développement de nouveaux services
innovants par l’Open Banking et l’utilisation
de la DATA
Open banking
& Data
Les domaines d’activités
Pour le compte de Crédit Agricole S.A.
auprès des instances de place nationales,
européennes et internationales
Représentation
interbancaire
Gestion des échanges d’opérations
bancaires entre banques, entre clients sur
tous les marchés France et l’international
91. 10/10/2023
95
Chiffres en suivi cumulé
janvier à décembre 2022
Nos principaux chiffres clés
13,6 milliards
D’opérations paiement traitées
MONÉTIQUE FLUX
9,8 milliards
d’opérations carte (Groupe CA)
22,9 millions
de cartes dans le parc Crédit Agricole
(CR, LCL, CACF)
5,2 milliards
d’autorisations fournies (paiement, retrait)
1,3 milliard
d’opérations SCT (virements)
1,6 milliard
d’opérations SDD (prélèvements)
19 millions
de virements SWIFT (Groupe CA)
93. Affichage des opérations
d’autorisations (paiements &
retraits)
Affichage des
opérations temps réel
Mise à jour du solde
provisoire
Emissions de notifications INAPP
Alerting client
Besoins Clients d’opérations temps réel
Gestion du cycle de vie
des cartes bancaires
Prise en compte des évolutions
des statuts cartes
Refonte de la MAJ des soldes
provisoires
Simplification de la
restitution
Enrichissement des données
existantes externes et restitution
dans un message unique
Utilisation des données statiques &
supervision business
Sauvegarde &
supervision
94. Enjeux
Des refontes d’architectures techniques et
fonctionnelles
Choix de la solution technique
MQ Séries
KAFKA
API
Protocoles d’échanges techniques
Diversités du format fonctionnel des
messages
Structure fonctionnelle historiquement
complexe
Collecte de données externes
Restitution d’avis unique
Enrichissement des données
Le chef
Les équipes
solutions
Diversités des échanges
Utilisation statique des données
Monitoring métier
Utilisations des données
95. Architecture existante
SAE SPAA
Cluster
MQ
Demande
d’autorisation
Application 3
Application 4
Application 5
Application 6
API Format de données 1
MQ Format de données 2
API Spécificités techniques 1
Format de données 3
API Spécificités techniques 2
Format de données 4
Application 2
MQ Format de données 2
Application 1
Cluster
MQ
S.A.E - Serveur d’autorisation émetteur
S.P.A.A - Service de publication des avis d’autorisations
AVANT
Diffusion des avis d’autorisations en échanges
synchrones
96. Architecture KAFKA
SAE
SPAA
KAFKA
Demande
d’autorisation
Diffusion des avis d’autorisations en échanges
asynchrones
Application 1
Consumers
S.A.E - Serveur d’autorisation émetteur
S.P.A.A - Service de publication des avis d’autorisation
Application 2
Application 3
Application 4
Application 5
Application 6
Stream
Producteur
Format de données 1
Prométhéus ELK
Grafana
APRES
Schéma registry
97. Architecture KAFKA
SAE
SPAA
KAFKA
Demande
d’autorisation
Diffusion des avis d’autorisations en échanges
asynchrones
Application 1
Consumers
S.A.E - Serveur d’autorisation émetteur
S.P.A.A - Service de publication des avis d’autorisation
Application 2
Application 3
Application 4
Application 5
Application 6
Stream
Producteur
Format de données 1
Prométhéus ELK
Grafana
APRES
Schéma registry
98. Focus SPAA
Une application stateless
Evènement unique
SAE
DLT
LCL
CAPS
C.R
Validation
Autorisation
Autorisation
Notification
Carte
Contrat
Autorisation
Autorisation
Notification
Carte
Contrat
Autorisation
Autorisation
Notification
Carte
Contrat
Identification
clients
Split
Split
Split
99. Données fonctionnelles – Répartition par topics
Pics de volume ~550 TPS soit ~15M de transactions / jour
Cycle de vie carte
Cycle de vie contrat
Avis de paiement
Avis de retrait
Avis de redressement
3 %
96 %
1 %
Autorisation /
notifications
Opposition
Activation du sans contact
Ouverture de service VAD
Création de carte
Suppression de carte
Changement de plafonds carte
100. Architecture KAFKA
SAE
SPAA
KAFKA
Demande
d’autorisation
Passage d’une application stateless à statefull
Application 1
Consumers
S.A.E - Serveur d’autorisation émetteur
S.P.A.A - Service de publication des avis d’autorisation
Application 2
Application 3
Application 4
Application 5
Application 6
Stream
Producteur
Format de données 1
Prométhéus ELK
Grafana
AVANT
Schéma registry
101. Architecture KAFKA
SAE
SPAA
KAFKA
Demande
d’autorisation
Passage d’une application stateless à statefull
Application 1
Consumers
S.A.E - Serveur d’autorisation émetteur
S.P.A.A - Service de publication des avis d’autorisation
Application 2
Application 3
Application 4
Application 5
Application 6
Stream
Producteur
Format de données 1
Système
externe
Prométhéus ELK
Grafana
APRES
Schéma registry
102. Architecture KAFKA
SAE
SPAA
KAFKA
Demande
d’autorisation
Passage d’une application stateless à statefull
Application 1
Consumers
S.A.E - Serveur d’autorisation émetteur
S.P.A.A - Service de publication des avis d’autorisation
Application 2
Application 3
Application 4
Application 5
Application 6
Stream
Producteur
Format de données 1
Système
externe
Prométhéus ELK
Grafana
APRES
Schéma registry
103. Architecture KAFKA
Passage d’une application stateless à statefull
Messages CAPS
Accounts
Client HTTP
Producteur
Connector HTTP SINK
Questions / Réponse
Mise au format Success Error
Response
Messages CAPS
enrichis
Messages CAPS
Left Join
Merge
Join
Évènement unique
SAE
DLT
CAPS
Validation Identification
clients
Split
APRES
Système externe
104. Architecture KAFKA
Passage d’une application stateless à statefull
Autorisation
Input
Évènement unique
SAE
DLT
CAPS
Validation Identification
clients
Split
AVANT
105. Architecture KAFKA
SAE
SPAA
KAFKA
Demande
d’autorisation
Logstash & mongoDB
Application 1
Consumer
S.A.E - Serveur d’autorisation émetteur
S.P.A.A - Service de publication des avis d’autorisation
Application 2
Application 3
Application 4
Application 5
Application 6
Stream
Producteur
Format de données 1
Système
externe
Envoi et récupération des
données
Connector HTTPS SINK
Prométhéus ELK
Grafana
AVANT
Schéma registry
106. Architecture KAFKA
SAE
SPAA
KAFKA
Demande
d’autorisation
Logstash & mongoDB
Application 1
Consumer
Schéma registry
S.A.E - Serveur d’autorisation émetteur
S.P.A.A - Service de publication des avis d’autorisation
Application 2
Application 3
Application 4
Application 5
Application 6
Stream
Producteur
Format de données 1
Système
externe
Envoi et récupération des
données
Connector HTTPS SINK
Prométhéus ELK
Grafana
Connector MongoDB
Consumer Logstash
APRES
107. 10/10/2023
111
Offre Topic as a Service
Des fonctionnalités déjà disponibles
❖ Création d’un compte technique
associé au contexte applicatif
SPAA via un call HTTP KAPI.
❖ Export des données vers
mongoDB ou Elasticsearch ou
ingestion de données issues
d’api HTTP via le cluster de
worker Kafka Connect déjà
disponible.
❖ Sollicitation de
l’accompagnement de la squad
Streaming ou de l’expertise
Confluent à tout moment.
114. 118
Cédric BARBIN
Architecte SI, Lactalis Informatique
• 20+ années d'expérience
• Développeur, expert technique, architecte, manager, …
• Transformation digitale des entreprises
• Expériences en SSII, Cabinet de conseil, Client final
• Passionné par la technologie et l’innovation
• Entrepreneur dans l’âme
• Certifications Dev et Ops sur Kafka et MongoDB
115. 119
Le groupe Lactalis
Premier groupe Laitier au monde
270
Sites de production
dans 51 Pays
85 500
Collaborateurs
dans 84 pays
28
Milliards d’euros
de chiffre d’affaire
120. 124
• Une DSI groupe, Française, à Laval
• Des correspondants internationaux rattachés aux Pays
• Des projets d’envergure en France et à l’international
• Une forte croissance externe du groupe
• Une stratégie Cloud Privé (LACTIC)
Et des postes à pourvoir,
notamment sur Kafka !
Direction des Systèmes d’Informations
Internationale et basée en France, à Laval
80
Salles
serveurs
~200
Personnes en
France
~500
Personnes à
l’international
2 Po
De données
123. 127
Le projet (périmètre Kafka)
Modernisation de la traçabilité produit fini
5 WMS as
Data Source
(CDC & Connect)
450
Utilisateurs
300k
business
event/day
(output)
63 lieux de
conditionnement
23 entrepôts
Source : GS1
124. 128
• Fraicheur des données
• Aujourd’hui : plusieurs dizaines de minutes (mode batch)
• Cible : moins de 1 minute
• Capacité de corriger / rejouer
• Problématique de référentiels pas à jour
• Traçabilité technique des données
• Expliquer d’où vient la donnée,
• Comment elle a été calculée,
• Le cas échéant pourquoi elle a été rejouée
Les objectifs business
Amélioration du nouveau système de traçabilité
126. 130
• Une refonte des Batch BigData / Scala « as is »
• Un principe de « migration technique »
• Des règles métiers à priori simples
• Donc utilisation de ksqlDB
• Langage SQL connu des analystes
• Pas de micro-services à gérer
• Complément à ksqlDB/Connect : la boite à outils
• SMT / Plugins
• UDF
• CI/CD JulieOps
L’orientation projet initiale
KSQL/Connect + Boite à outils
129. 133
Talend CDC
ex GammaSoft
Kafka Connect
JDBC
ksqlDB
Jointures
Formatage
Règles de gestions
Kafka Connect
MongoDB
Micro-services
KafkaStream
130. 134
• Des données en BDD (pas de « push » métier)
• Une captation des changements : CDC ou Connect
• Refonte milieu de projet : émission d’événement par certaines sources
• L’insertion dans un TOPIC d’entrée dit « RAW »
• Un traitement optionnel de préfiltre (lié au modèle CDC)
• Des requêtes ksqlDB
• Données brutes (pointeur sur le TOPIC initial ou pré-filtré)
• Données préparées : formatage, conversion, clés externes, …
• Données consolidées : jointure et transcodifications
• Données exposées : règles de gestions
La « topologie » classique d’un de nos flux
26 flux très proches d’un point de vue structure
Plus des nouveaux flux 100% KafkaStream !
133. 137
1 déploiement comporte aujourd’hui :
• Des configurations CDC
• Des configurations JSON pour les Connect Source
• La création de TOPICS avec paramétrage « Stream ou Table »
• Une gestion des consumer-group
• Des inventaires de déploiement de « PréFiltre » Kafka Stream
• Des données de références de transcodification
• Les mapping KSQL (ensemble de requêtes cohérent) = 1 flux
• Des configuration JSON pour les Connect Sink
Un déploiement complexe
Ecart démarrage : pas simplement du SQL…
134. 138
Un de nos grosses problématiques :
• Un système toujours en mouvement : pas de début et de fin, pas de « OK/KO »
• Comment détecter des écarts business ? Comment les corriger ?
On a donc besoin de s’outiller pour, entre autres choses :
• Cartographier nos flux
• Déployer (et dé-comissionner) ces flux
• Lancer des rejeux métier sur ces flux
• Gérer nos tables de transco et le cache des UDF
• Superviser les traitements (compteurs / KPI / latence)
On a intégré la supervision à nos outils d’exploitation (EON),
de BI (Qlik) et dans notre outils de gestion des flux : LactaFlux !
Une exploitation complexe
Un « run » en production sous-estimé
139. 143
• Migration technique = pas si évident
• Complexité du « In Motion »
• Concepts temporels à intégrer
• Besoin de maîtrise de la donnée et des systèmes amonts
• Principe d’idempotence à intégrer au projet
• Besoin fort d’expertise dès le début
• Besoin d’experts (externes) mais d’une implication forte interne
• Besoin d’optimisation pour ne pas exploser les volumes/perfs
• Un outillage spécifique à concevoir et développer
• Une plateforme technique complexe (on premise) : go to cloud ?
Les enseignements de ce démarrage
Data in motion <> Data at rest !!
140. 144
• Fraîcheur & qualité des données
• Cible : 1 minute 🡺 moyenne à 1 seconde !
• Responsabilisation des systèmes sources (pivot / event)
• Capacité de corriger / rejouer
• Rejeu sur plusieurs dizaines de milliers de lignes simple
(quelques clics) exécuté en quelques secondes
• Traçabilité technique des données
• Rejeu via topic = traçabilité
• Principe des topics Kafka = pas d’update
Les objectifs business
Un succès, on time !
Objectifs atteints
voir même dépassés
141. 145
• Nouveaux flux business & Machine Learning
• Déploiement à l’international (US notamment)
• Intégration de la traçabilité amont (production)
• Migration / passerelle vers normalisation GS1
• Plateforme commune (GS1/Distributeur)
• Blockchain & SmartContract
Et demain ?
Augmentation périmètre et nouveaux use case
143. L’Oréal Beauty Tech empowered by
event-driven architecture
Julien Brun
Head of APIs & EDA
Center of Enablement
L’Oréal
Sindhu Prasanna
EDA Expert
L’Oréal
144. C1 - Internal
use
C1 – Usage interne
L’ORÉAL BEAUTY TECH
EMPOWERED BY EDA
DATA IN MOTION
19TH OCTOBER 2023
145. C1 - Internal
use
149
MODERN INTEGRATION TO SUPPORT BEAUTY TECH
API not enough to address all
integration pattern
Give to program, project,
platform the rights tools for theirs
use cases
Provide the freedom and
autonomy by providing a
frame, best practices and
support
API FIRST
if NOT
Only…..
….ED
A
146. C1 - Internal
use
150
COE API & EDA
COE API & EDA
PROVIDE THE BEST PRACTICES AND FRAMEWORK
Z
Sindhu
PRASANNA
EDA expert
BUSINESS
Business Enablement (support API Product
Owner, projects)
governance & processes
Backlog management aligned with
business priorities
TECHNICAL
API/EDA expert community
Continuous improvement of
Framework
API & EDA community & Technology
Expertise
TRANSVERSAL
Training and upskilling programs
Modern integration sustainability
Analytics and reporting
API expert
BUSINESS TECHNICAL
Abdeladim
ABDELLAH
GLOBAL ARCHITECTURE & DATA
150. C1 - Internal
use
154
PLANNING
JAN
2023 MILESTONES
FEB
MAR
APR
JUN
MAY
JUL
AUG
SEP
CONTRACT
CLUSTERS SET UP
AUTOMATION
MONITORING (ELK)
PREPRD & PROD READY
(INTERNAL PROJECTS)
PREPRD & PROD READY
(EXTERNAL PROJECTS)
PROOF OF VALUE
GOVERNANCE & BEST
PRACTICES ONBOARDING
KT FOR SUPPORT
TEAM
FIRST PROJECT LIVE
151. C1 - Internal
use
155
CHALLENGES
.
.
.
.
.
Network configuration between the clusters
OAuth2: Compliancy between Confluent and our IDP
Lack of maturity
Hybrid use cases on private clusters
KsqlDB roles restrictions
153. C1 - Internal
use
157
TOPIC AS A SERVICE
Automatization of access management
to confluent using ITSM tool
(ServiceNow).
Automatization of topic management
for:
to create a topic
to subscribe to a topic
to publish into a topic
157. C1 - Internal
use
161
USE CASE
Example 3PL
L’ORÉAL
SAPS4HANA
APIGEE /
CONFLUENT
GEODIS
DHL
Event for Inbound delivery
Inbound delivery confirmation
158. C1 - Internal
use
162
NEXT STEP
Shared Domain Data
Sets on GCP
Governed Business APIs
on APIGEE
Use Cases
DATA
Product SellOut O+O …
TO SUSTAINABLE DATA PRODUCTS
OWNERSHIP
Data mapped and under
business ownership
ACCESSIBILITY
Data accessibility
to all use case
STANDARDISATION
Shared data and common
catalogue
QUALITY
Single source of truth
SECURITY
Follow group security rules
Governed Business
Event on CONFLUENT
159. C1 - Internal
use
163
ORGANIZATION
Confluent Account Team
Philippe Amiel Account Executive
Identifies opportunities with new and existing customers
and builds them into long-term profitable relationships. philippe@confluent.io
Eric Carlier
Senior Solutions
Engineer
Key technical advisor to customers, undertaking
technical design and development of end-to-end
solution.
eric.carlier@confluent.io
Camille de Rosier
Customer Success
Manager
Ensures customers are successful in their deployments of
Confluent service throughout onboarding and beyond.
cderosier@confluent.io
Sylvain Le Gouellec
Customer Success
Technical Architect
Ensures customers realize the full value of the Confluent
service. Runs point with customer and liaises with internal
account team from day to day.
slegouellec@confluent.io
Daniel Petisme
Customer Success
Technical Architect dpetisme@confluent.io
Nils Bouchardon
Senior Solutions
Architect
Your senior technical lead who will guide you through
design principles, deployment strategies, best practices
and much more.
nbouchardon@confluent.io
162. Agenda
Plénière
Horaire SESSION
09:30 Keynote: Reinventing Kafka in the Data Streaming Era
10:05 Adéo : Construire une plateforme de données sur-mesure
10:40 CA GIP - CA PS : Publication métier des évènements du système d’autorisation émetteur
11:15 Pause café - Networking
11:45 Lactalis : le bilan
12:20 L’Oréal : L’Oréal Beauty Tech empowered by event-driven architecture
12:55 Cocktail déjeunatoire - Networking
14:00 CDC Informatique : Scaling with Kafka
14:30 Keynote : Stream processing with Apache Flink
15:00 Europcar : De Kafka open-source à une stratégie multi-cloud avec Confluent Cloud
15:30 Everysens : How Everysens made its product pivot a success with confluent cloud
16:00 AWS : Building Modern Streaming Analytics with Confluent on AWS
163. Agenda
Breakout
Horaire SESSION - Auditorium
16:30
Confluent et Flink: le mariage parfait à l'ère des données en temps
réel
17:00
Comment gouverner une plateforme Confluent - un équilibre à
trouver entre anarchisme et autoritarisme
17:30 Cocktail - Networking- Clap de fin
H SESSION - Auditorium
Imply : Building an Event Analytics Pipeline with Confluent Cloud and
Imply Polaris
Tinybird : Speed Wins: From Kafka to APIs in Minutes
164. Agenda
Plénière
Horaire SESSION
09:30 Keynote: Reinventing Kafka in the Data Streaming Era
10:05 Adéo : Construire une plateforme de données sur-mesure
10:40 CA GIP - CA PS : Publication métier des évènements du système d’autorisation émetteur
11:15 Pause café - Networking
11:45 Lactalis : le bilan
12:20 L’Oréal : L’Oréal Beauty Tech empowered by event-driven architecture
12:55 Cocktail déjeunatoire - Networking
14:00 CDC Informatique : Scaling with Kafka
14:30 Keynote : Stream processing with Apache Flink
15:00 Europcar : De Kafka open-source à une stratégie multi-cloud avec Confluent Cloud
15:30 Everysens : How Everysens made its product pivot a success with confluent cloud
16:00 AWS : Building Modern Streaming Analytics with Confluent on AWS
165. Agenda
Breakout
Horaire SESSION - Auditorium
16:30
Confluent et Flink: le mariage parfait à l'ère des données en temps
réel
17:00
Comment gouverner une plateforme Confluent - un équilibre à
trouver entre anarchisme et autoritarisme
17:30 Cocktail - Networking- Clap de fin
H SESSION - Auditorium
Imply : Building an Event Analytics Pipeline with Confluent Cloud and
Imply Polaris
Tinybird : Speed Wins: From Kafka to APIs in Minutes
167. CDC Informatique
La Caisse des Dépôts
Le groupe Caisse des Dépôts, alliance
unique d'acteurs économiques publics et
privés, s’engage, au cœur des territoires,
pour accélérer la transformation écologique
et pour contribuer à une vie meilleure pour
toutes et tous.
1 320Md€
Bilan agrégé 2022 *
4,2 Md€
Résultat net agrégé
*
* Chiffres agrégés : :Section générale comptes consolidés en
normes IFRS + Fonds d’Epargne en normes françaises
171
169. Kafka introduit pour l'ingestion de
données en temps réel dans Hive.
CDC Informatique
L’arrivée de Kafka : Un tournant en 2019
173
Module HDF déployé en mars 2019.
170. Nécessité de revoir l'ISP pour être conforme aux nouveaux usages
Offre de service du socle non critique
Multiplicité des outils pour la création de ressource Kafka 3 équipes
Peur et résistance au changement
Présence de silos prégnants
CDC Informatique
Nos Constats : 2021
174
171. CDC Informatique
Pourquoi avons-nous évolué ?
175
Conviction que
l'état actuel
n'était pas viable
Soutien d'un
responsable de
squad engagé
Nouveaux enjeux
et jalons métier
(SRE, Instant
payment)
Élément
incontournable dans
l'approche Cloud
Native de notre
schéma directeur
Décryptage de Kafka
pour le rendre lisible
auprès de la DG et avoir
des sponsors
172. CDC Informatique
Notre démarche
◆ Diagnostic 360°
◆ Infrastructures
◆ Sécurité
◆ DevSecOps
◆ Supervision
◆ Enjeux et jalons métier (instant payment)
◆ Usage existant
◆ Questionnement sur la distribution de Kafka
176
Réalisation d'une étude complète comprenant :
173. CDC Informatique 177
La nouvelle cible
◆ Mise en œuvre rapide des projets et diffusion de la connaissance.
◆ Formation des équipes.
◆ Sécurité renforcée de la plateforme.
◆ Plateforme DevSecOps maîtrisée de bout en bout.
◆ Haute disponibilité et compatibilité avec le PSI.
◆ Prendre en compte les irritants collectés lors du constat
174. CDC Informatique
Rappel des scénarios éditeurs
pour le socle Kafka
Évolution de la plateforme actuelle
Cloudera HDF vers CDP
Nouvelle plateforme Confluent
Nouvelle plateforme Apache Kafka basée
sur les fonctionnalités de la LAPOSTE BSCC
178
175. CDC Informatique
◆ Professional Service
◆ MultiRegion cluster (PSI)
◆ Délai de mise en œuvre faible
179
Scénario choisi par le codir et
recommandé par nos équipes
Scénario Confluent
◆ Haut niveau d’industrialisation
◆ Meilleure gestion du contenu
◆ Support éditeur expert Kafka
176. CDC Informatique
Bilan en chiffre après 1 an de production
◆ 69 applications en recette.
◆ 30 personnes formées (120 jours
de formation).
◆ 40 jours de PS.
◆ 4 montées de version sans
interruption de service.
◆ 2 ops, 2 experts techniques, 1
archi 180
◆ Mars 2022:
Démarrage du projet.
◆ Octobre 2022 :
Ouverture de la production.
◆ 5 clusters : 3 clusters (8 brokers)
sur 2,5 Data Centers.
◆ 35 applications en production.
177. CDC Informatique
Bilan après 1 an de production
181
◆ La plateforme est devenue un exemple à suivre dans l’entreprise
◆ Retour très positif sur l'autonomie des équipes
◆ Documentation en ligne autoportante
◆ Pattern sur étagère transactional outbox
◆ Dashboard de métriques public de la plateforme
◆ Elastic qui offre l'accès à tous les logs des connecteurs par cluster
178. CDC Informatique
Les clés de notre réussite
Équipe
pluridisciplinaire
Intégration
précoce de
toutes les
équipes
Budget projet
complet
Beaucoup de PS
au démarrage
Transformation
organisationnelle
182
179. CDC Informatique
Prochaines étapes et défis à relever
183
◆ Industrialisation des secrets, des cas d’usage, et des tableaux de bord.
◆ Travaux de rework et convergence sur l’IAC (API, Kubernetes, S3).
◆ Vérification automatisée des normes d’entreprise.
◆ Interaction utilisateur via IHM avec la plateforme.
182. 01
02
03
Understanding the importance of stream processing
Why Apache Flink is becoming the de facto standard
Enhancing Apache Flink as a cloud-native service
Agenda
183. Keynote: Stream processing with
Apache Flink®
Konstantin Knauf
Director Solutions Engineering
Confluent
185. Stream processing is a critical part of data streaming
Enable frictionless
access to up-to-date
trustworthy data
products
Share
Reimagine data
streaming everywhere,
on-prem and in every
major public cloud
Stream
Make data in motion
self-service, secure,
compliant and
trustworthy
Govern
Drive greater
data reuse with
always-on stream
processing
Process
Make it easy to on-
ramp and off-ramp
data from existing
systems and apps
Connect
186. Stream processing acts as the compute layer to Kafka,
powering real-time applications & pipelines
DATA IN MOTION
Streaming
Applications
Apache
Flink
Apache
Kafka
DATA AT REST
Application
Layer
Processing
Layer
Storage
Layer
Traditional
Databases
File
Systems
Web
Applications
188. Processing data at
ingest improves
latency, data
portability, and cost
effectiveness
Custom
apps
3rd party
apps
Databases
Databas
e
Data
Warehouse
SaaS
app
Querie
s
Analytics
Interactions
Kafka
Storage
Flink
Compute
Stream Processing
Process your data once, process your data right
Maximized data reusability &
consistency
Improved cost-efficiency from
cleaning & enriching data once
Real-time apps & data systems
reflect current state
189. Stream processing enables users to filter, join, and enrich
streams on-the-fly to drive greater data reuse
Heatmap service
Payment service
Supply chain systems
Watch lists
Profile mgmt
Incident mgmt
Customer
profile data
ITSM systems
Central log systems
Fraud & SIEM systems
Alerting systems
AI/ML engines
Visualization apps
Threat vector
Transactions
Payments
Mainframe data
Inventory
Weather
Telemetry
IoT data
Notification engine
Payroll systems
CRM systems
Mobile application
Personalization
Web application
Clickstreams
Customer loyalty
Change logs
Customer data
Recommendation
engine
191. Flink growth has
mirrored the growth
of Kafka, the de facto
standard for
streaming data
>75% of the Fortune 500 estimated
to be using Kafka
>100,000+ orgs using Kafka
>41,000 Kafka meetup attendees
>750 Kafka Improvement Proposals
>12,000 Jiras for Apache Kafka
0
50,000
100,000
150,000
2020 2021 2022
2016 2017 2018
Flink
Kafka
Two Apache Projects, Born a
Few Years Apart
Monthly Unique Users
193. Digital natives leverage Flink to disrupt markets and gain
competitive advantage
UBER: Real-time Pricing NETFLIX: Personalized Recs STRIPE: Real-time Fraud Detection
194. Developers choose Flink because of its performance and
rich feature set
Scalability and
Performance
Fault
Tolerance
Flink is a top 5 Apache project and boasts a robust developer community
Unified
Processing
Flink is capable of
supporting stream
processing workloads
at tremendous scale
Language
Flexibility
Flink's fault tolerance
mechanisms ensure it
can handle failures
effectively and provide
high availability
Flink supports Java,
Python, & SQL with
150+ built-in functions,
enabling devs to work
in their language of
choice
Flink supports stream
processing, batch
processing, and ad-hoc
analytics through one
technology
195. Developers choose Flink because of its performance and
rich feature set
Scalability and
Performance
Fault
Tolerance
Flink is a top 5 Apache project and boasts a robust developer community
Unified
Processing
Flink is capable of
supporting stream
processing workloads
at tremendous scale
Language
Flexibility
Flink's fault tolerance
mechanisms ensure it
can handle failures
effectively and provide
high availability
Flink supports Java,
Python, & SQL with
150+ built-in functions,
enabling devs to work
in their language of
choice
Flink supports stream
processing, batch
processing, and ad-hoc
analytics through one
technology
196. Flink’s powerful runtime offers limitless scalability
Job Manager
Client
. . . . . .
Task Slot
. . . . . .
Task Slot
. . . . . .
Task Slot
. . . . . .
Task Slot
Data Streams
Deploy, Stop, Cancel
Tasks
Trigger Checkpoints
Submit
Job
Result
s
Applications are parallelized into possibly
thousands of tasks that are distributed and
concurrently executed in a cluster
197. Leverage in-memory performance
. . .
Durable
Storage
Logic State Logic State Logic State
Input
Tasks
Output
In-Memory or
On-Disk State
Local State
Access
Periodic, Asynchronous,
Incremental Snapshots
Stateful Flink applications are optimized for fast access to local state by maintaining
task state in memory or on-disk data structures, resulting in low latency processing.
198. Developers choose Flink because of its performance and
rich feature set
Scalability and
Performance
Fault
Tolerance
Flink is a top 5 Apache project and boasts a robust developer community
Unified
Processing
Flink is capable of
supporting stream
processing workloads
at tremendous scale
Language
Flexibility
Flink's fault tolerance
mechanisms ensure it
can handle failures
effectively and provide
high availability
Flink supports Java,
Python, & SQL with
150+ built-in functions,
enabling devs to work
in their language of
choice
Flink supports stream
processing, batch
processing, and ad-hoc
analytics through one
technology
199. Flink checkpoints and savepoints enable fault tolerance and
stateful processing
CHECKPOINTS SAVEPOINTS
Automatic snapshot
created by Flink
periodically
● Used to recover from failures
● Optimized for quick recovery
● Automatically created and managed
by Flink
User-triggered snapshot at
a specific point in time
● Enables manual operational tasks,
such as upgrades
● Optimized for operational flexibility
● Created and managed by the user
200. Flink recovers from failures in a timely and efficient manner
Job Manager
Client
. . . . . .
Task Slot
. . . . . .
Task Slot
. . . . . .
Task Slot
. . . . . .
Task Slot
Data Streams
Deploy, Stop, Cancel
Tasks
Trigger Checkpoints
Submit
Job
Result
s
If a task managers fails, the job manager will
detect the failure and arrange for the job to be
restarted from the most recent state snapshot
X
201. Developers choose Flink because of its performance and
rich feature set
Scalability and
Performance
Fault
Tolerance
Flink is a top 5 Apache project and boasts a robust developer community
Unified
Processing
Flink is capable of
supporting stream
processing workloads
at tremendous scale
Language
Flexibility
Flink's fault tolerance
mechanisms ensure it
can handle failures
effectively and provide
high availability
Flink supports Java,
Python, & SQL with
150+ built-in functions,
enabling devs to work
in their language of
choice
Flink supports stream
processing, batch
processing, and ad-hoc
analytics through one
technology
202. Flink offers layered APIs at different levels of of abstraction
to handle both common and specialized use cases
Flink SQL
Table API
DataStream API
ProcessFunction Apache Flink Runtime
Low-level Stream Operator API
DataStream
API
ProcessFunction
Table / SQL API
Planner/Optimize
r
Flink SQL
High-level, declarative API that allows you to write SQL
queries to process data streams and batch data as
dynamic tables
Table API
Programmatic equivalent of Flink SQL, allowing you to
define your business logic in either Java or Python, or
combine it with SQL
DataStream API
Low-level, expressive API that exposes the building
blocks for stream processing, giving you direct access to
things like state and timers
ProcessFunction
The most low-level API, allowing for fine-grained
processing of individual elements for complex event-
driven processing logic and state management
203. Process real-time
data streams with
Flink SQL
Flink SQL is an ANSI-compliant SQL
engine that can define both simple
and complex queries, making it well-
suited for most stream processing
use cases, particularly building real-
time data products and pipelines.
GROUP BY color
events
results
COUNT
WHERE color <> orange
4
3
204. Developers choose Flink because of its performance and
rich feature set
Scalability and
Performance
Fault
Tolerance
Flink is a top 5 Apache project and boasts a robust developer community
Unified
Processing
Flink is capable of
supporting stream
processing workloads
at tremendous scale
Language
Flexibility
Flink's fault tolerance
mechanisms ensure it
can handle failures
effectively and provide
high availability
Flink supports Java,
Python, & SQL with
150+ built-in functions,
enabling devs to work
in their language of
choice
Flink supports stream
processing, batch
processing, and ad-hoc
analytics through one
technology
205. Flink supports unified stream and batch processing
● Entire pipeline must always be running ● Execution proceeds in stages, running as needed
● Input must be processed as it arrives ● Input may be pre-sorted by time and key
● Results are reported as they become ready ● Results are reported at the end of the job
● Failure recovery resumes from a recent snapshot ● Failure recovery does a reset and full restart
● Flink guarantees effectively exactly-once results
despite out-of-order data and restarts due to
failures, etc.
● Effectively exactly-once guarantees are more
straightforward
207. Operating Flink on your own (along with the Kafka storage
layer) is difficult
Deployment
Complexity
Setting up Flink requires a
deep understanding of
resource allocation and
management
Management &
Monitoring
Picking relevant metrics can
be overwhelming for a
DevOps team just starting
with stream processing
Limited
Ecosystem
Flink lacks pre-built
integrations with observability,
metadata management, data
governance, and security
tooling
Cost &
Risk
Self-supporting Flink
incurs significant costs &
resources in terms of infra
footprint and Dev & Ops
FTEs
208. Effortlessly filter, join, and enrich your
data streams with Flink, the de facto
standard for stream processing
Enable high-performance and efficient
stream processing at any scale,
without the complexities of
infrastructure management
Experience Kafka and Flink as a
unified platform, with fully integrated
monitoring, security, and governance
Confluent Cloud for
Apache Flink®
Simple, Serverless Stream Processing
Easily build high-quality,
reusable data streams with
the industry’s only cloud-
native, serverless Flink service
Available for preview in select regions – see the docs for regional availability
209. Effortlessly filter, join, and enrich your data streams with Apache Flink
Real-time processing
Power low-latency applications and pipelines that react
to real-time events and provide timely insights
Data reusability
Share consistent and reusable data streams widely with
downstream applications and systems
Data enrichment
Curate, filter, and augment data on-the-fly with
additional context to improve completeness, accuracy, &
compliance
Efficiency
Improve resource utilization and cost-effectiveness by
avoiding redundant processing across silos
“With Confluent’s fully managed Flink offering, we can access, aggregate, and enrich data from IoT sensors,
smart cameras, and Wi-Fi analytics, to swiftly take action on potential threats in real time, such as intrusion
detection. This enables us to process sensor data as soon as the events occur, allowing for faster detection and
response to security incidents without any added operational burden.”
210. Analyze real-time
data streams to
generate important
business insights
Get up-to-date results to power
dashboards or applications requiring
continuous updates using:
● Materialized views
● Temporal analytic functions
● Interactive queries
Account Balance
A $15
B $2
C $15
Account A,
+$10
Account B,
+$12
Account C, +$5
Account B, -
$10
Account C,
+$10
Account A, -$5
Account A,
+$10
Time
REAL-TIME ANALYTICS
211. Build streaming data
pipelines to inform
real-time decision
making
Create new enriched and curated
streams of higher value using:
● Data transformations
● Streaming joins, temporal joins,
lookup joins, and versioned joins
● Fan out queries, multi-cluster
queries
215
t1, 21.5 USD
t3, 55 EUR
t5, 35.3
EUR
t0, EUR:USD=1.00
t2, EUR:USD=1.05
t4: EUR:USD=1.10
t1, 21.5 USD
t3, 57.75 USD
t5, 38.83 USD
Currency rate
Orders
STREAMING DATA PIPELINES
212. Recognize patterns
and react to events in
a timely manner
Develop applications using fine-
grained control over how time
progresses and data is grouped
together using:
● Hopping, tumbling, session
windows
● OVER aggregations
● Pattern matching with
MATCH_RECOGNIZE
EVENT-DRIVEN APPLICATIONS
C
price>lag(price)
D
price<lag(price)
C
price>lag(price)
B
price<lag(price)
A
Double Bottom
Period & Volume
Price
213. Enrich real-time data streams with Generative AI directly
from Flink SQL
INSERT INTO enriched_reviews
SELECT id
,
review
,
invoke_openai(prom
pt,review) as score
FROM
product_reviews
;
K
N
Kate
4 hours ago
This was the worst decision ever.
Nikola
1 day ago
Not bad. Could have been
cheaper.
K
N
B
Kate
★★★★★ 4 hours ago
This was the worst decision ever.
Nikola
★★★★★ 1 day ago
Not bad. Could have been
cheaper.
Brian
★★★★★ 3 days ago
Amazing! Game Changer!
The Prompt
“Score the following text on a scale of
1 and 5 where 1 is negative and 5 is
positive returning only the number”
DATA STREAMING PLATFORM
B
Brian
3 days ago
Amazing! Game Changer!
COMING SOON
214. Fully managed
Easily develop Flink applications with a serverless, SaaS-
based experience instantly available & without ops
burden
Elastic scalability
Automatically scale up or down to meet the demands of
the most complex workloads without overprovisioning
Usage-based billing
Pay only for resources used instead of infrastructure
provisioned, with scale-to-zero pricing
Continuous, no touch updates
Build using an always up-to-date platform with
declarative, versionless APIs and interfaces
Throughput/Data Traffic Over Time
Capacity Demand
Enable high-performance and efficient stream processing at any scale
"Offloading that day-to-day burden of operations has been a huge help. A lot of overall operations-type work
gets offloaded when you move to Confluent Cloud… Where we’re saving time now is on the DevOps side of
maintenance of all those systems — patching underlying systems or upgrading(them) — those were big things
to be able to offload."
215. Go from zero to production in minutes versus months
Minutes
Weeks
Open Source
Apache Flink
In-house development and
maintenance without
support
Cloud-hosted
Flink services
Manual Day 2 operations
with basic tooling and/or
support
Apache Flink on
Confluent Cloud
Fully managed, elastic,
and automated product
capabilities with zero
overhead
Months
216. Throughput over Time Capacity
Demand
Maximize resource utilization & avoid over-provisioning infrastructure
Scale elastically to
meet changing
business needs
Automatically scale up or down to
meet the demands of the most
complex workloads
● Avoid underutilized infrastructure
resources
● Pay only for resources used, with
scale-to-zero pricing
217. Tap into a next-generation, serverless SQL experience …
SQL client in Confluent
Cloud CLI
Different teams with different skills and needs can access
stream processing using the interface of their choice
Rich SQL editing
user interface
218. "When used in combination, Apache Flink & Apache Kafka can enable data reusability and avoid redundant
downstream processing. The delivery of Flink & Kafka as fully managed services delivers stream processing
without the complexities of infrastructure management, enabling teams to focus on building real-time streaming
applications & pipelines that differentiate the business."
Enterprise-grade security
Secure stream processing with built-in identity and
access management, RBAC, and audit logs
Stream governance
Enforce data policies and avoid metadata duplication
leveraging native integration with Stream Governance
Monitoring
Ensure the health and uptime of your Flink queries in the
Confluent UI or via 3rd party monitoring services
Connectors
Ensure the health and uptime of your Flink queries in the
Confluent UI or via 3rd party monitoring services
Monitoring Connectors
Enterprise-grade
Security
Stream
Governance
Experience Kafka and Flink seamlessly integrated as a unified
platform
219. Provide platform-wide security with granular access to
critical resources
Flink
Admin
Flink
Developer
Flink
Developer
Flink SQL
queries
Flink Control
Plane requests
220. Automate metadata
synchronization for
effortless data
exploration
Integration with Schema Registry
enables Flink to easily access and
process data from multiple Kafka
clusters and Confluent environments
in a consistent and unified way:
● Kafka topics → Flink tables
● Confluent environments → catalogs
● Kafka clusters → databases …
…
…
221. Connect your entire business with just a few clicks
70+
fully
managed
connectors
Amazon S3
Amazon Redshift
Amazon DynamoDB
Google Cloud
Spanner
AWS Lambda
Amazon SQS
Amazon Kinesis
Azure Service Bus
Azure Event Hubs
Azure Synapse
Analytics
Azure Blob
Storage
Azure
Functions
Azure Data Lake
Google
BigTable
222. Scan to get started
Start your free trial of Confluent
Cloud & get $500 in credits
Get started with Confluent Cloud!
$400 to spend immediately, plus an
additional $100 credit voucher
Code: DIMT2023 confluent.io/get-started/
224. De Kafka Open Source à la mise en
place d’une stratégie multi-cloud
avec Confluent
Ahmed Tali
Group Head of Architecture & Foundations Engineering
Europcar Mobility Group
225. 229
229
Agenda
1. Europcar Mobility Group Global Context
1. Group Information System
1. Internal Kafka Usage
1. Event Driven Architecture Study
1. Why Confluent Cloud
1. Migration Plan
1. Project Status & Next Steps
226. 230
Europcar Mobility Group in a nutshell
230
Global Context
- Part of Green Mobility Holding led by VW
- Extensive network in more than 140 countries
- Almost 9000 employees
- 3 Billions revenue / 256 000 Vehicles
- 5 million customers worldwide
227. 231
231
Group Information System
EMOBG Information System in nutshell
Business Oriented
IS Components
Domain Driven Design
Brand Agnostic
Products based organization
Multi-cloud
Strategy
AWS First approach
Specialized business domains in
GCP
Composable
Architecture
Interoperability with 3rd party
solutions
API Products & Events as main
communication flows
Technology
Transformation
Monoliths to Microservice
Architecture
API First Approach
Event Driven Architecture
228. 232
Overview & Architecture Patterns
232
Event Driven Usage
Event Driven Patterns
- Publish-Subscribe
- Kafka Connectors
- Change Data Capture
Microservices Based Architecture
- Autonomous Microservices
(own storage)
- Microfrontend apps
Distributed and Open IS
- Multicloud / Multi region
- Full Integration with 3rd Party
and Partners Systems
229. 233
Former Situation
233
Group Event Driven Study
Main Issues
- Difficult to setup a stable and extensible platform
- Tricky to scale Kafka platforms causing performance issues
- Hard to achieve high availability
- Costly Integration :
- Several development workloads
- 3 Implementations to maintain
- Lack of visibility on group level
230. 234
Target Situation
234
Group Event Driven Study
Main Expectations
- Unify our event driven layer and setup a well
governed kafka based solution
- Adopt latest Kafka Market Standards
- Be focus on business flows instead of managing
Kafka platform
Studied Options
- Self Hosted Event Driven Solution
- Fully Managed Event Driven Solution
231. 235
Why fully managed model
235
Group Event Driven Study
1. Kafka Technology difficult to master by internal teams (based on years of
experience)
- We need permanent high level kafka expertise
1. Autonomous teams operating in a multicloud and distributed environments are not
adopting same industry standards
- We need central goverened Kafka solution with group policies
(security, monitoring…) and applied by everybody
1. Managing, Scaling & maintaining Kafka platform reduces teams autonomy and
impact focus on business aspects
- We need stable, performant and auto scaled solution with low
internal effort
232. 236
Why Confluent Cloud
236
Why Confluent Cloud
- We need high level of kafka expertise
- Confluent Cloud are original creators of Apache Kafka
- We need fully managed, stable and auto scaled solution
- Confluent Cloud provides Fully Managed and Hybrid services
- We need central governed Kafka solution where we can apply group
policies (security, monitoring…)
- Confluent Cloud brings features over Kafka such as monitoring,
security, connectors…
- We need cloud agnostic solution offering good level of our infrastructure
coverage
- Confluent Cloud covers all our cloud providers and aligns to our
multicloud strategy
233. 237
High level Architecture
237
EMOBG Confluent Cloud Integration
- Confluent Cloud cluster for each Cloud Provider
- Private Links to secure access for each cloud
provider
- Using CI / CD automation, based on terraform
- Self hosted Connectors on EMOBG clouds
(Internal flows)
- Fully managed connectors for external sources /
sinks (Salesforce, SAP..)
- Cluster linking feature as migration enabler
234. 238
238
Migration Plan to Confluent Cloud
- Stop all local Kafka brokers evolutions (No more flow on
them)
- Migration of technical flows : CDC, JDBC Connectors
- Replication of current kafka local configuration in new
confluent cluster
- Connection of Data sources and Data sinks to new clusters
- Assessment, Quality assurance & Validation with teams
- Migration of functional flows : Publish / Subscribe
- Confluent Cloud CI / CD pipeline shared and used in full
autonomy by teams
- Pilote phase with selected teams (Learning path)
- Full Migration Tribe by tribe (10 tribes)
235. 239
239
Project Status and Next Steps
Project status
- Foundations
- AWS & GCP Terraform CICD pipeline
- Production & Non production environments & Clusters
- Self Hosted Cluster Connect on AWS
- Security flow Access through OIDC & CC Identity Pool
- Migration Status
- Tech flows : Self Hosted Debezium connectors migrated to Confluent Cloud
- Functional flows :
- Kafka Legacy Topics Replication to confluent Cloud
- Connect sources & sinks to CC topics (end of Q1 2024)
236. 240
240
Project Status and Next Steps
Next steps & Opportunities
- Big milestone : Data platform BI, Data Analytics Integration
- Tech Transformation & Azure cloud extension
- Buy First approach & Third parties flows (SAP, Salesforce connectors)
238. How Everysens made its product
pivot a success with confluent cloud
Dai-Chinh Nguyen
CTO
Everysens
Luc Jallerat
Senior Back Developer
Everysens
239. How Everysens made its product pivot a success
with confluent.cloud
Luc Jallerat (Senior Backend Developer)
Dai-Chinh Nguyen (CTO)
October 2023
240.
241. Titre
Everysens: Smart collaboration to decarbonise freight transportation
Why
Why
How
How
What
What
✔ 55+ employees: 60% engineers & products
✔ 3 Offices in Paris, Lille and Duisburg
✔ One-stop shop for rail users
✔ 8 years of expertise in Rail Freight Digitisation
✔ A team experienced in deploying international
projects
Decarbonize Freight Transport
Collaborative SaaS Solution “TVMS”
● The largest integrated rail freight
ecosystem
● A SaaS tool made by and for
shippers
● Single Source of Truth for shared
Data
● Leveraging real-time data in rail
freight processes
242. Titre
What does a TVMS do ?
Day-to-day challenges of a logistic operator
● Plan & operate freight transports
● Anticipate loading/unloading operations
● Challenge carriers’ performance
● Secure communication with partners
● Optimize wagon fleet size
● Reduce logistic operation costs
● Reduce logistic operation CO2 emissions
● …
Everysens TVMS facilitates those operations
243. Titre
Once upon a time…
2016
2019
2020-2021
RAIL FREIGHT VISIBILITY
SYSTEM (SaaS)
RAIL FREIGHT TRANSPORT
MANAGEMENT SYSTEM (SaaS)
RAIL FREIGHT TRANSPORT AND
VISIBILITY MANAGEMENT SYSTEM (SaaS)
2015
IOT DEVICE MAKER FOR ASSET
LOCALISATION
Move to Cloud (GCP) From self hosted Kafka to
Confluent cloud
2022-2023
OPENING OF OUR
GERMAN OFFICE IN
DUISBURG
AND
FUNDRAISING OF 6M€
244. Titre
How Technology supported those transformations ? (1/2)
1.
From IoT sensors to a SaaS
Visibility System
Main Challenge :
SaaS system design principles
1 2
Modular service-based architecture
API & Event-based communication
Agility & Continuous Delivery
Container orchestration
Cloud infrastructure & managed services
Cloud Native Interoperable
Standard public API
Data Integration middleware
Master Data Standards
3
Data Centric
Data Analytics
Real time data processing
Data Science
4
Reliable & Secured
Scalability
Resiliency
Recoverability
Security policy & Legal compliance
245. Titre
How Technology supported those transformations ? (2/2)
2.
Adding the “V” to
the TVMS
Main Challenge :
Seamless merging of Visibility & TMS systems
TMS VISIBILITY
Contract
Asset
Asset
Type
Route
Goods
Order
Contact
Goods
Route
Transpor
t
246. Titre
How Technology supported those transformations ? (2/2)
1 Golden Source
+
2 Domains
=
Exchanging Transactional
Data
+
Sharing Static Data
Referential
TMS VISIBILITY
Front TMS
static data
Front Visibility
static data
Front MDR
static data
251. Titre
The rest of the journey
???
???
???
Integrationof Flink for a global
Past + Present perspective in
real-time
General WebHook Catalogconnected to
our internalevents
ModularRealTime Fully-IntegratedGlobal TVMS
System
???
TrackingEngine computingimpactsof
unorderedevents on both the past and
the present
From a Batch Driven approach
To an Event Drivenone
254. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Speakers
Mohamed Hamza Ben Mansour
Senior Solution Architect , AWS, France
Mickael Baye,
Senior Solution Architect , AWS, France
255. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Agenda
1. Real time is everywhere
2. Confluent on AWS
3. What our customers do together
4. Wrap up
259
256. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
1. Real time is everywhere
2. Confluent on AWS
3. What customers do with Confluent on AWS
4. Wrap Up
260
257. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Event Streaming allow us to set Data in Motion:
Continuously processing evolving streams of data in real-time
Rich front-end
customer
experiences
Real-time
Events
Real-time
Event Streams and Analysis
A Sale A shipment
A Trade
A Customer
Experience
Real-time
backend
operations
258. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Real time in everyday life
262
Anomaly and fraud detection
Empowering IoT analytics
Nourishing marketing campaigns
Real-time personalization
Tailoring customer experience in real time
Supporting healthcare and emergency services
259. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Other
Systems
Other
Systems
Kafka
Connect
Kafka Cluster
Kafka
Connect
Apache Kafka is an Event Streaming Platform
260. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
ksqlDB
Meeting you where you are
261. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. 265
The standard across industries
Finance & Banking Insurance Telecom
Travel & Retail
10 OUT
OF 10 8 OUT
OF 8
Fortune 500 Companies
Using Apache Kafka
70%
Transportation Energy & Utilities Entertainment
Technology
8 OUT
OF 10 9 OUT
OF 10
10 OUT
OF 10
10 OUT
OF 10
10 OUT
OF 10 8 OUT
OF 10
262. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
1. Real time is everywhere
2. Confluent on AWS
3. What customers do with Confluent on AWS
4. Wrap Up
266
263. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Cloud-native, complete, everywhere
Re-imagined Kafka Experience
Fully Managed
No Ops
On AWS
264. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Cloud-native, complete, everywhere
Integrated Solutions
● Data Lake/Warehouse
modernization
● Mainframe offload
● Streaming Analytics
● Hybrid Cloud App Modernization
● Industry specific use-cases
OSS Developer Traction
● 100s of thousands of Kafka OSS
developers in the enterprise
Accelerate Cloud
Migrations
● No complex Lift-n-Shift
● Maintain business continuity with
zero-downtime
● Break silos to enable immediate
App/Data innovation in cloud
True Hybrid-Cloud
Architectures
● Across global multi-DCs & cloud
● Leverage legacy investments with
Hybrid Kafka & bidirectional sync
● Shift legacy $ spend to AWS by
offloading Mainframe, Oracle,...
Meet you
where you are
● 200+ pre-built connectors
including S3, RedShift, Lambda,...
● Support Well–Architected
Scenarios
265. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. 269
Out-of-box integration with
popular services
AWS Native Services
Top-5 Global ISV for S3 Data Volume
3rd-Party ISV Services
Native integrations
266. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. 270
Confluent and AWS: Better together
267. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Lots of integrations ☺
271
268. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. 272
Redshift Sink
Lambda Sink
AWS Direct
Connect
Replicator
LEGACY EDW
MAINFRAME
LEGACY DB
JDBC / CDC
connectors
Connect
Leverage 130+ Confluent pre-built connectors
Modernize
Value added apps, increase agility, reduce TCO
On-prem AWS
Cloud
Bridge
Hybrid cloud streaming
Amazon Athena
AWS Glue
SageMaker
Lake Formation
Amazon
DynamoDB
Amazon
Aurora
S3 Sink
Data Streams
Apps
ksqlDB
Connect to all AWS
269. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. 273
270. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
1. Real time is everywhere
2. Confluent on AWS
3. What customers do with Confluent on AWS
4. Wrap Up
274
271. Challenge: Modernizing legacy systems for traditional
banks to enable them to innovate faster, deliver hyper-
personalized customer experiences, and compete with
digital- native banks.
Solution: Deliver a cloud-native SaaS solution—
powered by Confluent Cloud’s real-time data streaming
platform.
Results:
● Reduced costs with increased agility and faster time
to market for traditional banks
● Achieved better hyper-personalized experiences for
banking customers
● Delivered a resilient and highly available platform
● Enhanced enterprise-grade security
● Reduced TCO with simplified management
“Our mission is to make banking 10x better for banks, for
customers, and society. To do that, we need a cloud-
native data streaming platform that is also 10x more
reliable, 10x more performant than Apache Kafka.”
272. Challenge: Design and maintain a resilient IT
infrastructure that can ensure continued seamless
grocery delivery during a period of unprecedented
growth.
Solution: Confluent Cloud for a real-time, data platform
that unlocks the full value of streaming data and
empowers data visibility, agility, and flexibility across a
rapidly growing organization.
Results:
● Better inventory management via real-time data
● Reduced TCO
● Improved fraud detection
● Faster execution
“For me to go hire a bunch of engineers to babysit Kafka,
I don't have the ability to go do that. Being able to
offload those concerns [to Confluent] is such a relief for
us and lets us focus on delivering value to the
organization and not worrying about ops and the other
overhead”
– Nate Kupp, Director of Engineering, Instacart
273. Challenge: Address legacy tech-related operational
overhead and scalability issues to allow for better
customer behavior analytics and improve internal
processes.
Solution: Confluent Cloud to save time and money by
reducing operational overhead and allowing for real-
time processing and easy scalability of event data.
Results:
● Reduced infrastructure costs by 40%
● Simplified, future-proof data architecture
● Improved infrastructure monitoring for better SLAs
and system health
● Elimination of data loss
“Confluent provides exactly what we dreamed of: an
ecosystem of tools to source and sink data from data
streams. It’s provided us not only with great data
pipeline agility and flexibility but also a highly simplified
infrastructure that’s allowed us to reduce costs.”
— Dima Kalashnikov, Technical Lead
274. Challenge: Build a conversational chatbot service that
incorporates complex technologies such as fulfillment,
natural-language understanding, and real-time analytics.
Solution: Use Confluent to build a fast, super-scalable
event-driven architecture that could handle immense
traffic spikes and also provide other guarantees around
delivery semantics.
Results:
● Near-zero downtime even during huge traffic spikes
● Rapid acceleration of new-skill onboarding
● Doubling of NPS rating
“We chose event-driven architecture as the core of our
platform, for which we needed a messaging service that
gave us all the guarantees…not to mention that it had to
be extremely scalable, highly available, and simple to use.
Kafka hit all of these markers, and by using Confluent
Cloud, our team was able to reduce the bottom line and
operational burden.”
— Ravi Vankamamidi, Senior Director, Technology, at Expedia Group
275. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
1. Real time is everywhere
2. Confluent on AWS
3. What customers do with Confluent on AWS
4. Wrap Up
279
276. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Amazon Redshift Warehousing with Confluent Cloud
Serverless with AWS and Confluent
Cloud
Real-time Sentiment Analysis with Confluent
Amazon ElastiCache and Confluent Cloud
confluent.awsworkshop.io
Try it out yourself !
277. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Learn more
Working with streaming data on AWS
https://aws.amazon.com/streaming-data/
Modern Data Architecture on AWS
https://go.aws/3OJDhFk
Build Modern Data Streaming Analytics
Architectures on AWS
https://go.aws/3bt0HAm
Derive Insights from Modern Data
https://go.aws/3xVU3dn