Developing custom transformation in the Kafka connect to minimize data redundancy | Siavash Sedghi, United Health Group
•
0 likes•525 views
Compacted topics grow over time and are often utilizing high performance, low latency and relatively expensive storage solutions. Reducing duplicated data plays a critical role in the size of compacted topics. with less data on the topics, the Kafka cluster consumes less disk space which in turn it leads to lower operation cost. in this use case-driven talk, we are going to demonstrate how our team at UnitedHealth Group leveraged existing transformers to extract data from the message metadata in the topic as well as how we developed our customized transformers to minimize the amount of duplicated data in each message in the topic.
Report
Share
Report
Share
Download to read offline
Recommended
Modernizing to a Cloud Data Architecture
Organizations with on-premises Hadoop infrastructure are bogged down by system complexity, unscalable infrastructure, and the increasing burden on DevOps to manage legacy architectures. Costs and resource utilization continue to go up while innovation has flatlined. In this session, you will learn why, now more than ever, enterprises are looking for cloud alternatives to Hadoop and are migrating off of the architecture in large numbers. You will also learn how elastic compute models’ benefits help one customer scale their analytics and AI workloads and best practices from their experience on a successful migration of their data and workloads to the cloud.
The Top 5 Apache Kafka Use Cases and Architectures in 2022
This document discusses the top 5 use cases and architectures for data in motion in 2022. It describes:
1) The Kappa architecture as an alternative to the Lambda architecture that uses a single stream to handle both real-time and batch data.
2) Hyper-personalized omnichannel experiences that integrate customer data from multiple sources in real-time to provide personalized experiences across channels.
3) Multi-cloud deployments using Apache Kafka and data mesh architectures to share data across different cloud platforms.
4) Edge analytics that deploy stream processing and Kafka brokers at the edge to enable low-latency use cases and offline functionality.
5) Real-time cybersecurity applications that use streaming data
When NOT to use Apache Kafka?
Apache Kafka is the de facto standard for data streaming to process data in motion. With its significant adoption growth across all industries, I get a very valid question every week: When NOT to use Apache Kafka? What limitations does the event streaming platform have? When does Kafka simply not provide the needed capabilities? How to qualify Kafka out as it is not the right tool for the job?
This session explores the DOs and DONTs. Separate sections explain when to use Kafka, when NOT to use Kafka, and when to MAYBE use Kafka.
No matter if you think about open source Apache Kafka, a cloud service like Confluent Cloud, or another technology using the Kafka protocol like Redpanda or Pulsar, check out this slide deck.
A detailed article about this topic:
https://www.kai-waehner.de/blog/2022/01/04/when-not-to-use-apache-kafka/
AWS Partner Webcast - Data Center Migration to the AWS Cloud
Migrating your data center to the cloud can reduce operating costs, and improve the service you deliver to your clients. However, preparing for migration takes thoughtful planning, and awareness of proven solutions.
Review this webinar to learn how RISO Inc. used Apps Associates services to move their data center to the AWS cloud. In so doing, RISO Inc. avoided capital intensive hardware purchases and redeployed IT staff to projects that directly supported their business goals. We'll also describe how you can integrate your existing environment with the cloud to create a hybrid architecture.
What you’ll learn:
• Considerations for moving workloads to the AWS Cloud
• An approach to design and implementation of infrastructure, and application migration
• Customer case study, RISO Inc., on successful data center migration with Apps Associates
Denodo’s Data Catalog: Bridging the Gap between Data and Business (APAC)
Watch full webinar here: https://bit.ly/3nxGFam
Self service is a major goal of modern data strategists. Denodo’s data catalog is a key piece in Denodo’s portfolio to bridge the gap between the technical data infrastructure and business users. It provides documentation, search, governance and collaboration capabilities, and data exploration wizards. It’s the perfect companion for a virtual layer to fully empower those self service initiatives with minimal IT intervention. It provides business users with the tool to generate their own insights with proper security, governance and guardrails.
In this session you will learn about:
- The role of a virtual semantic layer in self service initiatives
- What are the key capabilities of Denodo’s new Data Catalog
- Best practices and advanced tips for a successful deployment
- How customers are using the Denodo’s Data Catalog to enable self-service initiatives
DW Migration Webinar-March 2022.pptx
The document discusses migrating a data warehouse to the Databricks Lakehouse Platform. It outlines why legacy data warehouses are struggling, how the Databricks Platform addresses these issues, and key considerations for modern analytics and data warehousing. The document then provides an overview of the migration methodology, approach, strategies, and key takeaways for moving to a lakehouse on Databricks.
Getting Started with Delta Lake on Databricks
In this session, we'll cover how with delta lake we can store the data and tables in databricks in an optimized way
Time to Talk about Data Mesh
This document discusses data mesh, a distributed data management approach for microservices. It outlines the challenges of implementing microservice architecture including data decoupling, sharing data across domains, and data consistency. It then introduces data mesh as a solution, describing how to build the necessary infrastructure using technologies like Kubernetes and YAML to quickly deploy data pipelines and provision data across services and applications in a distributed manner. The document provides examples of how data mesh can be used to improve legacy system integration, batch processing efficiency, multi-source data aggregation, and cross-cloud/environment integration.
Recommended
Modernizing to a Cloud Data Architecture
Organizations with on-premises Hadoop infrastructure are bogged down by system complexity, unscalable infrastructure, and the increasing burden on DevOps to manage legacy architectures. Costs and resource utilization continue to go up while innovation has flatlined. In this session, you will learn why, now more than ever, enterprises are looking for cloud alternatives to Hadoop and are migrating off of the architecture in large numbers. You will also learn how elastic compute models’ benefits help one customer scale their analytics and AI workloads and best practices from their experience on a successful migration of their data and workloads to the cloud.
The Top 5 Apache Kafka Use Cases and Architectures in 2022
This document discusses the top 5 use cases and architectures for data in motion in 2022. It describes:
1) The Kappa architecture as an alternative to the Lambda architecture that uses a single stream to handle both real-time and batch data.
2) Hyper-personalized omnichannel experiences that integrate customer data from multiple sources in real-time to provide personalized experiences across channels.
3) Multi-cloud deployments using Apache Kafka and data mesh architectures to share data across different cloud platforms.
4) Edge analytics that deploy stream processing and Kafka brokers at the edge to enable low-latency use cases and offline functionality.
5) Real-time cybersecurity applications that use streaming data
When NOT to use Apache Kafka?
Apache Kafka is the de facto standard for data streaming to process data in motion. With its significant adoption growth across all industries, I get a very valid question every week: When NOT to use Apache Kafka? What limitations does the event streaming platform have? When does Kafka simply not provide the needed capabilities? How to qualify Kafka out as it is not the right tool for the job?
This session explores the DOs and DONTs. Separate sections explain when to use Kafka, when NOT to use Kafka, and when to MAYBE use Kafka.
No matter if you think about open source Apache Kafka, a cloud service like Confluent Cloud, or another technology using the Kafka protocol like Redpanda or Pulsar, check out this slide deck.
A detailed article about this topic:
https://www.kai-waehner.de/blog/2022/01/04/when-not-to-use-apache-kafka/
AWS Partner Webcast - Data Center Migration to the AWS Cloud
Migrating your data center to the cloud can reduce operating costs, and improve the service you deliver to your clients. However, preparing for migration takes thoughtful planning, and awareness of proven solutions.
Review this webinar to learn how RISO Inc. used Apps Associates services to move their data center to the AWS cloud. In so doing, RISO Inc. avoided capital intensive hardware purchases and redeployed IT staff to projects that directly supported their business goals. We'll also describe how you can integrate your existing environment with the cloud to create a hybrid architecture.
What you’ll learn:
• Considerations for moving workloads to the AWS Cloud
• An approach to design and implementation of infrastructure, and application migration
• Customer case study, RISO Inc., on successful data center migration with Apps Associates
Denodo’s Data Catalog: Bridging the Gap between Data and Business (APAC)
Watch full webinar here: https://bit.ly/3nxGFam
Self service is a major goal of modern data strategists. Denodo’s data catalog is a key piece in Denodo’s portfolio to bridge the gap between the technical data infrastructure and business users. It provides documentation, search, governance and collaboration capabilities, and data exploration wizards. It’s the perfect companion for a virtual layer to fully empower those self service initiatives with minimal IT intervention. It provides business users with the tool to generate their own insights with proper security, governance and guardrails.
In this session you will learn about:
- The role of a virtual semantic layer in self service initiatives
- What are the key capabilities of Denodo’s new Data Catalog
- Best practices and advanced tips for a successful deployment
- How customers are using the Denodo’s Data Catalog to enable self-service initiatives
DW Migration Webinar-March 2022.pptx
The document discusses migrating a data warehouse to the Databricks Lakehouse Platform. It outlines why legacy data warehouses are struggling, how the Databricks Platform addresses these issues, and key considerations for modern analytics and data warehousing. The document then provides an overview of the migration methodology, approach, strategies, and key takeaways for moving to a lakehouse on Databricks.
Getting Started with Delta Lake on Databricks
In this session, we'll cover how with delta lake we can store the data and tables in databricks in an optimized way
Time to Talk about Data Mesh
This document discusses data mesh, a distributed data management approach for microservices. It outlines the challenges of implementing microservice architecture including data decoupling, sharing data across domains, and data consistency. It then introduces data mesh as a solution, describing how to build the necessary infrastructure using technologies like Kubernetes and YAML to quickly deploy data pipelines and provision data across services and applications in a distributed manner. The document provides examples of how data mesh can be used to improve legacy system integration, batch processing efficiency, multi-source data aggregation, and cross-cloud/environment integration.
Making Data Timelier and More Reliable with Lakehouse Technology
Enterprise data architectures usually contain many systems—data lakes, message queues, and data warehouses—that data must pass through before it can be analyzed. Each transfer step between systems adds a delay and a potential source of errors. What if we could remove all these steps? In recent years, cloud storage and new open source systems have enabled a radically new architecture: the lakehouse, an ACID transactional layer over cloud storage that can provide streaming, management features, indexing, and high-performance access similar to a data warehouse. Thousands of organizations including the largest Internet companies are now using lakehouses to replace separate data lake, warehouse and streaming systems and deliver high-quality data faster internally. I’ll discuss the key trends and recent advances in this area based on Delta Lake, the most widely used open source lakehouse platform, which was developed at Databricks.
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Learn the differences between an event-driven streaming platform and middleware like MQ, ETL and ESBs – including best practices and anti-patterns, but also how these concepts and tools complement each other in an enterprise architecture.
Extract-Transform-Load (ETL) is still a widely-used pattern to move data between different systems via batch processing. Due to its challenges in today’s world where real time is the new standard, an Enterprise Service Bus (ESB) is used in many enterprises as integration backbone between any kind of microservice, legacy application or cloud service to move data via SOAP / REST Web Services or other technologies. Stream Processing is often added as its own component in the enterprise architecture for correlation of different events to implement contextual rules and stateful analytics. Using all these components introduces challenges and complexities in development and operations.
This session discusses how teams in different industries solve these challenges by building a native streaming platform from the ground up instead of using ETL and ESB tools in their architecture. This allows to build and deploy independent, mission-critical streaming real time application and microservices. The architecture leverages distributed processing and fault-tolerance with fast failover, no-downtime rolling deployments and the ability to reprocess events, so you can recalculate output when your code changes. Integration and Stream Processing are still key functionality but can be realized in real time natively instead of using additional ETL, ESB or Stream Processing tools.
Building Modern Data Platform with Microsoft Azure
This document provides an overview of building a modern cloud analytics solution using Microsoft Azure. It discusses the role of analytics, a history of cloud computing, and a data warehouse modernization project. Key challenges covered include lack of notifications, logging, self-service BI, and integrating streaming data. The document proposes solutions to these challenges using Azure services like Data Factory, Kafka, Databricks, and SQL Data Warehouse. It also discusses alternative implementations using tools like Matillion ETL and Snowflake.
AWS Community Day 2022 Joe Daly FinOps
The document discusses FinOps, which is a cloud financial management discipline that aims to help engineering, finance, and business teams collaborate on data-driven spending decisions to maximize business value from cloud usage. It provides an overview of the FinOps Foundation's mission to advance individuals and build mastery in cloud cost optimization. It also briefly outlines the growth of FinOps roles and teams over time from initial small teams to more mature larger teams, and common challenges organizations face in implementing FinOps.
Architect’s Open-Source Guide for a Data Mesh Architecture
Data Mesh is an innovative concept addressing many data challenges from an architectural, cultural, and organizational perspective. But is the world ready to implement Data Mesh?
In this session, we will review the importance of core Data Mesh principles, what they can offer, and when it is a good idea to try a Data Mesh architecture. We will discuss common challenges with implementation of Data Mesh systems and focus on the role of open-source projects for it. Projects like Apache Spark can play a key part in standardized infrastructure platform implementation of Data Mesh. We will examine the landscape of useful data engineering open-source projects to utilize in several areas of a Data Mesh system in practice, along with an architectural example. We will touch on what work (culture, tools, mindset) needs to be done to ensure Data Mesh is more accessible for engineers in the industry.
The audience will leave with a good understanding of the benefits of Data Mesh architecture, common challenges, and the role of Apache Spark and other open-source projects for its implementation in real systems.
This session is targeted for architects, decision-makers, data-engineers, and system designers.
Your Journey to the Cloud
Most organizations have heard of the cloud, but not everyone is at the same stage on their cloud journey. This SlideShare overviews how to best leverage the cloud, the multitude of options and the typical stages of transition. Also outlined are the steps you can take to get where you want to go on your cloud journey faster and more efficiently.
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Enterprise integration is more challenging than ever before. The IT evolution requires the integration of more and more technologies. Applications are deployed across the edge, hybrid, and multi-cloud architectures. Traditional middleware such as MQ, ETL, ESB does not scale well enough or only processes data in batch instead of real-time.
This presentation explores why Apache Kafka is the new black for integration projects, how Kafka fits into the discussion around cloud-native iPaaS (Integration Platform as a Service) solutions, and why event streaming is a new software category.
A concrete real-world example shows the difference between event streaming and traditional integration platforms respectively cloud-native iPaaS.
Video Recording of this presentation:
https://www.youtube.com/watch?v=I8yZwKg_IJc&t=2842s
Blog post about this topic:
https://www.kai-waehner.de/blog/2021/11/03/apache-kafka-cloud-native-ipaas-versus-mq-etl-esb-middleware/
App Modernisation with Microsoft Azure
There are options beyond a straight forward lift and shift into Infrastructure as a Service. This session is about learning about how Azure helps modernize applications faster utilising modern technologies like PaaS, containers and serverless
Snowflake: The Good, the Bad, and the Ugly
Learn how to solve the top 3 challenges Snowflake customers face, and what you can do to ensure high-performance, intelligent analytics at any scale. Ideal for those currently using Snowflake and those considering it. Learn more at: https://kyligence.io/
Databricks for Dummies
Tech talk on what Azure Databricks is, why you should learn it and how to get started. We'll use PySpark and talk about some real live examples from the trenches, including the pitfalls of leaving your clusters running accidentally and receiving a huge bill ;)
After this you will hopefully switch to Spark-as-a-service and get rid of your HDInsight/Hadoop clusters.
This is part 1 of an 8 part Data Science for Dummies series:
Databricks for dummies
Titanic survival prediction with Databricks + Python + Spark ML
Titanic with Azure Machine Learning Studio
Titanic with Databricks + Azure Machine Learning Service
Titanic with Databricks + MLS + AutoML
Titanic with Databricks + MLFlow
Titanic with DataRobot
Deployment, DevOps/MLops and Operationalization
Mass Migration Strategy - A Key Step in the Enterprise Transformation - AWS C...
Vortrag "Mass Migration Strategy - A Key Step in the Enterprise Transformation" von Angelo Crippa beim AWS Cloud Web Day für Mittelstand und Großunternehmen. Alle Videos und Präsentationen finden Sie hier: http://amzn.to/1VUJZsT
Data Center Migration to the AWS Cloud
This webinar discusses RISO Inc.'s experience migrating their on-premise data center to the AWS cloud with assistance from Apps Associates. [1] Apps Associates designed and implemented the new infrastructure on AWS, migrating applications like Oracle ERP and SQL servers. [2] This provided benefits like a 35% reduction in backup costs, 50% fewer IT vendors, and the ability to relocate offices without interrupting operations. [3] The webinar explores considerations for cloud migrations and the hybrid cloud model.
Apache Kafka and the Data Mesh | Michael Noll, Confluent
Data mesh is a relatively recent term that describes a set of principles that good modern data systems uphold. A kind of “microservices” for the data-centric world. While the data mesh is not technology-specific as a pattern, the building of systems that adopt and implement data mesh principles have a relatively long history under different guises.
In this talk, we share our recommendations and picks of what every developer should know about building a streaming data mesh with Kafka. We introduce the four principles of the data mesh: domain-driven decentralization, data as a product, self-service data platform, and federated governance. We then cover topics such as the differences between working with event streams versus centralized approaches and highlight the key characteristics that make streams a great fit for implementing a mesh, such as their ability to capture both real-time and historical data. We’ll examine how to onboard data from existing systems into a mesh, modelling the communication within the mesh, how to deal with changes to your domain’s “public” data, give examples of global standards for governance, and discuss the importance of taking a product-centric view on data sources and the data sets they share.
Databricks on AWS.pptx
Databricks on AWS provides a unified analytics platform using Apache Spark. It allows companies to unify their data science, engineering, and business teams on one platform. Databricks accelerates innovation across the big data and machine learning lifecycle. It uniquely combines data and AI technologies on Apache Spark. Enterprises face challenges beyond just Apache Spark, including having data scientists and engineers in separate silos with complex data pipelines and infrastructure. Azure Databricks provides a fast, easy, and collaborative Apache Spark-based analytics platform on Azure that is optimized for the cloud. It offers the benefits of Databricks and Microsoft with one-click setup, a collaborative workspace, and native integration with Azure services. Over 500 customers participated in the
Azure data analytics platform - A reference architecture
This document provides an overview of Azure data analytics architecture using the Lambda architecture pattern. It covers Azure data and services, including ingestion, storage, processing, analysis and interaction services. It provides a brief overview of the Lambda architecture including the batch layer for pre-computed views, speed layer for real-time views, and serving layer. It also discusses Azure data distribution, SQL Data Warehouse architecture and design best practices, and data modeling guidance.
Evolution from EDA to Data Mesh: Data in Motion
Thoughtworks Zhamak Dehghani observations on these traditional approaches’s failure modes, inspired her to develop an alternative big data management architecture that she aptly named the Data Mesh. This represents a paradigm shift that draws from modern distributed architecture and is founded on the principles of domain-driven design, self-serve platform, and product thinking with Data. In the last decade Apache Kafka has established a new category of data management infrastructure for data in motion that has been leveraged in modern distributed data architectures.
How to Streamline DataOps on AWS
The quest for the insight-driven enterprise has spurned a mass exodus to the cloud. But cloud data ecosystems can be very complex with multiple data storage and processing options.
These slides-based on the webinar featuring leading IT analyst firm EMA, Amazon Web Services (AWS), and Trifacta--will help you: understand technology trends that simplify your analytics modernization journey; learn best practices to operationalize data management on AWS; establish operational excellence leveraging AWS data storage and processing; accelerate time-to-value for analytics projects with data preparation on AWS.
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse Analytics (Presented at Global Azure Norway on April 16th, 2021)
DataOps: An Agile Method for Data-Driven Organizations
DataOps expands DevOps philosophy to include data-heavy roles (data engineering & data science). DataOps uses better cross-functional collaboration for flexibility, fast time to value and an agile workflow for data-intensive applications including machine learning pipelines. (Strata Data San Jose March 2018)
SAP on Azure Technical Pitch Deck
This technical pitch deck summarizes SAP solutions on Microsoft Azure. It outlines challenges with on-premises SAP environments and how moving to SAP HANA in the cloud on Azure can enable faster processes, accelerated innovation, and 360-degree insights. It then covers the journey to migrating SAP landscapes to SAP HANA and Azure, including lifting SAP systems with any database to Azure, migrating to SAP HANA, and migrating to S/4HANA. Finally, it discusses how Azure enables insights from SAP and non-SAP data.
apidays LIVE Australia 2021 - Composable data for the composable enterprise b...
apidays LIVE Australia 2021 - Accelerating Digital
September 15 & 16, 2021
Composable data for the composable enterprise
Matt McLarty, Global Leader, API Strategy at MuleSoft
Digital Transformation Journey
This document discusses digital transformation and how companies can reimagine customer value in a digital world. It outlines how digital disruption will reshape markets in 3 years or less, displacing 40% of incumbents. It advocates adopting a platform approach and developing powerful ecosystems across industries. It also describes Cisco's operating model transformation to deliver IT services using a hybrid cloud approach with everything as a service. This simplifies operations and delivers data-driven insights for continuous innovation.
More Related Content
What's hot
Making Data Timelier and More Reliable with Lakehouse Technology
Enterprise data architectures usually contain many systems—data lakes, message queues, and data warehouses—that data must pass through before it can be analyzed. Each transfer step between systems adds a delay and a potential source of errors. What if we could remove all these steps? In recent years, cloud storage and new open source systems have enabled a radically new architecture: the lakehouse, an ACID transactional layer over cloud storage that can provide streaming, management features, indexing, and high-performance access similar to a data warehouse. Thousands of organizations including the largest Internet companies are now using lakehouses to replace separate data lake, warehouse and streaming systems and deliver high-quality data faster internally. I’ll discuss the key trends and recent advances in this area based on Delta Lake, the most widely used open source lakehouse platform, which was developed at Databricks.
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Learn the differences between an event-driven streaming platform and middleware like MQ, ETL and ESBs – including best practices and anti-patterns, but also how these concepts and tools complement each other in an enterprise architecture.
Extract-Transform-Load (ETL) is still a widely-used pattern to move data between different systems via batch processing. Due to its challenges in today’s world where real time is the new standard, an Enterprise Service Bus (ESB) is used in many enterprises as integration backbone between any kind of microservice, legacy application or cloud service to move data via SOAP / REST Web Services or other technologies. Stream Processing is often added as its own component in the enterprise architecture for correlation of different events to implement contextual rules and stateful analytics. Using all these components introduces challenges and complexities in development and operations.
This session discusses how teams in different industries solve these challenges by building a native streaming platform from the ground up instead of using ETL and ESB tools in their architecture. This allows to build and deploy independent, mission-critical streaming real time application and microservices. The architecture leverages distributed processing and fault-tolerance with fast failover, no-downtime rolling deployments and the ability to reprocess events, so you can recalculate output when your code changes. Integration and Stream Processing are still key functionality but can be realized in real time natively instead of using additional ETL, ESB or Stream Processing tools.
Building Modern Data Platform with Microsoft Azure
This document provides an overview of building a modern cloud analytics solution using Microsoft Azure. It discusses the role of analytics, a history of cloud computing, and a data warehouse modernization project. Key challenges covered include lack of notifications, logging, self-service BI, and integrating streaming data. The document proposes solutions to these challenges using Azure services like Data Factory, Kafka, Databricks, and SQL Data Warehouse. It also discusses alternative implementations using tools like Matillion ETL and Snowflake.
AWS Community Day 2022 Joe Daly FinOps
The document discusses FinOps, which is a cloud financial management discipline that aims to help engineering, finance, and business teams collaborate on data-driven spending decisions to maximize business value from cloud usage. It provides an overview of the FinOps Foundation's mission to advance individuals and build mastery in cloud cost optimization. It also briefly outlines the growth of FinOps roles and teams over time from initial small teams to more mature larger teams, and common challenges organizations face in implementing FinOps.
Architect’s Open-Source Guide for a Data Mesh Architecture
Data Mesh is an innovative concept addressing many data challenges from an architectural, cultural, and organizational perspective. But is the world ready to implement Data Mesh?
In this session, we will review the importance of core Data Mesh principles, what they can offer, and when it is a good idea to try a Data Mesh architecture. We will discuss common challenges with implementation of Data Mesh systems and focus on the role of open-source projects for it. Projects like Apache Spark can play a key part in standardized infrastructure platform implementation of Data Mesh. We will examine the landscape of useful data engineering open-source projects to utilize in several areas of a Data Mesh system in practice, along with an architectural example. We will touch on what work (culture, tools, mindset) needs to be done to ensure Data Mesh is more accessible for engineers in the industry.
The audience will leave with a good understanding of the benefits of Data Mesh architecture, common challenges, and the role of Apache Spark and other open-source projects for its implementation in real systems.
This session is targeted for architects, decision-makers, data-engineers, and system designers.
Your Journey to the Cloud
Most organizations have heard of the cloud, but not everyone is at the same stage on their cloud journey. This SlideShare overviews how to best leverage the cloud, the multitude of options and the typical stages of transition. Also outlined are the steps you can take to get where you want to go on your cloud journey faster and more efficiently.
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Enterprise integration is more challenging than ever before. The IT evolution requires the integration of more and more technologies. Applications are deployed across the edge, hybrid, and multi-cloud architectures. Traditional middleware such as MQ, ETL, ESB does not scale well enough or only processes data in batch instead of real-time.
This presentation explores why Apache Kafka is the new black for integration projects, how Kafka fits into the discussion around cloud-native iPaaS (Integration Platform as a Service) solutions, and why event streaming is a new software category.
A concrete real-world example shows the difference between event streaming and traditional integration platforms respectively cloud-native iPaaS.
Video Recording of this presentation:
https://www.youtube.com/watch?v=I8yZwKg_IJc&t=2842s
Blog post about this topic:
https://www.kai-waehner.de/blog/2021/11/03/apache-kafka-cloud-native-ipaas-versus-mq-etl-esb-middleware/
App Modernisation with Microsoft Azure
There are options beyond a straight forward lift and shift into Infrastructure as a Service. This session is about learning about how Azure helps modernize applications faster utilising modern technologies like PaaS, containers and serverless
Snowflake: The Good, the Bad, and the Ugly
Learn how to solve the top 3 challenges Snowflake customers face, and what you can do to ensure high-performance, intelligent analytics at any scale. Ideal for those currently using Snowflake and those considering it. Learn more at: https://kyligence.io/
Databricks for Dummies
Tech talk on what Azure Databricks is, why you should learn it and how to get started. We'll use PySpark and talk about some real live examples from the trenches, including the pitfalls of leaving your clusters running accidentally and receiving a huge bill ;)
After this you will hopefully switch to Spark-as-a-service and get rid of your HDInsight/Hadoop clusters.
This is part 1 of an 8 part Data Science for Dummies series:
Databricks for dummies
Titanic survival prediction with Databricks + Python + Spark ML
Titanic with Azure Machine Learning Studio
Titanic with Databricks + Azure Machine Learning Service
Titanic with Databricks + MLS + AutoML
Titanic with Databricks + MLFlow
Titanic with DataRobot
Deployment, DevOps/MLops and Operationalization
Mass Migration Strategy - A Key Step in the Enterprise Transformation - AWS C...
Vortrag "Mass Migration Strategy - A Key Step in the Enterprise Transformation" von Angelo Crippa beim AWS Cloud Web Day für Mittelstand und Großunternehmen. Alle Videos und Präsentationen finden Sie hier: http://amzn.to/1VUJZsT
Data Center Migration to the AWS Cloud
This webinar discusses RISO Inc.'s experience migrating their on-premise data center to the AWS cloud with assistance from Apps Associates. [1] Apps Associates designed and implemented the new infrastructure on AWS, migrating applications like Oracle ERP and SQL servers. [2] This provided benefits like a 35% reduction in backup costs, 50% fewer IT vendors, and the ability to relocate offices without interrupting operations. [3] The webinar explores considerations for cloud migrations and the hybrid cloud model.
Apache Kafka and the Data Mesh | Michael Noll, Confluent
Data mesh is a relatively recent term that describes a set of principles that good modern data systems uphold. A kind of “microservices” for the data-centric world. While the data mesh is not technology-specific as a pattern, the building of systems that adopt and implement data mesh principles have a relatively long history under different guises.
In this talk, we share our recommendations and picks of what every developer should know about building a streaming data mesh with Kafka. We introduce the four principles of the data mesh: domain-driven decentralization, data as a product, self-service data platform, and federated governance. We then cover topics such as the differences between working with event streams versus centralized approaches and highlight the key characteristics that make streams a great fit for implementing a mesh, such as their ability to capture both real-time and historical data. We’ll examine how to onboard data from existing systems into a mesh, modelling the communication within the mesh, how to deal with changes to your domain’s “public” data, give examples of global standards for governance, and discuss the importance of taking a product-centric view on data sources and the data sets they share.
Databricks on AWS.pptx
Databricks on AWS provides a unified analytics platform using Apache Spark. It allows companies to unify their data science, engineering, and business teams on one platform. Databricks accelerates innovation across the big data and machine learning lifecycle. It uniquely combines data and AI technologies on Apache Spark. Enterprises face challenges beyond just Apache Spark, including having data scientists and engineers in separate silos with complex data pipelines and infrastructure. Azure Databricks provides a fast, easy, and collaborative Apache Spark-based analytics platform on Azure that is optimized for the cloud. It offers the benefits of Databricks and Microsoft with one-click setup, a collaborative workspace, and native integration with Azure services. Over 500 customers participated in the
Azure data analytics platform - A reference architecture
This document provides an overview of Azure data analytics architecture using the Lambda architecture pattern. It covers Azure data and services, including ingestion, storage, processing, analysis and interaction services. It provides a brief overview of the Lambda architecture including the batch layer for pre-computed views, speed layer for real-time views, and serving layer. It also discusses Azure data distribution, SQL Data Warehouse architecture and design best practices, and data modeling guidance.
Evolution from EDA to Data Mesh: Data in Motion
Thoughtworks Zhamak Dehghani observations on these traditional approaches’s failure modes, inspired her to develop an alternative big data management architecture that she aptly named the Data Mesh. This represents a paradigm shift that draws from modern distributed architecture and is founded on the principles of domain-driven design, self-serve platform, and product thinking with Data. In the last decade Apache Kafka has established a new category of data management infrastructure for data in motion that has been leveraged in modern distributed data architectures.
How to Streamline DataOps on AWS
The quest for the insight-driven enterprise has spurned a mass exodus to the cloud. But cloud data ecosystems can be very complex with multiple data storage and processing options.
These slides-based on the webinar featuring leading IT analyst firm EMA, Amazon Web Services (AWS), and Trifacta--will help you: understand technology trends that simplify your analytics modernization journey; learn best practices to operationalize data management on AWS; establish operational excellence leveraging AWS data storage and processing; accelerate time-to-value for analytics projects with data preparation on AWS.
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse Analytics (Presented at Global Azure Norway on April 16th, 2021)
DataOps: An Agile Method for Data-Driven Organizations
DataOps expands DevOps philosophy to include data-heavy roles (data engineering & data science). DataOps uses better cross-functional collaboration for flexibility, fast time to value and an agile workflow for data-intensive applications including machine learning pipelines. (Strata Data San Jose March 2018)
SAP on Azure Technical Pitch Deck
This technical pitch deck summarizes SAP solutions on Microsoft Azure. It outlines challenges with on-premises SAP environments and how moving to SAP HANA in the cloud on Azure can enable faster processes, accelerated innovation, and 360-degree insights. It then covers the journey to migrating SAP landscapes to SAP HANA and Azure, including lifting SAP systems with any database to Azure, migrating to SAP HANA, and migrating to S/4HANA. Finally, it discusses how Azure enables insights from SAP and non-SAP data.
What's hot (20)
Making Data Timelier and More Reliable with Lakehouse Technology
Making Data Timelier and More Reliable with Lakehouse Technology
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Building Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft Azure
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Mass Migration Strategy - A Key Step in the Enterprise Transformation - AWS C...
Mass Migration Strategy - A Key Step in the Enterprise Transformation - AWS C...
Apache Kafka and the Data Mesh | Michael Noll, Confluent
Apache Kafka and the Data Mesh | Michael Noll, Confluent
Azure data analytics platform - A reference architecture
Azure data analytics platform - A reference architecture
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
DataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven Organizations
Similar to Developing custom transformation in the Kafka connect to minimize data redundancy | Siavash Sedghi, United Health Group
apidays LIVE Australia 2021 - Composable data for the composable enterprise b...
apidays LIVE Australia 2021 - Accelerating Digital
September 15 & 16, 2021
Composable data for the composable enterprise
Matt McLarty, Global Leader, API Strategy at MuleSoft
Digital Transformation Journey
This document discusses digital transformation and how companies can reimagine customer value in a digital world. It outlines how digital disruption will reshape markets in 3 years or less, displacing 40% of incumbents. It advocates adopting a platform approach and developing powerful ecosystems across industries. It also describes Cisco's operating model transformation to deliver IT services using a hybrid cloud approach with everything as a service. This simplifies operations and delivers data-driven insights for continuous innovation.
Streaming Data and Stream Processing with Apache Kafka
Apache Kafka is an open-source streaming platform that can be used to build real-time data pipelines and streaming applications. It addresses challenges with diverse data sets arriving at increasing rates. The document discusses how Apache Kafka can help with challenges around data integration, stream processing, and managing streaming platforms at scale. It also outlines key features of Apache Kafka like the Kafka Connect API for data integration, the Kafka Streams API for stream processing, and Confluent Control Center for monitoring and management.
Event Driven Architecture
This document discusses event driven architecture and stream data platforms. It describes how a stream data platform can be used for data integration and stream processing. The platform acts as a central hub that captures event streams from various systems and makes the data available to other systems like Hadoop and databases. It also discusses best practices like using a common data format (Avro) and schemas to simplify integration across systems.
Application Networks: Microservices and APIs at Netflix
Who better to talk about microservices — one of the hottest technology trends for 2016 — than Netflix? This streaming-entertainment giant began adopting them in 2009, years before the exact term even existed. Join MuleSoft and Netflix as they co-present the value that a microservices architecture can bring to your business, and see first hand the real-world implementation of APIs at Netflix. Then learn from MuleSoft’s CTO how APIs and DevOps are two important pillars of microservices and discover how they can become part of your application network.
Pivotal Big Data Suite: A Technical Overview
How and why are companies like Uber, Netflix and AirBnB so successful, what you need to in order to become successful in the same way that they are and how Pivotal can help you with that.
Speaker: Les Klein, EMEA CTO Data, Pivotal
Microservices at Netflix
Netflix has built a network of microservices to power its platform in a way that provides robustness, agility and speed at scale. The key aspects that enable this are:
1) Dividing the system into independent, self-contained services that are developed, operated and scaled independently but work together through well-defined APIs.
2) Establishing two pillars - API excellence through atomic services with clear boundaries and dependencies, and devops excellence through automation, containers and tools for resilience.
3) Planning for failure through chaos engineering, circuit breakers and fallbacks to ensure services can fail independently without bringing down the whole system.
Oracle GoldenGate
Oracle GoldenGate is the leading real-time data integration software provider in the industry - customers include 3 of the top 5 commercial banks, 3 of the top 3 busiest ATM networks, and 4 of the top 5 telecommunications providers.
Oracle GoldenGate moves transactional data in real-time across heterogeneous database, hardware and operating systems with minimal impact. The software platform captures, routes, and delivers data in real time, enabling organizations to maintain continuous uptime for critical applications during planned and unplanned outages.
Additionally, it moves data from transaction processing environments to read-only reporting databases and analytical applications for accurate, timely reporting and improved business intelligence for the enterprise.
Learn the new rules of cloud storage
Learn the new rules of cloud storage. SoftNAS is now Buurst, and we're about to change the enterprise cloud storage industry as you know it.
Read the slides from our groundbreaking live webinar announcement on 4/15/20 and learn how:
• To reduce your cloud storage costs, and save up to 80% on cloud storage costs and increase performance (yes, you read that right!)
• Applying configuration variables will maximize data performance, without storage limitations
• Companies such as Halliburton, SAP, and Boeing are already taking advantage of these rules and effectively managing Petabytes of data in the cloud
Who can benefit?
• Cloud Architects, CIO, CTO, VP Infrastructure, Data Center Architects, Platform Architects, Application Developers, Systems Engineers, Network Engineers, VP
Technology, VP IT, VP BI/Data Analytics, Solutions Architects
• Amazon Elastic File System (EFS) customers, Amazon FSx customers, Azure NetApp File System (NFS) customers, Isilon customers
apidays LIVE Australia 2021 - A cloud-native approach for open banking in act...
apidays LIVE Australia 2021 - Accelerating Digital
September 15 & 16, 2021
A cloud-native approach for open banking in action
Rafael Marins, Principal Product Marketing Manager at Red Hat
Pharmacy one source speed saves lives
Pharmacy OneSource chose to implement an in-memory data grid solution from GigaSpaces to address performance and scalability issues with their existing architecture. They conducted a pilot that showed a 6x increase in message processing. This allowed them to more quickly and cost-effectively deliver critical healthcare information to customers in real-time. The new architecture provides a foundation to process millions of messages daily and transition to cloud-based hosting.
Cloud computing and health care - Facing the Future
Jerry Fahrni discusses cloud computing and its potential advantages for healthcare. Cloud computing provides on-demand access to shared computing resources over the internet and has several benefits including lower costs, flexibility, and scalability. While security concerns remain, more healthcare organizations are adopting cloud-based solutions like electronic health records and image sharing to improve care coordination and lower costs. If implemented properly, cloud computing can help make patient health information more accessible to providers and support remote monitoring of chronic conditions.
Enterprise Innovations
AWS continues to rapidly innovate and grow its services. Over 1 million customers now use AWS each month, representing over 120% year-over-year growth. Revenue is also growing significantly, with an annual run rate now over $7 billion. Key trends include the rise of analytics solutions, real-time streaming analytics, machine learning and predictions, more database and storage options, high performance containers and serverless computing.
132177_16x9_GlobalAlliances_Presentation_NetApp_01_D[1]
This document provides information on NetApp's partnerships with various technology companies including Cisco, Citrix, Microsoft, Oracle, Red Hat, SAP, VMware, Amazon Web Services, and Google Cloud Platform. It highlights the benefits customers experience from NetApp and its partners' integrated solutions, such as increased efficiency, scalability, cost savings, and innovation. NetApp and its partners have many joint customers and years of collaboration developing integrated solutions across hardware, software, cloud, and virtualization platforms.
SAP Forum Ankara 2017 - "Verinin Merkezine Seyahat"
The document discusses digital transformation and the journey to data-driven insights. It provides an overview of data types and how data has grown exponentially over time. Both structured and unstructured data are discussed, with examples of semi-structured data like emails and reports. The value of understanding all data sources is emphasized for gaining competitive advantages through analytics. New technologies like complex event processing are enabling lightning-fast action based on diverse data. Finally, the presentation introduces SAP HANA Vora for bridging the divide between enterprise and big data systems to facilitate precision decision making.
apidays Australia 2023 - The Playful Bond Between REST And Data Streams, Warr...
apidays Australia 2023 - Platforms, Products, and People: The Power of APIs
October 11 & 12, 2023
https://www.apidays.global/australia/
The Playful Bond Between REST And Data Streams
Warren Vella, Solutions Engineer at Confluent
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
The Playful Bond Between REST And Data Streams
1. REST APIs have proliferated as a way to integrate microservices but don't meet all integration needs and can result in tight coupling between systems.
2. Using streaming data platforms like Kafka can help reduce the number of integration lines needed between systems and provides stronger delivery guarantees compared to REST APIs.
3. While REST APIs are good for synchronous requests and responses, a data streaming platform that includes both REST and streaming data capabilities can help integrate application and data systems using the best approach for different use cases and requirements.
Ericsson hds 8000 wp 16
The document discusses a new hyperscale data center solution from Ericsson called Ericsson Hyperscale Datacenter System 8000. It claims this solution can deliver significant cost savings for enterprise data centers by enabling higher utilization rates compared to traditional architectures. Specifically, it argues the solution can achieve CPU utilization up to 4 times higher, network utilization 26% higher, and allow one administrator to manage thousands of servers rather than hundreds. These efficiencies are achieved through a pooled infrastructure that allows dynamic allocation of resources and improved matching of capacity to workload demands.
Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Curren...
Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Current 2022
We see data and the way organizations using it to create unique customer experiences is being completely reimagined. Every time a customer clicks, types or swipes data is in constant motion spanning systems, environments and applications. This in turn requires business’ manage complex integrations, synchronizations and processing of data spread across cloud and on-prem environments.
To accelerate time to value, data needs to be seamlessly ingested, integrated and/or replicated to a cloud environment to take advantage of its analytical, BI and AI use cases. Google Cloud delivers a simplified approach for all your Data Movement needs through a highly differentiated product portfolio.
In this session, learn how organizations can unlock data value using best-in-class, cloud native products on Google Cloud and its partners such as Confluent.
Future of Power: Power Strategy and Offerings for Denmark - Steve Sibley
IBM is promoting its Power Systems as optimized for big data, analytics, mobile, social, and cloud workloads. Key highlights include Power Systems providing a flexible, secure infrastructure to support next generation applications and analytics on big data. IBM also emphasizes partnerships with software vendors and the open innovation through the OpenPOWER consortium to deliver client choice and drive down costs.
Similar to Developing custom transformation in the Kafka connect to minimize data redundancy | Siavash Sedghi, United Health Group (20)
apidays LIVE Australia 2021 - Composable data for the composable enterprise b...
apidays LIVE Australia 2021 - Composable data for the composable enterprise b...
Streaming Data and Stream Processing with Apache Kafka
Streaming Data and Stream Processing with Apache Kafka
Application Networks: Microservices and APIs at Netflix
Application Networks: Microservices and APIs at Netflix
apidays LIVE Australia 2021 - A cloud-native approach for open banking in act...
apidays LIVE Australia 2021 - A cloud-native approach for open banking in act...
Cloud computing and health care - Facing the Future
Cloud computing and health care - Facing the Future
132177_16x9_GlobalAlliances_Presentation_NetApp_01_D[1]
132177_16x9_GlobalAlliances_Presentation_NetApp_01_D[1]
SAP Forum Ankara 2017 - "Verinin Merkezine Seyahat"
SAP Forum Ankara 2017 - "Verinin Merkezine Seyahat"
apidays Australia 2023 - The Playful Bond Between REST And Data Streams, Warr...
apidays Australia 2023 - The Playful Bond Between REST And Data Streams, Warr...
Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Curren...
Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Curren...
Future of Power: Power Strategy and Offerings for Denmark - Steve Sibley
Future of Power: Power Strategy and Offerings for Denmark - Steve Sibley
More from HostedbyConfluent
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way.
Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks.
Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case.
Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."
Renaming a Kafka Topic | Kafka Summit London
"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics.
Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"
Evolution of NRT Data Ingestion Pipeline at Trendyol
"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights.
We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string.
Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring.
The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service.
While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications.
The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection.
Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."
Exactly-once Stream Processing with Arroyo and Kafka
"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output.
Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing.
In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."
Fish Plays Pokemon | Kafka Summit London
"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium.
This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption.
By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."
Tiered Storage 101 | Kafla Summit London
What is tiered storage and what is it good for? After this session you will know how to leverage the tiered storage feature to enable longer retention than the storage attached to brokers allows. You will get acquainted with the different configuration options and know what to expect when you enable the feature, like for example when will the first upload to the remote object storage take place.
Building a Self-Service Stream Processing Portal: How And Why
"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work.
However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business.
You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want.
Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware.
What you’ll take away from our talk:
- What were the pain points in the previous environment
- How we transitioned to Confluent without service downtime
- Creating a self-service stream processing portal built on top of Connect and ksqlDB
- Use case of stream process portal"
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics.
Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application.
At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."
Navigating Private Network Connectivity Options for Kafka Clusters
"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option!
In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met.
I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines.
Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"
Explaining How Real-Time GenAI Works in a Noisy Pub
"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many.
LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data.
Large language models for your streaming application - explained with a little maths and a lot of pub stories"
TL;DR Kafka Metrics | Kafka Summit London
"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them.
This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."
A Window Into Your Kafka Streams Tasks | KSL
Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance.
- Introduction to Kafka producer internal and workflow
- Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance
- Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient.
- Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees.
- Q&A session with attendees to address specific questions and concerns."
Data Contracts Management: Schema Registry and Beyond
"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element.
In this talk, we’ll:
1. see why data contracts are so important but also difficult to implement;
2. identify the characteristics of a well-designed data contract:
discuss the anatomy of a data contract, its main elements and, how to formally describe them;
3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."
Code-First Approach: Crafting Efficient Flink Apps
"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications.
We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications.
Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."
Debezium vs. the World: An Overview of the CDC Ecosystem
"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you!
[1] https://debezium.io/"
Beyond Tiered Storage: Serverless Kafka with No Local Disks
"Separation of compute and storage has become the de-facto standard in the data industry for batch processing.
The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world.
In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks.
Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it.
This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."
More from HostedbyConfluent (20)
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Evolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at Trendyol
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Exactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and Kafka
Building a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And Why
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Navigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka Clusters
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Explaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy Pub
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Data Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and Beyond
Code-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink Apps
Debezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC Ecosystem
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Recently uploaded
Programming Foundation Models with DSPy - Meetup Slides
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Inconsistent user experience and siloed data, high costs, and changing customer expectations – Citizens Bank was experiencing these challenges while it was attempting to deliver a superior digital banking experience for its clients. Its core banking applications run on the mainframe and Citizens was using legacy utilities to get the critical mainframe data to feed customer-facing channels, like call centers, web, and mobile. Ultimately, this led to higher operating costs (MIPS), delayed response times, and longer time to market.
Ever-changing customer expectations demand more modern digital experiences, and the bank needed to find a solution that could provide real-time data to its customer channels with low latency and operating costs. Join this session to learn how Citizens is leveraging Precisely to replicate mainframe data to its customer channels and deliver on their “modern digital bank” experiences.
HCL Notes and Domino License Cost Reduction in the World of DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks creates AI-boosted business software that helps employees work more efficiently and effectively. Managing data across multiple RDBMS and NoSQL databases was already a challenge at their current scale. To prepare for 10X growth, they knew it was time to rethink their database strategy. Learn how they architected a solution that would simplify scaling while keeping costs under control.
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/temporal-event-neural-networks-a-more-efficient-alternative-to-the-transformer-a-presentation-from-brainchip/
Chris Jones, Director of Product Management at BrainChip , presents the “Temporal Event Neural Networks: A More Efficient Alternative to the Transformer” tutorial at the May 2024 Embedded Vision Summit.
The expansion of AI services necessitates enhanced computational capabilities on edge devices. Temporal Event Neural Networks (TENNs), developed by BrainChip, represent a novel and highly efficient state-space network. TENNs demonstrate exceptional proficiency in handling multi-dimensional streaming data, facilitating advancements in object detection, action recognition, speech enhancement and language model/sequence generation. Through the utilization of polynomial-based continuous convolutions, TENNs streamline models, expedite training processes and significantly diminish memory requirements, achieving notable reductions of up to 50x in parameters and 5,000x in energy consumption compared to prevailing methodologies like transformers.
Integration with BrainChip’s Akida neuromorphic hardware IP further enhances TENNs’ capabilities, enabling the realization of highly capable, portable and passively cooled edge devices. This presentation delves into the technical innovations underlying TENNs, presents real-world benchmarks, and elucidates how this cutting-edge approach is positioned to revolutionize edge AI across diverse applications.Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
JavaLand 2024: Application Development Green Masterplan
My presentation slides I used at JavaLand 2024
GNSS spoofing via SDR (Criptored Talks 2024)
In the realm of cybersecurity, offensive security practices act as a critical shield. By simulating real-world attacks in a controlled environment, these techniques expose vulnerabilities before malicious actors can exploit them. This proactive approach allows manufacturers to identify and fix weaknesses, significantly enhancing system security.
This presentation delves into the development of a system designed to mimic Galileo's Open Service signal using software-defined radio (SDR) technology. We'll begin with a foundational overview of both Global Navigation Satellite Systems (GNSS) and the intricacies of digital signal processing.
The presentation culminates in a live demonstration. We'll showcase the manipulation of Galileo's Open Service pilot signal, simulating an attack on various software and hardware systems. This practical demonstration serves to highlight the potential consequences of unaddressed vulnerabilities, emphasizing the importance of offensive security practices in safeguarding critical infrastructure.
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Folding is a recent technique for building efficient recursive SNARKs. Several elegant folding protocols have been proposed, such as Nova, Supernova, Hypernova, Protostar, and others. However, all of them rely on an additively homomorphic commitment scheme based on discrete log, and are therefore not post-quantum secure. In this work we present LatticeFold, the first lattice-based folding protocol based on the Module SIS problem. This folding protocol naturally leads to an efficient recursive lattice-based SNARK and an efficient PCD scheme. LatticeFold supports folding low-degree relations, such as R1CS, as well as high-degree relations, such as CCS. The key challenge is to construct a secure folding protocol that works with the Ajtai commitment scheme. The difficulty, is ensuring that extracted witnesses are low norm through many rounds of folding. We present a novel technique using the sumcheck protocol to ensure that extracted witnesses are always low norm no matter how many rounds of folding are used. Our evaluation of the final proof system suggests that it is as performant as Hypernova, while providing post-quantum security.
Paper Link: https://eprint.iacr.org/2024/257
FREE A4 Cyber Security Awareness Posters-Social Engineering part 3
Free A4 downloadable and printable Cyber Security, Social Engineering Safety and security Training Posters . Promote security awareness in the home or workplace. Lock them Out From training providers datahops.com
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Tomaz Bratanic
Graph ML and GenAI Expert - Neo4j
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Recently uploaded (20)
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
FREE A4 Cyber Security Awareness Posters-Social Engineering part 3
FREE A4 Cyber Security Awareness Posters-Social Engineering part 3
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Developing custom transformation in the Kafka connect to minimize data redundancy | Siavash Sedghi, United Health Group
- 1. Optum Technology Provider Data Platform Siavash Sedghi Lead Software Engineer September 2021
- 2. Developing Custom Transformations in the Kafka Connect to Minimize Data Redundancy Sep 2021 – Kafka Summit Americas
- 3. Our Mission To help people live healthier lives and to help make the health system work better for everyone.
- 4. Optum by numbers o Optum serves 127 million individual consumers o Optum serves 9 out of 10 U.S. hospitals o Optum serves ~9 out of 10 Fortune 100 employers o Optum serves more than 90,000 physicians, practices and other health care facilities as well as non-profit associations and organizations o Optum serves a network of over 67,000 pharmacies 4
- 6. The Journey Begins Data ingestion from multiple resources (Kafka Connect) Data enrichment (KTable Joins & Streaming API) Aggregation and metrics calculation (Kafka Streaming API) Sinking data to database (Kafka Connect) Near real-time APIs to serve the data 6
- 7. Broker Instance Fee Broker Storage Fee Data Transfer Fee Cost of Operating a Kafka Cluster 7
- 8. Broker Instance Fee Broker Storage Fee Data Transfer Fee Cost of Operating a Kafka Cluster 8 Reducing Broker Storage Cost by Minimizing Data Redundancy in Topics
- 10. Example: 10
- 11. Example: 11
- 12. Example: 12
- 13. Example: 13
- 14. Example: 14
- 16. Implementing a Transformation 16 Add your Jar file to the base docker image
- 17. Implementing a Transformation 17 docker logs -f kafka-connect cat /var/log/kafka/connect.log
- 18. DEMO
- 20. Optum® is a registered trademark of Optum, Inc. in the U.S. and other jurisdictions. All other brand or product names are the property of their respective owners. Because we are continuously improving our products and services, Optum reserves the right to change specifications without prior notice. Optum is an equal opportunity employer. © 2021 Optum, Inc. All rights reserved.