SlideShare a Scribd company logo

Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evernorth Health Services

In this talk, we look at textbook examples of using kafka at scale. Specifically focused on Evernorth Health Service's journey of implementing microservices data pipelines, we provide an overview of the patterns we used while implementing CDC data pipelines for these micro services using confluent kafka, the challenges we faced, lessons learned and the unique solutions that we developed over the years to overcome those challenges. We also look at how we are moving these micro services eco systems to public cloud (AWS) and the strategies that we have/are implementing to ensure a smooth consumer cutover. We peek into how kafka consumers can cutover to a replicated topic using offsets based on create time timestamps and how/why this is critical for a downtime-less and data-loss-less consumer cutover for streaming consumers. To conclude, we take a look at how we are re-imagining these pipelines on AWS and how SaaS offerings like Confluent Cloud and Confluent connectors could play a major role.

1 of 15
Download to read offline
© 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 1
Using Kafka at Scale – A Case Study
of Micro Services Data Pipelines at
Evernorth Health Services
Nilay Sundarkar
© 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 2
© 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 3
Agenda
•Evernorth – Microservices
CDC Data Pipelines journey
PAST
•Moving to public cloud -
AWS
PRESENT
•Reimagining the CDC
pipeline on AWS
FUTURE
© 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 4
Background
Mainframes
Entity A
Entity B
Entity C Entity A
Entity B
Entity C
Legacy Integration Approach Strategic Integration Approach
Qlik
© 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 5
Typical Entity Data Pipeline
Transform Legacy to
Entity Format
State Management,
ResourceId
generation
Qlik
Source
Topic
Source
Topic
Source
Topic
Transform Update
Topic
Update
Entity Data
Store
Change
Streams
Entity API
Mainframes
Legacy Data
Store
Data Pipeline
Entity
Legacy
Entity
Stream
Qlik Kafka
Spring
Boot
PCF Mongo
Tech Stack
© 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 6
Aggregation and Enrichment
Transform Legacy to
Entity Format
State Management,
ResourceId
generation
Qlik
Source
Topic
Source
Topic
Source
Topic
Transform Update
Topic
Update
Entity Data
Store
Change
Streams
Entity API
Mainframes
Legacy Data
Store
Data Pipeline
Entity
Legacy
Entity
Stream
Aggregation
Join with another
table for a
stream
Windowing Local Lookup
Enrichment
API or DB
Lookup

Recommended

How To Track Performance and Fault in a Multi-layer, Software-Defined Network...
How To Track Performance and Fault in a Multi-layer, Software-Defined Network...How To Track Performance and Fault in a Multi-layer, Software-Defined Network...
How To Track Performance and Fault in a Multi-layer, Software-Defined Network...CA Technologies
 
See It Live - Single Pane of Glass Enterprise Monitoring with CA Unified Infr...
See It Live - Single Pane of Glass Enterprise Monitoring with CA Unified Infr...See It Live - Single Pane of Glass Enterprise Monitoring with CA Unified Infr...
See It Live - Single Pane of Glass Enterprise Monitoring with CA Unified Infr...CA Technologies
 
Full Keynote Session- AWS Public Sector Summit Singapore 2017
Full Keynote Session- AWS Public Sector Summit Singapore 2017Full Keynote Session- AWS Public Sector Summit Singapore 2017
Full Keynote Session- AWS Public Sector Summit Singapore 2017Amazon Web Services
 
Balance agility and governance with #TrueDataOps and The Data Cloud
Balance agility and governance with #TrueDataOps and The Data CloudBalance agility and governance with #TrueDataOps and The Data Cloud
Balance agility and governance with #TrueDataOps and The Data CloudKent Graziano
 
Leveraging CA Technologies Application Development Solutions to Rapidly Deliv...
Leveraging CA Technologies Application Development Solutions to Rapidly Deliv...Leveraging CA Technologies Application Development Solutions to Rapidly Deliv...
Leveraging CA Technologies Application Development Solutions to Rapidly Deliv...CA Technologies
 
SAP Hack2Build hackathon - SAP Commerce Cloud & Kyma runtime
SAP Hack2Build hackathon - SAP Commerce Cloud & Kyma runtimeSAP Hack2Build hackathon - SAP Commerce Cloud & Kyma runtime
SAP Hack2Build hackathon - SAP Commerce Cloud & Kyma runtimeSAP HANA Cloud Platform
 
Speed time to value with CA IAM solutions deployed on Docker, Kubernetes, and...
Speed time to value with CA IAM solutions deployed on Docker, Kubernetes, and...Speed time to value with CA IAM solutions deployed on Docker, Kubernetes, and...
Speed time to value with CA IAM solutions deployed on Docker, Kubernetes, and...CA Technologies
 
Technology Primer: New Cloud Monitoring Capabilities in CA Unified Infrastruc...
Technology Primer: New Cloud Monitoring Capabilities in CA Unified Infrastruc...Technology Primer: New Cloud Monitoring Capabilities in CA Unified Infrastruc...
Technology Primer: New Cloud Monitoring Capabilities in CA Unified Infrastruc...CA Technologies
 

More Related Content

Similar to Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evernorth Health Services

CA Gen Exploration – What's New and Cool in Application Development
CA Gen Exploration – What's New and Cool in Application DevelopmentCA Gen Exploration – What's New and Cool in Application Development
CA Gen Exploration – What's New and Cool in Application DevelopmentCA Technologies
 
VMworld_PivotalCF_And_Containers
VMworld_PivotalCF_And_Containers VMworld_PivotalCF_And_Containers
VMworld_PivotalCF_And_Containers James Watters
 
Tech Talk: CA Workload Automation Agent Monitor, Agents and Advanced Integrat...
Tech Talk: CA Workload Automation Agent Monitor, Agents and Advanced Integrat...Tech Talk: CA Workload Automation Agent Monitor, Agents and Advanced Integrat...
Tech Talk: CA Workload Automation Agent Monitor, Agents and Advanced Integrat...CA Technologies
 
Hands-On Lab: Integrate Your Monitoring Tools into an Automated Service Impac...
Hands-On Lab: Integrate Your Monitoring Tools into an Automated Service Impac...Hands-On Lab: Integrate Your Monitoring Tools into an Automated Service Impac...
Hands-On Lab: Integrate Your Monitoring Tools into an Automated Service Impac...CA Technologies
 
Hands-On Lab: Test Drive the Enhanced HTML5 Dashboard Designer With Release 8...
Hands-On Lab: Test Drive the Enhanced HTML5 Dashboard Designer With Release 8...Hands-On Lab: Test Drive the Enhanced HTML5 Dashboard Designer With Release 8...
Hands-On Lab: Test Drive the Enhanced HTML5 Dashboard Designer With Release 8...CA Technologies
 
Hands-On Lab: Smart Instrumentation
Hands-On Lab: Smart InstrumentationHands-On Lab: Smart Instrumentation
Hands-On Lab: Smart InstrumentationCA Technologies
 
CA World 2015 Using Docker and Containers to create Bespoke Systems Under Test
CA World 2015 Using Docker and Containers to create Bespoke Systems Under Test CA World 2015 Using Docker and Containers to create Bespoke Systems Under Test
CA World 2015 Using Docker and Containers to create Bespoke Systems Under Test Ian Kelly
 
apidays LIVE New York 2021 - Simplify Open Policy Agent with Styra DAS by Tim...
apidays LIVE New York 2021 - Simplify Open Policy Agent with Styra DAS by Tim...apidays LIVE New York 2021 - Simplify Open Policy Agent with Styra DAS by Tim...
apidays LIVE New York 2021 - Simplify Open Policy Agent with Styra DAS by Tim...apidays
 
Using Artificial Intelligence to power Service Virtualization
Using Artificial Intelligence to power Service VirtualizationUsing Artificial Intelligence to power Service Virtualization
Using Artificial Intelligence to power Service VirtualizationCA Technologies
 
Using OpenContrail with Kubernetes
Using OpenContrail with KubernetesUsing OpenContrail with Kubernetes
Using OpenContrail with KubernetesMatt Baldwin
 
Tech Talk: Self-Certify New Devices with the CA Unified Infrastructure Manage...
Tech Talk: Self-Certify New Devices with the CA Unified Infrastructure Manage...Tech Talk: Self-Certify New Devices with the CA Unified Infrastructure Manage...
Tech Talk: Self-Certify New Devices with the CA Unified Infrastructure Manage...CA Technologies
 
Digital Supply Chains & the Internet of Things
Digital Supply Chains & the Internet of ThingsDigital Supply Chains & the Internet of Things
Digital Supply Chains & the Internet of ThingsKevin Ross
 
Technology Primer: Software-Defined Networking and Its Impact on Infrastructu...
Technology Primer: Software-Defined Networking and Its Impact on Infrastructu...Technology Primer: Software-Defined Networking and Its Impact on Infrastructu...
Technology Primer: Software-Defined Networking and Its Impact on Infrastructu...CA Technologies
 
Technology Primer: Building Applications the New-Fashioned Way
Technology Primer: Building Applications the New-Fashioned WayTechnology Primer: Building Applications the New-Fashioned Way
Technology Primer: Building Applications the New-Fashioned WayCA Technologies
 
Hands-On Labs: Introduction to CA Unified Infrastructure Management
Hands-On Labs: Introduction to CA Unified Infrastructure Management Hands-On Labs: Introduction to CA Unified Infrastructure Management
Hands-On Labs: Introduction to CA Unified Infrastructure Management CA Technologies
 
Splunk conf2014 - Getting Deeper Insights into your Virtualization and Storag...
Splunk conf2014 - Getting Deeper Insights into your Virtualization and Storag...Splunk conf2014 - Getting Deeper Insights into your Virtualization and Storag...
Splunk conf2014 - Getting Deeper Insights into your Virtualization and Storag...Splunk
 
Case Study: Intel Corporation - The Benefits of, and Need for Agile Operation...
Case Study: Intel Corporation - The Benefits of, and Need for Agile Operation...Case Study: Intel Corporation - The Benefits of, and Need for Agile Operation...
Case Study: Intel Corporation - The Benefits of, and Need for Agile Operation...CA Technologies
 
Technology Primer: Monitor Microservices, Containers, Cloud Foundry and Node ...
Technology Primer: Monitor Microservices, Containers, Cloud Foundry and Node ...Technology Primer: Monitor Microservices, Containers, Cloud Foundry and Node ...
Technology Primer: Monitor Microservices, Containers, Cloud Foundry and Node ...CA Technologies
 

Similar to Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evernorth Health Services (20)

Enterprise Transformation
Enterprise TransformationEnterprise Transformation
Enterprise Transformation
 
CA Gen Exploration – What's New and Cool in Application Development
CA Gen Exploration – What's New and Cool in Application DevelopmentCA Gen Exploration – What's New and Cool in Application Development
CA Gen Exploration – What's New and Cool in Application Development
 
VMworld_PivotalCF_And_Containers
VMworld_PivotalCF_And_Containers VMworld_PivotalCF_And_Containers
VMworld_PivotalCF_And_Containers
 
Tech Talk: CA Workload Automation Agent Monitor, Agents and Advanced Integrat...
Tech Talk: CA Workload Automation Agent Monitor, Agents and Advanced Integrat...Tech Talk: CA Workload Automation Agent Monitor, Agents and Advanced Integrat...
Tech Talk: CA Workload Automation Agent Monitor, Agents and Advanced Integrat...
 
Hands-On Lab: Integrate Your Monitoring Tools into an Automated Service Impac...
Hands-On Lab: Integrate Your Monitoring Tools into an Automated Service Impac...Hands-On Lab: Integrate Your Monitoring Tools into an Automated Service Impac...
Hands-On Lab: Integrate Your Monitoring Tools into an Automated Service Impac...
 
Hands-On Lab: Test Drive the Enhanced HTML5 Dashboard Designer With Release 8...
Hands-On Lab: Test Drive the Enhanced HTML5 Dashboard Designer With Release 8...Hands-On Lab: Test Drive the Enhanced HTML5 Dashboard Designer With Release 8...
Hands-On Lab: Test Drive the Enhanced HTML5 Dashboard Designer With Release 8...
 
Hands-On Lab: Smart Instrumentation
Hands-On Lab: Smart InstrumentationHands-On Lab: Smart Instrumentation
Hands-On Lab: Smart Instrumentation
 
CA World 2015 Using Docker and Containers to create Bespoke Systems Under Test
CA World 2015 Using Docker and Containers to create Bespoke Systems Under Test CA World 2015 Using Docker and Containers to create Bespoke Systems Under Test
CA World 2015 Using Docker and Containers to create Bespoke Systems Under Test
 
apidays LIVE New York 2021 - Simplify Open Policy Agent with Styra DAS by Tim...
apidays LIVE New York 2021 - Simplify Open Policy Agent with Styra DAS by Tim...apidays LIVE New York 2021 - Simplify Open Policy Agent with Styra DAS by Tim...
apidays LIVE New York 2021 - Simplify Open Policy Agent with Styra DAS by Tim...
 
Using Artificial Intelligence to power Service Virtualization
Using Artificial Intelligence to power Service VirtualizationUsing Artificial Intelligence to power Service Virtualization
Using Artificial Intelligence to power Service Virtualization
 
Using OpenContrail with Kubernetes
Using OpenContrail with KubernetesUsing OpenContrail with Kubernetes
Using OpenContrail with Kubernetes
 
Tech Talk: Self-Certify New Devices with the CA Unified Infrastructure Manage...
Tech Talk: Self-Certify New Devices with the CA Unified Infrastructure Manage...Tech Talk: Self-Certify New Devices with the CA Unified Infrastructure Manage...
Tech Talk: Self-Certify New Devices with the CA Unified Infrastructure Manage...
 
Digital Supply Chains & the Internet of Things
Digital Supply Chains & the Internet of ThingsDigital Supply Chains & the Internet of Things
Digital Supply Chains & the Internet of Things
 
Technology Primer: Software-Defined Networking and Its Impact on Infrastructu...
Technology Primer: Software-Defined Networking and Its Impact on Infrastructu...Technology Primer: Software-Defined Networking and Its Impact on Infrastructu...
Technology Primer: Software-Defined Networking and Its Impact on Infrastructu...
 
Technology Primer: Building Applications the New-Fashioned Way
Technology Primer: Building Applications the New-Fashioned WayTechnology Primer: Building Applications the New-Fashioned Way
Technology Primer: Building Applications the New-Fashioned Way
 
Hands-On Labs: Introduction to CA Unified Infrastructure Management
Hands-On Labs: Introduction to CA Unified Infrastructure Management Hands-On Labs: Introduction to CA Unified Infrastructure Management
Hands-On Labs: Introduction to CA Unified Infrastructure Management
 
Splunk conf2014 - Getting Deeper Insights into your Virtualization and Storag...
Splunk conf2014 - Getting Deeper Insights into your Virtualization and Storag...Splunk conf2014 - Getting Deeper Insights into your Virtualization and Storag...
Splunk conf2014 - Getting Deeper Insights into your Virtualization and Storag...
 
SnappyFlow Presentation.pdf
SnappyFlow Presentation.pdfSnappyFlow Presentation.pdf
SnappyFlow Presentation.pdf
 
Case Study: Intel Corporation - The Benefits of, and Need for Agile Operation...
Case Study: Intel Corporation - The Benefits of, and Need for Agile Operation...Case Study: Intel Corporation - The Benefits of, and Need for Agile Operation...
Case Study: Intel Corporation - The Benefits of, and Need for Agile Operation...
 
Technology Primer: Monitor Microservices, Containers, Cloud Foundry and Node ...
Technology Primer: Monitor Microservices, Containers, Cloud Foundry and Node ...Technology Primer: Monitor Microservices, Containers, Cloud Foundry and Node ...
Technology Primer: Monitor Microservices, Containers, Cloud Foundry and Node ...
 

More from HostedbyConfluent

Build Real-time Machine Learning Apps on Generative AI with Kafka Streams
Build Real-time Machine Learning Apps on Generative AI with Kafka StreamsBuild Real-time Machine Learning Apps on Generative AI with Kafka Streams
Build Real-time Machine Learning Apps on Generative AI with Kafka StreamsHostedbyConfluent
 
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...HostedbyConfluent
 
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...HostedbyConfluent
 
Rule Based Asset Management Workflow Automation at Netflix
Rule Based Asset Management Workflow Automation at NetflixRule Based Asset Management Workflow Automation at Netflix
Rule Based Asset Management Workflow Automation at NetflixHostedbyConfluent
 
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...HostedbyConfluent
 
Indeed Flex: The Story of a Revolutionary Recruitment Platform
Indeed Flex: The Story of a Revolutionary Recruitment PlatformIndeed Flex: The Story of a Revolutionary Recruitment Platform
Indeed Flex: The Story of a Revolutionary Recruitment PlatformHostedbyConfluent
 
Forecasting Kafka Lag Issues with Machine Learning
Forecasting Kafka Lag Issues with Machine LearningForecasting Kafka Lag Issues with Machine Learning
Forecasting Kafka Lag Issues with Machine LearningHostedbyConfluent
 
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...HostedbyConfluent
 
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...HostedbyConfluent
 
Accelerating Path to Production for Generative AI-powered Applications
Accelerating Path to Production for Generative AI-powered ApplicationsAccelerating Path to Production for Generative AI-powered Applications
Accelerating Path to Production for Generative AI-powered ApplicationsHostedbyConfluent
 
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...HostedbyConfluent
 
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...HostedbyConfluent
 
Go Big or Go Home: Approaching Kafka Replication at Scale
Go Big or Go Home: Approaching Kafka Replication at ScaleGo Big or Go Home: Approaching Kafka Replication at Scale
Go Big or Go Home: Approaching Kafka Replication at ScaleHostedbyConfluent
 
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2HostedbyConfluent
 
A Trifecta of Real-Time Applications: Apache Kafka, Flink, and Druid
A Trifecta of Real-Time Applications: Apache Kafka, Flink, and DruidA Trifecta of Real-Time Applications: Apache Kafka, Flink, and Druid
A Trifecta of Real-Time Applications: Apache Kafka, Flink, and DruidHostedbyConfluent
 
From Raw Data to an Interactive Data App in an Hour: Powered by Snowpark Python
From Raw Data to an Interactive Data App in an Hour: Powered by Snowpark PythonFrom Raw Data to an Interactive Data App in an Hour: Powered by Snowpark Python
From Raw Data to an Interactive Data App in an Hour: Powered by Snowpark PythonHostedbyConfluent
 
Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Archite...
Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Archite...Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Archite...
Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Archite...HostedbyConfluent
 
Exactly-Once Semantics Revisited: Distributed Transactions across Flink and K...
Exactly-Once Semantics Revisited: Distributed Transactions across Flink and K...Exactly-Once Semantics Revisited: Distributed Transactions across Flink and K...
Exactly-Once Semantics Revisited: Distributed Transactions across Flink and K...HostedbyConfluent
 
From the Battlefield: Squeezing the Most From Your Kafka Infrastructure
From the Battlefield: Squeezing the Most From Your Kafka InfrastructureFrom the Battlefield: Squeezing the Most From Your Kafka Infrastructure
From the Battlefield: Squeezing the Most From Your Kafka InfrastructureHostedbyConfluent
 

More from HostedbyConfluent (20)

Build Real-time Machine Learning Apps on Generative AI with Kafka Streams
Build Real-time Machine Learning Apps on Generative AI with Kafka StreamsBuild Real-time Machine Learning Apps on Generative AI with Kafka Streams
Build Real-time Machine Learning Apps on Generative AI with Kafka Streams
 
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...
 
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...
 
Rule Based Asset Management Workflow Automation at Netflix
Rule Based Asset Management Workflow Automation at NetflixRule Based Asset Management Workflow Automation at Netflix
Rule Based Asset Management Workflow Automation at Netflix
 
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...
 
Indeed Flex: The Story of a Revolutionary Recruitment Platform
Indeed Flex: The Story of a Revolutionary Recruitment PlatformIndeed Flex: The Story of a Revolutionary Recruitment Platform
Indeed Flex: The Story of a Revolutionary Recruitment Platform
 
Forecasting Kafka Lag Issues with Machine Learning
Forecasting Kafka Lag Issues with Machine LearningForecasting Kafka Lag Issues with Machine Learning
Forecasting Kafka Lag Issues with Machine Learning
 
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...
 
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...
 
Accelerating Path to Production for Generative AI-powered Applications
Accelerating Path to Production for Generative AI-powered ApplicationsAccelerating Path to Production for Generative AI-powered Applications
Accelerating Path to Production for Generative AI-powered Applications
 
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...
 
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...
 
Streaming is a Detail
Streaming is a DetailStreaming is a Detail
Streaming is a Detail
 
Go Big or Go Home: Approaching Kafka Replication at Scale
Go Big or Go Home: Approaching Kafka Replication at ScaleGo Big or Go Home: Approaching Kafka Replication at Scale
Go Big or Go Home: Approaching Kafka Replication at Scale
 
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2
 
A Trifecta of Real-Time Applications: Apache Kafka, Flink, and Druid
A Trifecta of Real-Time Applications: Apache Kafka, Flink, and DruidA Trifecta of Real-Time Applications: Apache Kafka, Flink, and Druid
A Trifecta of Real-Time Applications: Apache Kafka, Flink, and Druid
 
From Raw Data to an Interactive Data App in an Hour: Powered by Snowpark Python
From Raw Data to an Interactive Data App in an Hour: Powered by Snowpark PythonFrom Raw Data to an Interactive Data App in an Hour: Powered by Snowpark Python
From Raw Data to an Interactive Data App in an Hour: Powered by Snowpark Python
 
Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Archite...
Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Archite...Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Archite...
Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Archite...
 
Exactly-Once Semantics Revisited: Distributed Transactions across Flink and K...
Exactly-Once Semantics Revisited: Distributed Transactions across Flink and K...Exactly-Once Semantics Revisited: Distributed Transactions across Flink and K...
Exactly-Once Semantics Revisited: Distributed Transactions across Flink and K...
 
From the Battlefield: Squeezing the Most From Your Kafka Infrastructure
From the Battlefield: Squeezing the Most From Your Kafka InfrastructureFrom the Battlefield: Squeezing the Most From Your Kafka Infrastructure
From the Battlefield: Squeezing the Most From Your Kafka Infrastructure
 

Recently uploaded

IT Nation Evolve event 2024 - Quarter 1
IT Nation Evolve event 2024  - Quarter 1IT Nation Evolve event 2024  - Quarter 1
IT Nation Evolve event 2024 - Quarter 1Inbay UK
 
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Umar Saif
 
iOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostingeriOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostingerssuser9354ce
 
Artificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeArtificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeJosh Gellers
 
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...ShapeBlue
 
Large Language Models and Applications in Healthcare
Large Language Models and Applications in HealthcareLarge Language Models and Applications in Healthcare
Large Language Models and Applications in HealthcareAsma Ben Abacha
 
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlueCloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlueShapeBlue
 
TrustArc Webinar - TrustArc's Latest AI Innovations
TrustArc Webinar - TrustArc's Latest AI InnovationsTrustArc Webinar - TrustArc's Latest AI Innovations
TrustArc Webinar - TrustArc's Latest AI InnovationsTrustArc
 
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31shyamraj55
 
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...htrindia
 
Confoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceConfoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceSusan Ibach
 
ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...
ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...
ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...SearchNorwich
 
Enterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewEnterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewAshraf Fouad
 
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...UiPathCommunity
 
Transcript: Trending now: Book subjects on the move in the Canadian market - ...
Transcript: Trending now: Book subjects on the move in the Canadian market - ...Transcript: Trending now: Book subjects on the move in the Canadian market - ...
Transcript: Trending now: Book subjects on the move in the Canadian market - ...BookNet Canada
 
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoRevolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoProduct School
 
National Institute of Standards and Technology (NIST) Cybersecurity Framework...
National Institute of Standards and Technology (NIST) Cybersecurity Framework...National Institute of Standards and Technology (NIST) Cybersecurity Framework...
National Institute of Standards and Technology (NIST) Cybersecurity Framework...MichaelBenis1
 
Learning About GenAI Engineering with AWS PartyRock [AWS User Group Basel - F...
Learning About GenAI Engineering with AWS PartyRock [AWS User Group Basel - F...Learning About GenAI Engineering with AWS PartyRock [AWS User Group Basel - F...
Learning About GenAI Engineering with AWS PartyRock [AWS User Group Basel - F...Chris Bingham
 
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueCloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueShapeBlue
 
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...DianaGray10
 

Recently uploaded (20)

IT Nation Evolve event 2024 - Quarter 1
IT Nation Evolve event 2024  - Quarter 1IT Nation Evolve event 2024  - Quarter 1
IT Nation Evolve event 2024 - Quarter 1
 
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
 
iOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostingeriOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostinger
 
Artificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeArtificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human Justice
 
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
 
Large Language Models and Applications in Healthcare
Large Language Models and Applications in HealthcareLarge Language Models and Applications in Healthcare
Large Language Models and Applications in Healthcare
 
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlueCloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
 
TrustArc Webinar - TrustArc's Latest AI Innovations
TrustArc Webinar - TrustArc's Latest AI InnovationsTrustArc Webinar - TrustArc's Latest AI Innovations
TrustArc Webinar - TrustArc's Latest AI Innovations
 
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
 
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
 
Confoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceConfoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data science
 
ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...
ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...
ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...
 
Enterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewEnterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book Review
 
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
 
Transcript: Trending now: Book subjects on the move in the Canadian market - ...
Transcript: Trending now: Book subjects on the move in the Canadian market - ...Transcript: Trending now: Book subjects on the move in the Canadian market - ...
Transcript: Trending now: Book subjects on the move in the Canadian market - ...
 
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoRevolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
 
National Institute of Standards and Technology (NIST) Cybersecurity Framework...
National Institute of Standards and Technology (NIST) Cybersecurity Framework...National Institute of Standards and Technology (NIST) Cybersecurity Framework...
National Institute of Standards and Technology (NIST) Cybersecurity Framework...
 
Learning About GenAI Engineering with AWS PartyRock [AWS User Group Basel - F...
Learning About GenAI Engineering with AWS PartyRock [AWS User Group Basel - F...Learning About GenAI Engineering with AWS PartyRock [AWS User Group Basel - F...
Learning About GenAI Engineering with AWS PartyRock [AWS User Group Basel - F...
 
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueCloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
 
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
 

Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evernorth Health Services

  • 1. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH. CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO AUTHORIZED PERSONNEL. 9/28/23 1 Using Kafka at Scale – A Case Study of Micro Services Data Pipelines at Evernorth Health Services Nilay Sundarkar
  • 2. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH. CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO AUTHORIZED PERSONNEL. 9/28/23 2
  • 3. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH. CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO AUTHORIZED PERSONNEL. 9/28/23 3 Agenda •Evernorth – Microservices CDC Data Pipelines journey PAST •Moving to public cloud - AWS PRESENT •Reimagining the CDC pipeline on AWS FUTURE
  • 4. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH. CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO AUTHORIZED PERSONNEL. 9/28/23 4 Background Mainframes Entity A Entity B Entity C Entity A Entity B Entity C Legacy Integration Approach Strategic Integration Approach Qlik
  • 5. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH. CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO AUTHORIZED PERSONNEL. 9/28/23 5 Typical Entity Data Pipeline Transform Legacy to Entity Format State Management, ResourceId generation Qlik Source Topic Source Topic Source Topic Transform Update Topic Update Entity Data Store Change Streams Entity API Mainframes Legacy Data Store Data Pipeline Entity Legacy Entity Stream Qlik Kafka Spring Boot PCF Mongo Tech Stack
  • 6. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH. CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO AUTHORIZED PERSONNEL. 9/28/23 6 Aggregation and Enrichment Transform Legacy to Entity Format State Management, ResourceId generation Qlik Source Topic Source Topic Source Topic Transform Update Topic Update Entity Data Store Change Streams Entity API Mainframes Legacy Data Store Data Pipeline Entity Legacy Entity Stream Aggregation Join with another table for a stream Windowing Local Lookup Enrichment API or DB Lookup
  • 7. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH. CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO AUTHORIZED PERSONNEL. 9/28/23 7 Bi Directional Writes Transform Legacy to Entity Format State Management, ResourceId generation Qlik Source Topic Source Topic Source Topic Transform Update Topic Update Entity Data Store Change Streams Entity API Mainframes Legacy Data Store Data Pipeline Entity Legacy Entity Stream Bi Directional Sync Write Back Adapter Multi Master Check timestamps Consumer Segregation Big Bang
  • 8. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH. CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO AUTHORIZED PERSONNEL. 9/28/23 8 Initial Loads & Backfills Transform Legacy to Entity Format State Management, ResourceId generation Qlik Source Topic Source Topic Source Topic Transform Update Topic Update Entity Data Store Change Streams Entity API Mainframes Legacy Data Store Data Pipeline Entity Legacy Entity Stream Seeding Data Using the pipeline Custom Micro Batching REFRESH Data Loader READ WRITE Check timestamps Simultaneous CDC and Load
  • 9. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH. CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO AUTHORIZED PERSONNEL. 9/28/23 9 Reliability and Observability Transform Legacy to Entity Format State Management, ResourceId generation Qlik Source Topic Source Topic Source Topic Transform Update Topic Update Entity Data Store Change Streams Entity API Mainframes Legacy Data Store Data Pipeline Entity Legacy Entity Stream Reconciliation Data Check Count Check Event Check Error Handling/Retries Reconciliation READ READ Kafka retries Error/retry topics Observability New Relic Control Center Error/Retry
  • 10. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH. CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO AUTHORIZED PERSONNEL. 9/28/23 10 AWS Re-platform Qlik Source Topic Transform Update Topic Update Entity Data Store Change Streams Entity API Mainframes Legacy Data Store Legacy Entity Stream Entity Data Store Source Topic Entity Stream Mongo Mirror Replicator Replicator AWS
  • 11. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH. CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO AUTHORIZED PERSONNEL. 9/28/23 11 Resetting offsets by timestamp . Consumer ConsumerGroup = C1 8000 2000 10000 ConsumerGroup = C1 earliest latest 8000 (msg) ts = 1 Offset = 23 8000 (msg) ts = 1 Offset = 10 we need to be somewhere here Kafka Broker Kafka Broker Reasons for offset unreliability 1. Garbage collection on source before replication. 2. Retries on replicator. 3. Lag in replicating consumer offset topic A B Steps to reset offsets by createTime timestamps 1. Retrieve offsets for C1 for all partitions on topic A 2. Retrieve corresponding timestamps for all partitions on topic A 3. Set offsets on topic B for all partitions using timestamps retrieved in step 2. Can consumer start from where it left?
  • 12. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH. CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO AUTHORIZED PERSONNEL. 9/28/23 12 Limitations Transform Legacy to Entity Format State Management, ResourceId generation Qlik Source Topic Source Topic Source Topic Transform Update Topic Update Entity Data Store Change Streams Entity API Mainframes Legacy Data Store Data Pipeline Entity Legacy Entity Stream Time to Market Lack of Self Service Need for components More focus on domain logic than integration logic
  • 13. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH. CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO AUTHORIZED PERSONNEL. 9/28/23 13 Reimagining the pipeline on public cloud Qlik Mainframes Legacy Data Store Legacy Automated Integration Logic Self Managed Infrastructure Pluggable/Extensible components Confluent Connector Confluent Connector Confluent Connector
  • 14. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH. CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO AUTHORIZED PERSONNEL. 9/28/23 14 • Follow EIP approach of separating integration logic from domain logic • Migration to cloud for streaming systems and data should be done incrementally. • Use of Confluent Cloud/Gitops for kafka, Confluent schema registry and confluent connectors. Key Takeaways
  • 15. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH. CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO AUTHORIZED PERSONNEL. 9/28/23 15 Thank You