In this talk, we look at textbook examples of using kafka at scale. Specifically focused on Evernorth Health Service's journey of implementing microservices data pipelines, we provide an overview of the patterns we used while implementing CDC data pipelines for these micro services using confluent kafka, the challenges we faced, lessons learned and the unique solutions that we developed over the years to overcome those challenges. We also look at how we are moving these micro services eco systems to public cloud (AWS) and the strategies that we have/are implementing to ensure a smooth consumer cutover. We peek into how kafka consumers can cutover to a replicated topic using offsets based on create time timestamps and how/why this is critical for a downtime-less and data-loss-less consumer cutover for streaming consumers. To conclude, we take a look at how we are re-imagining these pipelines on AWS and how SaaS offerings like Confluent Cloud and Confluent connectors could play a major role.

1. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 1
Using Kafka at Scale – A Case Study
of Micro Services Data Pipelines at
Evernorth Health Services
Nilay Sundarkar

2. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 2

3. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 3
Agenda
•Evernorth – Microservices
CDC Data Pipelines journey
PAST
•Moving to public cloud -
AWS
PRESENT
•Reimagining the CDC
pipeline on AWS
FUTURE

4. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 4
Background
Mainframes
Entity A
Entity B
Entity C Entity A
Entity B
Entity C
Legacy Integration Approach Strategic Integration Approach
Qlik

5. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 5
Typical Entity Data Pipeline
Transform Legacy to
Entity Format
State Management,
ResourceId
generation
Qlik
Source
Topic
Source
Topic
Source
Topic
Transform Update
Topic
Update
Entity Data
Store
Change
Streams
Entity API
Mainframes
Legacy Data
Store
Data Pipeline
Entity
Legacy
Entity
Stream
Qlik Kafka
Spring
Boot
PCF Mongo
Tech Stack

6. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 6
Aggregation and Enrichment
Transform Legacy to
Entity Format
State Management,
ResourceId
generation
Qlik
Source
Topic
Source
Topic
Source
Topic
Transform Update
Topic
Update
Entity Data
Store
Change
Streams
Entity API
Mainframes
Legacy Data
Store
Data Pipeline
Entity
Legacy
Entity
Stream
Aggregation
Join with another
table for a
stream
Windowing Local Lookup
Enrichment
API or DB
Lookup

7. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 7
Bi Directional Writes
Transform Legacy to
Entity Format
State Management,
ResourceId
generation
Qlik
Source
Topic
Source
Topic
Source
Topic
Transform Update
Topic
Update
Entity Data
Store
Change
Streams
Entity API
Mainframes
Legacy Data
Store
Data Pipeline
Entity
Legacy
Entity
Stream
Bi Directional Sync
Write Back Adapter
Multi Master
Check timestamps
Consumer Segregation
Big Bang

8. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 8
Initial Loads & Backfills
Transform Legacy to
Entity Format
State Management,
ResourceId
generation
Qlik
Source
Topic
Source
Topic
Source
Topic
Transform Update
Topic
Update
Entity Data
Store
Change
Streams
Entity API
Mainframes
Legacy Data
Store
Data Pipeline
Entity
Legacy
Entity
Stream
Seeding Data
Using the pipeline Custom Micro Batching
REFRESH
Data Loader
READ
WRITE
Check timestamps
Simultaneous CDC and Load

9. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 9
Reliability and Observability
Transform Legacy to
Entity Format
State Management,
ResourceId
generation
Qlik
Source
Topic
Source
Topic
Source
Topic
Transform Update
Topic
Update
Entity Data
Store
Change
Streams
Entity API
Mainframes
Legacy Data
Store
Data Pipeline
Entity
Legacy
Entity
Stream
Reconciliation
Data Check
Count Check
Event Check
Error Handling/Retries
Reconciliation
READ
READ
Kafka retries
Error/retry
topics
Observability
New Relic
Control Center
Error/Retry

10. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 10
AWS Re-platform
Qlik
Source
Topic
Transform Update
Topic
Update
Entity Data
Store
Change
Streams
Entity API
Mainframes
Legacy Data
Store
Legacy
Entity
Stream
Entity Data
Store
Source
Topic Entity
Stream
Mongo
Mirror
Replicator
Replicator
AWS

11. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 11
Resetting offsets by timestamp
.
Consumer
ConsumerGroup = C1
8000 2000 10000
ConsumerGroup = C1 earliest latest
8000
(msg)
ts = 1 Offset = 23
8000
(msg)
ts = 1 Offset = 10
we need to be
somewhere here
Kafka Broker Kafka Broker Reasons for offset unreliability
1. Garbage collection on source
before replication.
2. Retries on replicator.
3. Lag in replicating consumer
offset topic
A B
Steps to reset offsets by createTime
timestamps
1. Retrieve offsets for C1 for all
partitions on topic A
2. Retrieve corresponding
timestamps for all partitions on topic
A
3. Set offsets on topic B for all
partitions using timestamps
retrieved in step 2.
Can consumer start
from where it left?

12. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 12
Limitations
Transform Legacy to
Entity Format
State Management,
ResourceId
generation
Qlik
Source
Topic
Source
Topic
Source
Topic
Transform Update
Topic
Update
Entity Data
Store
Change
Streams
Entity API
Mainframes
Legacy Data
Store
Data Pipeline
Entity
Legacy
Entity
Stream
Time to Market Lack of Self Service
Need for
components
More focus on domain logic
than integration logic

13. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 13
Reimagining the pipeline on public cloud
Qlik
Mainframes
Legacy Data
Store
Legacy
Automated Integration Logic
Self Managed Infrastructure Pluggable/Extensible
components
Confluent
Connector
Confluent
Connector
Confluent
Connector

14. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 14
• Follow EIP approach of separating integration logic from domain
logic
• Migration to cloud for streaming systems and data should be done
incrementally.
• Use of Confluent Cloud/Gitops for kafka, Confluent schema registry
and confluent connectors.
Key Takeaways

15. © 2022 EVERNORTH. ALL RIGHTS RESERVED. ALL PRODUCTS AND SERVICES ARE
PROVIDED BY OR THROUGH OPERATING SUBSIDIARIES OR AFFILIATES OF EVERNORTH.
CONFIDENTIAL, UNPUBLISHED PROPERTY OF EVERNORTH. DO NOT DUPLICATE OR
DISTRIBUTE. FOR INTERNAL USE ONLY. USE AND DISTRIBUTION LIMITED SOLELY TO
AUTHORIZED PERSONNEL.
9/28/23 15
Thank You