SpringOne Platform Speakers: Dmitry Milman, Director, Platform Software Engineering, Express Scripts; Ankur Kaneria Youtube: https://www.youtube.com/watch?v=sPz3HNtgy8A&feature=youtu.be

1. From Mainframe to Microservices with
Pivotal Platform & Kafka
SpringOne Platform - October 10, 2019
Dmitry Milman & Ankur Kaneria - Express Scripts, a Cigna Company
Bridging the Data Divide

2. 2
© 2018 Express Scripts. All Rights Reserved.
Agenda
üCompany Background
üPrior Architecture
üMicroservice Architecture
üData Sync Platform
üKafka Libraries and Connectivity Options
üKafka Partitioning Strategies
üPerformance & Health Monitoring
üPerformance Optimization With Caching
üWhat’s Next

3. 3
© 2018 Express Scripts. All Rights Reserved.
What does Express Scripts do ?
üPart of Cigna Corporation
üJust like your medical plan covers visits to your doctor, your
Express Scripts prescription plan covers the medicine your
doctor prescribes
üWe’re one of the largest pharmacy benefits managers in the
United States and one of the country’s largest pharmacies
üServe more than 100 million people
üClaims Volume
üDaily ~6 million
üAnnual ~2 Billion

4. 4
© 2018 Express Scripts. All Rights Reserved.
Executive Summary
• In the past, we rebuilt the same functionality for every big project
• Entity Platform strategy focuses on building once and reusing
• This increases up-front costs, but decreases capital spend over time & increases
speed of delivery
WHY WE’RE MOVING TO ENTITIES
Pharmacy
As A
Platform
Rebates
20/20
Support
Clients
Manage
Products
Orders
Member
Prescription
Orders
Claim
Benefit
Contract
Claim
Benefit
Contract
Member
Benefit
Pharmacy
As A
Platform
Rebates
20/20
Support
Clients
Manage
Products
Orders Prescription Benefit
Member Claim Contract
Data
Sync

5. 5
© 2018 Express Scripts. All Rights Reserved.
Walking the Interaction and Business Tier Stack
User Interaction Channels
(Applications)
Business Platforms
User Interface
Logic
Human Workflow
UI Frameworks
Business platforms
consists of entities
with REST-based
APIs that
completely control
access to a data
domain.
They are first class
citizens.RequiredRequired
Web Sites
Mobile Apps
…and more...
Entity
Platform
Entity
Platform
Entity
Platform
Data Data Data
Person
Membership
Referral
Order
Shipment
Drug
Price Quote
and more...
Interaction APIs
Aggregations
UX-Specific Views
Optional
User-specific order
status
Cross-entity items, not aligned with a UI
that don’t map to an entity.
Optional
Business Process & Logic

6. 6
© 2018 Express Scripts. All Rights Reserved.
Claims History Lookup
Member Site
(Applications)
Business Platforms
Member Website
Used by our members to
view claims history,
orders etc.
RequiredRequired
User
Accounts
Person
Entity
Claims
Entity
Data Data Data
1. Member login-in. Authenticated
through User Accounts
2. A resource id that uniquely identifies
the member is retrieved
3. Using the unique member key,
appropriate member specific claims
history data is accessed and
displayed to the member
1 2 3

7. 7
© 2018 Express Scripts. All Rights Reserved.
Target Sync
Platform
Data Sync Platform (System Architecture)
Source
Service
Platform
Source
DB
Source
Service
Target
Service
Platform
Kafka
Retry Topic
Retry
Stream
Processor
Target
DB*
Target
Service
CDC Topic
Update Topic
CDC
Stream
Processor
Update
Stream
Processor
on error
üChange Data Capture (CDC)
üKafka
üTarget Sync Platform
Error Topic
CDC
Pivotal Platform
* Outside Pivotal Platform

8. 8
© 2018 Express Scripts. All Rights Reserved.
Data Sync Platform (Multi-tenancy, HA & Scalability)
Source
Platform
(Mainframe)
CDC Kafka
Near real-time Streaming
(Commodity Hardware)
Org 1
Org 2
Org 3
Foundation
Target Service Platform (Pivotal Platform)
AZ 1 AZ 2 AZ 3

9. 9
© 2018 Express Scripts. All Rights Reserved.
Kafka Libraries and Connectivity Options
üSeveral options
üProducer & consumer APIs
üStreams API
üConnectors ecosystem (Confluent hub) & Connector API
üSpring’s wrapper (spring-kafka, spring-cloud-stream)
üNo silver bullet
üEvery use-case is different and unique
üDo your own due diligence

10. 10
© 2018 Express Scripts. All Rights Reserved.
General Guidelines
üProducer & Consumer APIs
üApplications directly producing data streams (e.g. clickstreams, logs, IoT…)
üRead a stream and perform real-time actions on it (e.g. send email,
generate alarm, send notification…)
üStreams API
üApplications wanting to consume from Kafka and produce back into Kafka,
also called stream processing
üVery powerful & offers advanced features for complex processing logic
üKstream, Ktable, stateless/stateful processing, state store, aggregations,
punctuation…

11. 11
© 2018 Express Scripts. All Rights Reserved.
General Guidelines (continued)
üConnectors ecosystem (Confluent hub) & connector API
üInteractions with data stores ( e.g. S3, PostgreSQL, MongoDB, HDFS…)
üSpring’s Framework (spring-kafka, spring-cloud-stream)
üJava based applications
üSpring native out of the box capabilities
üHybrid Approach
üStreams API for advanced/complex logic
üSpring-cloud-stream for applications with generic streams processing
and handling

12. 12
© 2018 Express Scripts. All Rights Reserved.
Topic1
Topic2
Topic3
CDC
Stream Join
Table1
Table2
Table3
Tran1 Tran2 Tran3 Tran4 Tran5 Tran6
Merged
Stream
Kafka Partitioning Strategies for CDC
n Tables : n Topics With Multiple Partitions

13. 13
© 2018 Express Scripts. All Rights Reserved.
CDC
Stream Join
Table1
Table2
Table3
Tran1 Tran2 Tran3 Tran4 Tran5 Tran6
Merged
Stream
Partition1
Partition2
Partition3
Kafka Partitioning Strategies for CDC
n Tables : 1 Topic With Multiple Partitions

14. 14
© 2018 Express Scripts. All Rights Reserved.
CDC
Stream Join
Table1
Table2
Table3
Tran1 Tran2 Tran3 Tran4 Tran5 Tran6
Kafka Partitioning Strategies for CDC
n Tables : 1 Topic With 1 Partition
T1.Partition1
T2.Partition1
T2.Partition2
T2.Partition3
Re-Partition
Merged
Stream

15. 15
© 2018 Express Scripts. All Rights Reserved.
Kafka Partitioning strategies for CDC
ü1:1 Table to Kafka topic partitioned by table key
üAllows for highly parallelized consumption
üGood strategy when eventual consistency suffices
üChallenging if strong transactional consistency is need
üMultiple tables to 1 topic with 1 partition
üFirehose type approach
üAllows for strong transactional consistency
üRe-partition for parallelism
üMultiple tables to 1 topic partitioned by common logical key
üAllows for parallelism due to multiple partitions
üAllows for strong transactional consistency
üChallenging if a common key is not present across all table

16. 16
© 2018 Express Scripts. All Rights Reserved.
Performance & Health of the Pipeline
üLatency monitoring
üThroughput monitoring

17. 17
© 2018 Express Scripts. All Rights Reserved.
Target Sync
Platform
Performance Optimization with Caching
Source
Service
Platform
Source
DB
Source
Service
Target
Service
Platform
Kafka
ENTITY CHANGE
Topic
Target
DB
Target
Service
Source
Change Topic
Target
Change Topic
CDC
Stream
Processor
Update
Stream
Processor
ENTITY LATEST
Topic
Cache
Cache
Cache
Distributed
Kafka State
Store

18. 18
© 2018 Express Scripts. All Rights Reserved.
What’s Next
üData Lake for analytics
üMicroservices to act on data events in real time
üEntity first approach

19. 19
© 2018 Express Scripts. All Rights Reserved.
THANK YOU !