The document discusses Surescripts' implementation of DataStax Enterprise (DSE) to power its cloud applications. Surescripts processes millions of healthcare messages daily and needed a scalable data platform. It transitioned from Oracle to DSE for its persistent tier to gain horizontal scalability and high availability. Initial results have shown improved performance and efficiency over Oracle. Surescripts aims to create logical data centers spanning physical sites using DSE replication to enhance reliability.

1. Delivering Enhanced Message Processing
at Scale with DataStax Enterprise
Keith Willard Bill Liu Matt Griffin
Chief Software Architect Enterprise Architect Lead Database Engineer
Surescripts Surescripts Surescripts
Jie Wu
Director, Product Marketing
DataStax

2. From validation to momentum
400+
Employees
$190M
Funding
500+
Customers
Founded in April 2010
Santa Clara • San Francisco • Austin •
London • Paris • Berlin • Tokyo • Sydney
(Series E – Sept. 2014) 30% +
2016 World’s Best
100 Cloud Companies
Ranked #1 in multiple operational
database categories
© 2017 DataStax, All Rights Reserved. Company Confidential

3. World of cloud applications
Each new innovation accelerates expectations
© 2017 DataStax, All Rights Reserved. Company Confidential

4. CONTEXTUAL
These cloud applications
must meet expectations
ALWAYS-ON DISTRIBUTED SCALABLEREAL-TIME
© 2017 DataStax, All Rights Reserved. Company Confidential

5. DataStax Enterprise (DSE) -- the always-on data
management platform for cloud applications
© 2017 DataStax, All Rights Reserved. Company Confidential

6. Powering cloud applications
© DataStax, All Rights Reserved.6
EFFORTLESS
SCALE
• Take advantage of every opportunity.
• Focus on what matters most to you.
ALWAYS-ON
• Designed to handle any failure,
no matter how catastrophic.
INSTANT INSIGHT
• Built into your application to create
actionable, modern experiences.

7. DataStax Enterprise subscriptions
DSE Standard DSE Max
Data Platform
Certified Apache Cassandra™ ✓ ✓
Advanced Security ✓ ✓
Automatic Management Services ✓ ✓
Advanced Replication ✓ ✓
Tiered Storage ✓ ✓
Multi-Instance ✓ ✓
In-Memory ✓
Search ✓
Analytics ✓
Graph Option
Management and Development Solutions
DataStax OpsCenter ✓ ✓
DataStax Studio ✓ ✓
DSE MAX
For teams looking for an
integrated data management
platform with advanced
features
DSE STANDARD
For teams that need a
solution easy to build and
effortless to scale

8. DataStax Managed Cloud (DMC)
A Fully Managed, Secure Architecture
8
• DataStax Enterprise on AWS with Managed Provisioning
and Scaling by DataStax
• 24x7x365 Coverage, Lights-Out Management
• System Configuration and Tuning to Meet Customer
Specific Requirements
• Architecture Advisory Services, Guidance and Best
Practices

9. Why DMC Matters to our customers
• Focus on business innovation
• Offload operations to DataStax
• Reduce risk with a secure, white glove service by experts
• Grow easily as your business expands
• Accelerate time-to-market
• A single data layer for on-premises and public cloud
• No re-write of applications against every cloud provider’s proprietary data services
• Create APIs and micro-services to a consistent data layer
• Choice and flexibility for hybrid architecture needs
• Extend to fit your hybrid cloud architecture
• Span or migrate across infrastructure models with 100% compatibility
• Retain data autonomy without cloud vendor lock-in
9

10. Copyright © 2016 by Surescripts, LLC. All rights reserved.Copyright © 2016 by Surescripts, LLC. All rights reserved.
DataStax Enterprise in the Surescripts Context
Keith Willard, Bill Liu, Matt Griffin
August 2017

11. Copyright © 2016 by Surescripts, LLC. All rights reserved.
Surescripts network
Surescripts is responsible for the largest number of clinical oriented electronic transactions
in the country today
• The electronic prescription that flows from your doctor’s office computer to your local or mail order
pharmacy
Surescripts
Surescript Core
Network
Elig, Med Hx, SCRIPT
SCRIPT
Claim
EHR
PBM
Pharmacy

12. Copyright © 2016 by Surescripts, LLC. All rights reserved.
Growth still linear
 How to maintain ability to scale
 How to maintain and increase reliability numbers
12

13. Copyright © 2016 by Surescripts, LLC. All rights reserved.
A core load of our processing pipeline
 We are message centric
• Each operation is encapsulated by the receipt or transmission of a message from or to a
network participant
 Large volume of messages
• Average 50-60 million per week. ~1/2 between 10am and 2pm.
 Median size message ~4k
• But ranges in size upwards to 20 MB.
 Write challenge of scale to write
 Read problem
• Frequently need to refer to a related past message to properly process and route incoming
message.
13

14. Copyright © 2016 by Surescripts, LLC. All rights reserved.
Constant pressure to improve SLA
 External failures affecting the “fabric” of our data centers
 Failures related to data center synchronization processes
 Failures related to software mutation
• Infrastructure software upgrades
• Our own application software upgrades
14

15. Copyright © 2016 by Surescripts, LLC. All rights reserved.
Architecture transitions
 Past and near present
• mixed Java and .NET transaction application tier heavily virtualized
• Persistent tier an MQ/Oracle relational OLTP environment
• Analytical/Reporting tier Oracle Exadata data warehouse
• High reliability relies heavily on a dual physical data center model (Mpls and VA)
 Future (cluster centric)
• Similar mixed java and .NET transactions Docker or equivalent
• Persistent Tier Kafka/DSE/some relational at the OLTP tier
• Analytical Reporting Hadoop/Exadata
• Higher reliability and scalability multiple “logical data centers” with two physically distinct Hadoop
clusters.
15

16. Copyright © 2016 by Surescripts, LLC. All rights reserved.
Logical data center
 Restructure persistent tier around two themes
• Horizontally, economically scalable persistence
streaming
• Eventually consistent, locally clusterable WAN extended
queryable database
 Transition from Oracle and Oracle RAC to alternative
• Horizontally scalable with a practical
synchronization/replication infrastructure
Non-Audit
Relational
Data
Application
Pools
Microservice
Pools
Kafka
DSE

17. Copyright © 2016 by Surescripts, LLC. All rights reserved.
End game
17
Non-Audit
Relational
Data
Application
Pools
Microservice
Pools
Kafka
DSE
Non-Audit
Relational
Data
Application
Pools
Microservice
Pools
Kafka
DSE
Non-Audit
Relational
Data
Application
Pools
Microservice
Pools
Kafka
DSE
Non-Audit
Relational
Data
Application
Pools
Microservice
Pools
Kafka
DSE
ZFS ZFS
ZFS Replication
Hbase/Kudu Replication
Kafka Replicator
OR
Always

18. Copyright © 2016 by Surescripts, LLC. All rights reserved.
Pending transaction service
18

19. Copyright © 2016 by Surescripts, LLC. All rights reserved.
Overall operational experience
 In a word - outstanding
 Read & write performance has equaled or bettered our relational OLTP tier
• Now manageably scalable
 One rolling upgrade since going live in November without loss of uptime
 DataStax Support was very responsive
• When we were affected by a base bug (CASSANDRA-12247) that caused intermittent repair
failures and write timeouts
 Improved efficiency
• The involved DBE reports less on-going maintenance activity rather than more for equivalent Oracle
infrastructure.
19

20. Copyright © 2016 by Surescripts, LLC. All rights reserved.
Cluster performance summary (last 7 days)
Write throughput,
cluster peak
720 writes/sec
peak
~9.5 million/day
Read throughput,
Cluster peak
305 reads/sec
peak
~4.5 million/day
Write Latency 99% latency 65 ms/write 50% latency 15 ms/write
Read Latency 99% latency 79 ms/read 50% latency 38 ms/write
Exceptions none

21. Copyright © 2016 by Surescripts, LLC. All rights reserved.
CPU load
21

22. Copyright © 2016 by Surescripts, LLC. All rights reserved.
OS Load
22

23. Copyright © 2016 by Surescripts, LLC. All rights reserved.
Network I/O
23

24. Copyright © 2016 by Surescripts, LLC. All rights reserved.
Use case criteria guiding choice of DSE
 Stereotyped (ie. development time) access patterns
 Size thoughtfully
 Replication requirements
 Scale capabilities
 Reliability via cluster architecture
• We will require a very long baseline for full answer.
24

25. Copyright © 2016 by Surescripts, LLC. All rights reserved.
Things we think we did right, and wrong
 Things we did right
• Kept to the well understood wheelhouse of DSE - primary key access pattern
• Paid attention to the recommendation to stay under 50% disk capacity when determining sizing
• Blue/green deployment option has given us precise understanding of behavior of new versus prior
infrastructures.
 Things we could do better
• Have not yet broadly familiarized our db staff with DSE infrastructure
▪ So far only two individuals.
• We have partition keys that include identifiers generated by our trading partners. These should be
unique according to business rules, but we do not enforce uniqueness, so we are at risk of wide
partitions.
 Things we don’t yet know
• Major version/ major OS upgrade events haven’t occurred yet.
25

26. Next steps
Learn more about DataStax Enterprise:
 eBook
• The Magic Number 5 - Why Cloud Applications Require Always-
on Data Platform
• Why Your RDBMS Fails at Scale?
 Analyst report
• Gartner 2016 Critical Capabilities for Operational Database
Management Systems
 Webinar recording
• Introducing DataStax Enterprise 5.1
 Trainings
• https://academy.datastax.com/courses
 Download DSE and other toolkits
26
Learn more about DataStax Managed
Cloud (DMC):
 Dedicated website
 eBook
• Time to Innovate – Introducing DataStax Managed
Cloud
 DMC datasheet
 Webinar recording
• DataStax Enterprise Managed Cloud: Focus on
Innovation, not Administration

27. Copyright © 2016 by Surescripts, LLC. All rights reserved.
Thank You!
© 2017 DataStax, All Rights Reserved. Company Confidential