Case Study in Healthcare for MongoDB Mark Cassidy, CTO, Verity Solutions MongoDB Evenings Seattle October 20, 2016

1. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
Seattle Evenings
Mark Cassidy | October 2016

2. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
Introduction to Verity Solutions
 Founded in 2002 and previously known as Talyst Inc., Verity
Solutions is a leader in software and services for the federal
340B drug discount program
 Hundreds of integrated healthcare systems, acute-care
hospitals, pharmacies, and other 340B-eligible Covered
Entities throughout the United States rely on Verity
340BTM products to maintain compliance, maximize savings,
and keep them up-to-date with ever-changing regulations
 We partner with customers over the long-term to support on-
going compliance and audit readiness
Verity Solutions_2

3. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
More on 340B
 The 340B Drug Discount Program is a U.S. federal
government program created in 1992 that requires drug
manufacturers to provide outpatient drugs to eligible
health care organizations/covered entities at significantly
reduced prices.*
 The program was designed to help qualifying entities
defray their costs of providing care to indigent and
underinsured patients
 The program is funded by drug manufacturers, not
taxpayer dollars
 The rules of compliance to the program are both
complex and nuanced
Verity Solutions_3
* source: Wikipedia
3 4 0 BDRUG DISCOUNT PROGRAM

4. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
Conceptual Product Overview
Verity Solutions_4
Verity 340B
platform
SFTP
SFTP
SFTP
Inpatient dispenses
Outpatient dispenses
Medicaid dispenses
EDI
832
EDI
855
EDI
850
SFTP
SFTP
EDI
810
SFTP
SFTP
Web application interface
- Order management
- Configurations
- Reporting
HTTPS
Hospital systems
Wholesale drug
distributor systems
Patient admission &
discharge records
SFTP
Retail Pharmacy
Claims transactions
SFTP
Drug reference
Data(subscribed)
SFTP
Price catalogs
Orders
Invoices
Order acks

5. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary Verity Solutions_5
Verity had 2 separate legacy products, dating back to early 2000’s
Verity was losing market share to competitors
Budget was committed to develop a new platform in 2015
New development team recruited (internally & externally)
New Platform Initiative

6. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
Product Feature Goals
 Single platform supporting multiple products
 Modern, web-based front-end (no client installation required)
 Core features of legacy products plus new, differentiating
features unique among the competition
 Surface the value of the extensive data in the system
 Provide migration path for customers on legacy products
Verity Solutions_6

7. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
Non-Functional Goals
 Nimble: ability to add features & fix bugs quickly, respond to
customer needs
 Flexible to accommodate ever-changing regulatory requirements
and market dynamics
 Automate repetitive tasks: lower risk of human error & cost of
operation
 High availability/redundancy
 Streamline implementation & support processes
 High performance:
o Fast response time in the web interface
o Low latency for bulk processing
Verity Solutions_7

8. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
Scale Requirements
 400M hospital & retail pharmacy dispense transactions/month
 10M patient visit records/month
 10K hospitals
 1.5B(!) unique drug prices
 30K retail pharmacies
 Concurrent users: 500
 5-year data retention required for compliance
Verity Solutions_8

9. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
Timeline
Verity Solutions_9
Development
Sprint 1 began early
February, 2015
First product beta
began in
September, 2015
First live customer
in production
January 4, 2016

10. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
Platform
Architecture
Verity Solutions_10
...
REST services, security
framework, session
management
MongoDB
Client-side
application
framework
Verity 340B Platform
Secure file
exchange
File storage
...
Bulk data-processing
services
Hospitals, Pharmacies,
Distributors, Data Providers
End Users

11. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
Technology Decisions
Verity Solutions_11
Foundation Technologies Development tools
 Database: MongoDB
 Server-side programming language:
Go
 Client framework: Angular
 File exchange: MOVEit®
(Ipswitch, Inc)
 OS: CentOS
 MongoChef
 Mongo Shell
 Git
 Sublime Text
 Jenkins
 Jira

12. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
Data Modelling
 Logical data model developed first
o Clear partitioning of concepts
o Relationships/dependencies between different elements in the model
 Analysis of workloads
o Identify query patterns for data elements that need to be accessed together
o Analyze frequency/volume of different query types
 Physical data model design
o Map logical model to MDB collections
o De-normalize where appropriate for performance
o Balancing act: logical organization of data, keeping document scope from being too monolithic, minimizing
need to join across collections for high-volume queries
Verity Solutions_12

13. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
MongoDB Deployment Architecture
 No sharding
 3-node Replica Set
 Cloud Manager for admin and snapshot backups(using MDB’s
hosted backup service)
 Multiple pre-prod environments(in both single node & replica set
configurations)
Verity Solutions_13

14. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
Batch Workloads
 Data ingestion
o Validation, duplication logic, transformation, mostly new document writes
o Individual operations relatively simple computationally, just lots of them
 Matching(find dispenses eligible for discounted replenishment)
o Iterate over new dispenses, query for possible matching visits(and visa-versa)
o Evaluate each possible match
o Write new document when matches are found
o Individual operations relatively simple computationally, just lots of them
 Order & invoice processing
o Order generation/splitting(discounted & non-discounted portions)
o Reconcile invoices with orders
o Allocate eligible dispenses to invoices to justify discount
o Handling over-allocated scenarios, credits/returns
o Logic is complex, number of operations is relatively small
Verity Solutions_14

15. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
Request/Response Workloads
 Config/Admin – simple CRUD, mostly individual documents
 Match & invoice review & user actions
o Search/filter queries
o Override default logic on individual results
o Moderately complex logic, relatively few operations
 Reporting
o Search/filter queries
o Result sets constrained by limiting filter criteria
Verity Solutions_15

16. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
Performance & Data Growth Trends
Verity Solutions_16
01/01/16 02/01/16 03/01/16 04/01/16 05/01/16 06/01/16 07/01/16 08/01/16 09/01/16 10/01/16
Document Counts
PatientVisit
Dispense
MatchedDispense
AccumulatorTrans
PriceCatalogItem
0
2000
4000
6000
8000
10000
REST API Response Time

17. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
Summary/What’s Working Well
 Data Modelling
o Think through the model carefully before implementing (no different than if you’re using relational technology)
o Pay attention to how dynamic your data is when making de-normalization decisions
o Refactor as you go: de-normalize, shifting data to different collections as needed
 Data validation & constraint enforcement: our data has been highly reliable/consistent
 Integrity monitoring: verify counts & sums across collections containing correlated data
 “Bake Time” enabled us to find bugs before they impacted production data
 Resist/Avoid premature optimizations
 Agile/Scrum process works very well for us, with lots of automation
 Small, strong team with solid process can accomplish a LOT!
Verity Solutions_17

18. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
Lessons Learned
 Pay careful attention to concurrency where multiple batch processes update the same documents
 Adopt bake time at the outset
 Be prepared for data repairs
 Understand how to deal with lack of multi-phase commits and lack of foreign key integrity enforcement
Verity Solutions_18

19. ©2016 Verity Solutions Group, Inc.– Confidential & Proprietary
Thank You!
Verity340b.com | 800.581.1378