Downloaded 53 times
Uploaded byconfluent
Building an Enterprise Eventing Framework (Bryan Zelle, Centene; Neil Buesing, Object Partners, Inc) Kafka Summit SF 2019
Centene aims to enhance its enterprise eventing platform to support its rapid growth driven by mergers and acquisitions, facing challenges in data integration and migration. The platform will leverage asynchronous pub/sub messaging with Kafka for scalability and real-time data transfer, facilitating insight-driven decisions in healthcare operations. The document outlines Centene's strategic partnerships and the need for a customized real-time eventing solution that meets its specific requirements.
More Related Content
Similar to Building an Enterprise Eventing Framework (Bryan Zelle, Centene; Neil Buesing, Object Partners, Inc) Kafka Summit SF 2019
Building an Enterprise Eventing Framework (Bryan Zelle, Centene; Neil Buesing, Object Partners, Inc) Kafka Summit SF 2019
- 1. Building a Enterprise EventingPlatform Bryan Zelle and Neil Buesing
- 2. Centene Introduction Mission Statement: Transformingthe health of the community, one person at a time Medicaid: Medicare (Part D): Marketplace: Medicare: Other: Total: 12,700,000 4,000,000 2,000,000 1,000,000 3,700,000 23,400,000 30 States 50 States 21 States 28 States 33 States 50 States Membership Composition: Industry: Largest Medicaid and Medicare Managed Care Provider 0 5 10 15 20 25 Centene United Health Group Humana Anthem CVS Membership(Millions) Largest Managed Care Organizations Medicaid Medicare & Medicare PDP OtherGovernment Marketplace
- 3. $- $10,000 $20,000 $30,000 $40,000 $50,000 $60,000 $70,000 $80,000 $90,000 $100,000 2019 2018 20172016 2015 2014 2013 2012 2011 2010 2009 2008 2007 2006 2005 TotalRevenus(millions) Centene Yearly Revenue Centene Revenue WellCare Revenue Summary of Centene’s Key Challenges in one word… Growth $4.1 Billion Revenue to $96.9 Billion in 10 Years $80.4 Billion in growth in past 5 years $48.6 Billion in growth in past 2½ years Envolve Jan 2015 Wellcare Mar 2019 Fidelis Sep 2017 HealthNet Mar 2016 ? ? Cause of the growth… Mergers & Acquisitions By the numbers:
- 4. Medicare Medicaid International Federal Marketplace Addressable Market Federal Medicare$860B 40% State Medicaid International Market Federal Services Health Insurance Marketplace $2,000,000,000,000 + Centene Revenue $97,000,000,000 + Centene Revenue 4% Addressable Market 96% Additional Growth Opportunities $710 B 33% $260 B 12% $120B 6% $115 B 5% Centene Growth Outlook Targeted Pipeline ($270 Billion)
- 5. Medicare Medicaid International Federal Marketplace Addressable Market Federal Medicare$860B 40% State Medicaid International Market Federal Services Health Insurance Marketplace $2,000,000,000,000 + Centene Revenue $97,000,000,000 + Centene Revenue 4% Addressable Market 96% Additional Growth Opportunities $710 B 33% $260 B 12% $120B 6% $115 B 5% Centene Growth Outlook Targeted Pipeline ($270 Billion) Mergers & Acquisitions Data Integration & Data Migration
- 6. Data Integration &Data Migration 1 Shared Database • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure
- 7. Data Integration &Data Migration Shared Database Export Import • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure File 2 File Transfer (Batch ETL) • Latent Data • Direct Database Load • Consistency Challenges
- 8. Data Integration &Data Migration Export Import Shared Database File Transfer (Batch ETL) • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure • Latent Data • Direct Database Load • Consistency Challenges File API API Function Call Response 3 • Direct Coupling • Application Refactor • Availability Concerns • Scaling Concerns Remote Procedure Invocation
- 9. Data Integration &Data Migration Shared Database File Transfer (Batch ETL) Export Import • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure File • Latent Data • Direct Database Load • Consistency Challenges API API Function Call Response • Direct Coupling • Application Refactor • Availability Concerns • Scaling Concerns Remote Procedure Invocation 4 Pub / Sub Messaging (Streaming ETL) Event MessageBus • Loosely Coupled • No Application Refactor • Highly Availability • Highly Scalable • Real-Time Data
- 10. Data Integration &Data Migration Pub / Sub Messaging (Streaming ETL) Event MessageBus • Loosely Coupled • No Application Refactor • Highly Availability • Highly Scalable • Real-Time Data Shared Database File Transfer (Batch ETL) Export Import • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure File • Latent Data • Direct Database Load • Consistency Challenges API API Function Call Response • Direct Coupling • Application Refactor • Availability Concerns • Scaling Concerns Remote Procedure Invocation * What is a Event? Definition: “A significant change in state” • Statement of fact (immutable) • Expects no response (or call to action) • Has a defined “timepoint” Persistence • Stateless: Notification Event • Stateful: Event-Carried State Transfer How can you use events? E1 E2 E3+• Combine multiple Events E1 E2+• Absence of an Event E1 E2• Leverage Single Event
- 11. Pub / SubMessaging (Streaming ETL) Event MessageBus • Loosely Coupled • No Application Refactor* • Highly Availability • Highly Scalable • Real-Time Data Event Structure Example Event Payload (JSON vis REST) “Metadata” : { “Transaction ID” : “C7084816514A5D260”, “User ID” : “USER1”, “Time Stamp” : “201803051315400000000000”, “Transaction Type” : “UPDATE”, “Source System” : “d8amisou6p.MEMBER_CONTACT” } , “Event Body” : { “Event Type” : “Member-PCP-Change”, “Previous Value” : “Dr. John Smith”, “Updated Value” : “Dr. Bryan Zelle”, “Event Source” : “Inbound-Member-Call”, “Caller Information” : { “Name” : “Jane Doe”, “Inbound Number” : “1-614-847-0982”, “Call Resolution Status” : “5 - Highly Satisfied”, “First Call Resolution” : “Success”, “Internal Representative” : “CN-10238381”, ”Call Duration (Minutes)” : “8:19” } Transaction Metadata Who - Who changed the data ?* What - What data changed ? When - When the data changed ? Where - Where was the data changed ? Why - Why was the data changed ? What Event information are we capturing? Event Body *
- 12. Data Integration &Data Migration Pub / Sub Messaging (Streaming ETL) Event MessageBus • Loosely Coupled • No Application Refactor • Highly Availability • Highly Scalable • Real-Time Data Shared Database File Transfer (Batch ETL) Export Import • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure File • Latent Data • Direct Database Load • Consistency Challenges API API Function Call Response • Direct Coupling • Application Refactor • Availability Concerns • Scaling Concerns Remote Procedure Invocation * How do Events fit with Streaming? Business Driver: Business Enabler: Real Time Data Streaming Integrated Real Time Enterprise
- 13. Event Driven Apps DefiningCharacteristic: Stream Driven Apps Where is my driver? What is the temperature? How long till driver arrives? How long till room cools? Leverage real-time events to display current state Defining Characteristic: Continually combine multiple streams of real-time events to extract value from state
- 14. Event Driven Apps DefiningCharacteristic: Stream Driven Apps Where is my driver? What is the temperature? How long till driver arrives? How long till room cools? Leverage real-time events to display current state Defining Characteristic: Continually combine multiple streams of real-time events to extract value from state What is the temperature? How long till driver arrives? Events are KEY Apache Kafka Stores Events: Resilient / Durable Distributed / Highly Available High-Throughput / Low Latency But…. What Isn’t Provided by Kafka?
- 15. Event Driven Apps DefiningCharacteristic: Stream Driven Apps Where is my driver? What is the temperature? How long till driver arrives? How long till room cools? Leverage real-time events to display current state Defining Characteristic: Continually combine multiple streams of real-time events to extract value from state What is the temperature? How long till driver arrives? But…. What Isn’t Provided by Kafka? Synthetic Events Event Registration (Event Discovery) Sensitive Data Redaction Event Encryption Flexible Schema Validation (JSON not AVRO) Automated Disaster Recovery (Event Rehydration) Searchable Events (Query Event Store) Distributed Tracing of Event (Event Lineage) Event Metrics (Event Dashboards) Consistent Event Structure (Data Governance)
- 16. Pub / SubMessaging (Streaming ETL) Event MessageBus • Loosely Coupled • No Application Refactor* • Highly Availability • Highly Scalable • Real-Time Data Mediated (Orchestrated) Eventing Mediator Topology Mediator transfers events to assigned event channel (Topic) Centrally Coordinated Event Routing Complete Decoupling of Event Channels Increased Complexity at cost of increased coordination of event execution Advantages: • Consistent / Common Framework • Enforce Data governance • Economy of Scale Advantage • Technology abstraction / decoupling Disadvantages: • External bottleneck (Mediator Owner) • Single Point of Failure • Duplicative data storage
- 17. Generic Event Mediator Common Core Architecture: 1)Event Source 2) Event Intake 3) Event Channel 4) Event Router 5) Event Subscription 6) Event Destination Event Channel Event Router Event Subscription Event Destination Event Intake Event Source Event Mediator 321 4 5 6 Event Bridge Event Grid Apache Camel Knative Eventing Mule ESB Mediator Alternatives?
- 18. Generic Event Mediator Required Features& Functionality Event Channel Event Router Event Subscription Event Destination Event Intake Event Source Event Mediator 321 4 5 6 Design Criteria 1) AVRO Event Serialization 2) JSON Validation of Event Body 3) Centralized Event Registry 4) Distributed Tracing of Events 5) Sensitive Data Redaction 6) Turn / Key Self-Service 7) Cloud Agnostic 8) Permanent Event Storage 9) Flexible Ingestion Intake 10) Pre-built Monitoring / Dashboards 11) Synthetic Events Reduced Message Size -> Reduced Storage Cost in Cloud Data Validation -> Clean Data Easily Find Events -> Prevents Event Duplication & Increases Adoption Tracing -> Provides Event Lineage and Auditability Data Restriction -> Protects HIPPA data (including PHI/PII) Automated Configuration -> Reduced manual administrative burden Multi-Cloud Strategy -> No Reliance on Single Cloud Provider Event Persistence -> DR Strategy + Event Sourcing / Hydration Legacy Systems Limitations -> Offer REST, gRPC, SOAP Interfaces & API’s Universal Metrics -> Consistent / Granular Event Visibility Fictitious Event -> Blue/Green Deployments, Prod Smoke Testing, Etc. Business Value
- 19. 1) AVRO EventSerialization 2) JSON Validation of Event Body 3) Centralized Event Registry 4) Distributed Tracing of Events 5) Sensitive Data Redaction 6) Turn / Key Self-Service 7) Cloud Agnostic 8) Permanent Event Storage 9) Flexible Ingestion Intake 10) Pre-built Monitoring / Dashboards 11) Synthetic Events Reduced Message Size -> Reduced Storage Cost in Cloud Data Validation -> Clean Data Easily Find Events -> Prevents Event Duplication & Increases Adoption Tracing -> Provides Event Lineage and Auditability Data Restriction -> Protects HIPPA data (including PHI/PII) Automated Configuration -> Reduced manual administrative burden Multi-Cloud Strategy -> No Reliance on Single Cloud Provider Event Persistence -> DR Strategy + Event Sourcing / Hydration Legacy Systems Limitations -> Offer REST, gRPC, SOAP Interfaces & API’s Universal Metrics -> Consistent / Granular Event Visibility Fictitious Event -> Blue/Green Deployments, Prod Smoke Testing, Etc. Generic Event Mediator Event Channel Event Router Event Subscription Event Destination Event Intake Event Source Event Mediator 321 4 5 6 Required Features & Functionality Design Criteria Business ValueLeverage 3rd Party Frameworks or Build Custom? Assessment: Majority of frameworks focused engineering effort on how to get data into framework as easily as possible • Higher Data Ingest = Increased Revenue (SaaS) • Too many gaps with current features* • Limited flexibility because of so many customers Decision: Build Centralized Eventing Framework for Enterprise use across all Centene Domains * * * * * *
- 20. CentEvent Architecture Docker Container Kubernetes Intake Application Axway Gateway Serialize Deserializer ConfluentSchema Registry Caffeine Cache Authorization Tokens Event Types Routing Rules Routing HASH Firehose Topic Docker Container Kubernetes Router Application Consumer Topics Client Portal Event Discovery UI Admin UI Docker Container Kubernetes Admin API Mongo Charts Tracing Monitoring
- 21. Field Level DataRedaction Use Case Example: Data Science Predication Models • Restricting Member PHI / PII (SSN, Medicaid / Medicare ID, etc) Encounter Processing • Restricting Bank/ Account Payment Information Business Requirement: Enforce Least Privilege Access to HIPPA / PHI / PII Data Design: I. Capture Event Metadata – What fields are sensitive? II. Capture Team / App level permissions – Who can see what? III. Duplicate Event -> Modify Event - Redact Sensitive Fields with ***REDACTED***
- 22. Discard Synthetic Events Use CaseExample: Pass-Through Example - Service that validates Payment Account Number Discard Example - Service that processes Claim Payment Business Requirement: Integrated End-to-End Testing without Central Coordination • Synthetic Monitoring / Continuous Unit Testing • Leverage non-prod traffic for Blue / Green Deployments • Inline Production Troubleshooting (in real-time) Design: I. Mandate “Synthetic” Event Property II. Establish Micro-Service Pattern • Pass-Through Event • Discard Event Simulate Process Flow by Generating Synthetic Events EventChannel Pass Through Perform Action Discard Action = Synthetic Event
- 23. Event Tagging Use CaseExample: Health Plan Claim Processing - Tag Claim Events by which State its associated with Process Orchestration - Tag Events with which Step in the Process is next Business Requirement: Be able to route events based on “Event Tags” • Filter Kibana dashboard metrics by Tags • Aggregate SLA’s by Tags • Filter Jaeger Distributed Traces by Tags Design: I. Tags are Optional (0 to N) II. Tags are attached to event by Producer, to be leveraged by Consumer III. Can be used in any combination or order (flexible) • Same Event -> Different Tags • Different Events -> Same Tags Same Event (Different Tags) Different Events (Same Tags)
- 24. Demo Build a KafkaTopic Request Process leveraging all Asynchronous Events Business Requirement: Additional Requirements: • Only process creation of “approved” topics • Create audit trial of any work performed for SOX compliance • Create Real + Synthetic Flows KaaS UI N: KaaS V: Requested O: Topic SO: Creation Authorizer N: Authorizer V: Sent O: Notification-Email SO: Approval-Request N: Authorizer V: Received O: Approval-Response SO: Email Audit Tag: Approved KaaS API Kafka Cluster N: KaaS-API V: Created O: Topic SO: Successfully B C E A B C D E UI Tag: Unapproved A N: KaaS V: Requested O: Topic SO: Creation D
- 25. Demo Build a KafkaTopic Request Process leveraging all Asynchronous Events Business Requirement: Additional Requirements: • Only process creation of “approved” topics • Create audit trial of any work performed for SOX compliance • Create Real + Synthetic Flows KaaS UI N: KaaS V: Requested O: Topic SO: Creation Authorizer N: Authorizer V: Sent O: Notification-Email SO: Approval-Request N: Authorizer V: Received O: Approval-Response SO: Email Audit Tag: Approved KaaS API Kafka Cluster N: KaaS-API V: Created O: Topic SO: Successfully B C E A B C D E UI Tag: Unapproved A N: KaaS V: Requested O: Topic SO: Creation D Synthetic Process Flow Real Process Flow
- 26. Recap Recap 1 Centene’s CoreChallenge is Growth cause by Mergers & Acquisitions; causing us to revaluate our Enterprise Data Integration and Data Migration Strategies… Event MessageBus 2 Async Pub / Sub Eventing through Kafka provides us valuable capabilities: - Highly Scalable - High Autonomy / Decoupling - High Availability & Data Resiliency - Real Time Data Transfer - Complex Steam Processing “Metadata” : { “Transaction ID” : “C7084816514A5D260”, “User ID” : “USER1”, “Time Stamp” : “201803051315400000000000”, “Transaction Type” : “UPDATE”, “Source System” : “d8amisou6p.MEMBER_CONTACT” } , “Event Body” : { “Event Type” : “Member-PCP-Change”, “Previous Value” : “Dr. John Smith”, “Updated Value” : “Dr. Bryan Zelle”, “Event Source” : “Inbound-Member-Call”, “Caller Information” : { “Name” : “Jane Doe”, “Inbound Number” : “1-614-847-0982”, “Call Resolution Status” : “5 - Highly Satisfied”, “First Call Resolution” : “Success”, “Internal Representative” : “CN-10238381”, ”Call Duration (Minutes)” : “8:19” } 3 Leveraging a Mediator Topology enables the creation of meaningful events; which provide insight into why things are happening, so we can react to them in real time…
- 27. Recap (Part 2) 4 Existingframeworks don’t meet all of Centene’s Eventing requirements; investment made to build our own Real-Time Eventing Platform 5 Leverage our Strategic Partners and their IP to build a Scalable Platform • Apache Kafka • K-Streams • Confluent Schema Registry • MongoDB • Kafka / Mongo Connector • MongoDB Charts 6 Execute on a multi-faceted Data Integration strategy that uses Asynchronous Eventing and Real-Time Data Streaming to facilitate current and future Mergers and Acquisitions
- 28. 28 Join Centene? Help usin our migration to Cloud Data Streaming Team is hiring! • Data Engineers • Site Reliability Engineers • QA Testers • PO / SM / BA • Software Engineers Reach out to: Bryan.A.Zelle@centene.com
- 29. Sara Zeman sara.zeman@objectpartners.com 1515 CentralAve NE Suite 100 Minneapolis, MN 55413 Looking for a Real-Time Data Streaming Partner? ● Minneapolis (HQ), Omaha ● Founded in 1996 ● Clients Nationwide ● 150+ Consultants, all full-time employees Practice Areas • Cloud Engineering • Real-time Data • Modern APIs • Mobile and Web Special Thanks • Nick Larson • Ryan Hoffman • Neil Buesing
- 30.