SlideShare a Scribd company logo

Time to Talk about Data Mesh

Time to Talk about Data Mesh

1 of 21
Download to read offline
It is time to talk about
DATA MESH
Distributed Data Management for
Microservices
MICROSERVICE’S CHALLENGE
01
3
THE CHALLENGES OF MICROSERVICE DESIGN
TRANSFORM BUSINESS LOGIC AND DATA INTO MICROSERVICE ARCHITECTURE
Business flow combing,
Function and service separating
Data decoupling and fragmentation
management
◼ DDD (Domain-Driven Design)
◼ Split services by business flow
◼ EDA (Event-driven Architecture)
◼ Microservice component design
◼ Microservice horizontal scaling strategies
◼ Separating data by business flow
◼ Sharing data across domains
◼ Vast data deployment and caching strategies
◼ Data consistency(Data fragmentation management)
◼ The final data consistent
◼ Strong data consistent
◼ Absolute data consistent
CHALLENGE
DONE WELL
Business Logic
Data
CELL GATEWAY
Database
Service
Service
Analyze Decouple Aggregate
To implement the microservice architecture with
decision of design patterns, deployment and
maintenance strategies, integration of technical
components, planning and actual software
architectures.
Implement
API SERVICES/
EDGE SERVICES
COMPOSITE SERVICES/
INTEGRATION SERVICES
CORE SERVICES/
ATOMIC SERVICES
CLIENT APPLICATIONS
To implement the design and
choose an immediately available
solution to implement various
microservice design patterns and
basic architectures. Based on
stable software components and
platforms, determine service roles
and implement business logic.
Special design with stateful atomic services
4
TRANSFORM TO DATABASE PER SERVICE
For Microservice Architecture
App
service
App
service
App
service
DATABASE
App
service
App
service
App
service
Database per service
Shared Database
How?
Risk?
Performance?
Plan?
5
PROBLEMS WITH DATABASE PER SERVICE
WHEN WE IMPLEMENT IT
App
service
App
service
App
service
Round-trips Problem
Busy
App
service
App
service
App
service
Anti-Pattern
App
service
App
service
App
service
Confusion
OR
OR
How do we solve these problems ?
6
PROVISION ONCE, SNAPSHOT MANY
CACHEING FOR MULTIPLE SNAPSHOTS
CLIENT
APPLICATIONS
Service API
SNAPSHOT
Service API
DATA FLOW
Source
VIRTIAL TABLE
VIRTUAL TABLE BUSINESS
BUSINESS
100M Records
100M Records
100M Records
Service
Owner
According to requirements of business task, selecting
database, table and fields to make a copy into a
virtual table.
Update
Database per service
1
Provision
2
3
Program A
Program B

Recommended

Data Mesh for Dinner
Data Mesh for DinnerData Mesh for Dinner
Data Mesh for DinnerKent Graziano
 
Data Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshData Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshJeffrey T. Pollock
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureDatabricks
 
Enabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationEnabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationDenodo
 
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...HostedbyConfluent
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Databricks
 

More Related Content

What's hot

Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureDatabricks
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)James Serra
 
Modern Data architecture Design
Modern Data architecture DesignModern Data architecture Design
Modern Data architecture DesignKujambu Murugesan
 
Databricks Platform.pptx
Databricks Platform.pptxDatabricks Platform.pptx
Databricks Platform.pptxAlex Ivy
 
Data Architecture Best Practices for Advanced Analytics
Data Architecture Best Practices for Advanced AnalyticsData Architecture Best Practices for Advanced Analytics
Data Architecture Best Practices for Advanced AnalyticsDATAVERSITY
 
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...DATAVERSITY
 
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Tristan Baker
 
How to govern and secure a Data Mesh?
How to govern and secure a Data Mesh?How to govern and secure a Data Mesh?
How to govern and secure a Data Mesh?confluent
 
Big data architectures and the data lake
Big data architectures and the data lakeBig data architectures and the data lake
Big data architectures and the data lakeJames Serra
 
Five Things to Consider About Data Mesh and Data Governance
Five Things to Consider About Data Mesh and Data GovernanceFive Things to Consider About Data Mesh and Data Governance
Five Things to Consider About Data Mesh and Data GovernanceDATAVERSITY
 
Building Lakehouses on Delta Lake with SQL Analytics Primer
Building Lakehouses on Delta Lake with SQL Analytics PrimerBuilding Lakehouses on Delta Lake with SQL Analytics Primer
Building Lakehouses on Delta Lake with SQL Analytics PrimerDatabricks
 
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...DataScienceConferenc1
 
Activate Data Governance Using the Data Catalog
Activate Data Governance Using the Data CatalogActivate Data Governance Using the Data Catalog
Activate Data Governance Using the Data CatalogDATAVERSITY
 
Data Warehousing in the Cloud: Practical Migration Strategies
Data Warehousing in the Cloud: Practical Migration Strategies Data Warehousing in the Cloud: Practical Migration Strategies
Data Warehousing in the Cloud: Practical Migration Strategies SnapLogic
 
Data Architecture for Data Governance
Data Architecture for Data GovernanceData Architecture for Data Governance
Data Architecture for Data GovernanceDATAVERSITY
 
Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)James Serra
 
Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?DATAVERSITY
 
Introducing Databricks Delta
Introducing Databricks DeltaIntroducing Databricks Delta
Introducing Databricks DeltaDatabricks
 
Data Architecture Strategies: Data Architecture for Digital Transformation
Data Architecture Strategies: Data Architecture for Digital TransformationData Architecture Strategies: Data Architecture for Digital Transformation
Data Architecture Strategies: Data Architecture for Digital TransformationDATAVERSITY
 

What's hot (20)

Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data Architecture
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)
 
Modern Data architecture Design
Modern Data architecture DesignModern Data architecture Design
Modern Data architecture Design
 
Databricks Platform.pptx
Databricks Platform.pptxDatabricks Platform.pptx
Databricks Platform.pptx
 
Data Architecture Best Practices for Advanced Analytics
Data Architecture Best Practices for Advanced AnalyticsData Architecture Best Practices for Advanced Analytics
Data Architecture Best Practices for Advanced Analytics
 
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
 
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
 
How to govern and secure a Data Mesh?
How to govern and secure a Data Mesh?How to govern and secure a Data Mesh?
How to govern and secure a Data Mesh?
 
Big data architectures and the data lake
Big data architectures and the data lakeBig data architectures and the data lake
Big data architectures and the data lake
 
Five Things to Consider About Data Mesh and Data Governance
Five Things to Consider About Data Mesh and Data GovernanceFive Things to Consider About Data Mesh and Data Governance
Five Things to Consider About Data Mesh and Data Governance
 
Building Lakehouses on Delta Lake with SQL Analytics Primer
Building Lakehouses on Delta Lake with SQL Analytics PrimerBuilding Lakehouses on Delta Lake with SQL Analytics Primer
Building Lakehouses on Delta Lake with SQL Analytics Primer
 
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
 
Activate Data Governance Using the Data Catalog
Activate Data Governance Using the Data CatalogActivate Data Governance Using the Data Catalog
Activate Data Governance Using the Data Catalog
 
Data modelling 101
Data modelling 101Data modelling 101
Data modelling 101
 
Data Warehousing in the Cloud: Practical Migration Strategies
Data Warehousing in the Cloud: Practical Migration Strategies Data Warehousing in the Cloud: Practical Migration Strategies
Data Warehousing in the Cloud: Practical Migration Strategies
 
Data Architecture for Data Governance
Data Architecture for Data GovernanceData Architecture for Data Governance
Data Architecture for Data Governance
 
Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)
 
Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?
 
Introducing Databricks Delta
Introducing Databricks DeltaIntroducing Databricks Delta
Introducing Databricks Delta
 
Data Architecture Strategies: Data Architecture for Digital Transformation
Data Architecture Strategies: Data Architecture for Digital TransformationData Architecture Strategies: Data Architecture for Digital Transformation
Data Architecture Strategies: Data Architecture for Digital Transformation
 

Similar to Time to Talk about Data Mesh

Accelerating a Path to Digital with a Cloud Data Strategy
Accelerating a Path to Digital with a Cloud Data StrategyAccelerating a Path to Digital with a Cloud Data Strategy
Accelerating a Path to Digital with a Cloud Data StrategyMongoDB
 
Cloud what is the best model for vietnam
Cloud   what is the best model for vietnamCloud   what is the best model for vietnam
Cloud what is the best model for vietnamPhuc (Peter) Huynh
 
Cloud computing adoption in sap technologies
Cloud computing adoption in sap technologiesCloud computing adoption in sap technologies
Cloud computing adoption in sap technologiessveldanda
 
La creación de una capa operacional con MongoDB
La creación de una capa operacional con MongoDBLa creación de una capa operacional con MongoDB
La creación de una capa operacional con MongoDBMongoDB
 
2011.11.22 - Cloud Infrastructure Provider - 8ème Forum du Club Cloud des Par...
2011.11.22 - Cloud Infrastructure Provider - 8ème Forum du Club Cloud des Par...2011.11.22 - Cloud Infrastructure Provider - 8ème Forum du Club Cloud des Par...
2011.11.22 - Cloud Infrastructure Provider - 8ème Forum du Club Cloud des Par...Club Cloud des Partenaires
 
Big Data: It’s all about the Use Cases
Big Data: It’s all about the Use CasesBig Data: It’s all about the Use Cases
Big Data: It’s all about the Use CasesJames Serra
 
Technology Overview
Technology OverviewTechnology Overview
Technology OverviewLiran Zelkha
 
Conquering Disaster Recovery Challenges and Out-of-Control Data with the Hybr...
Conquering Disaster Recovery Challenges and Out-of-Control Data with the Hybr...Conquering Disaster Recovery Challenges and Out-of-Control Data with the Hybr...
Conquering Disaster Recovery Challenges and Out-of-Control Data with the Hybr...actualtechmedia
 
SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101
SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101
SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101Mithun T. Dhar
 
Azure Overview Csco
Azure Overview CscoAzure Overview Csco
Azure Overview Cscorajramab
 
Benefits of the Azure Cloud
Benefits of the Azure CloudBenefits of the Azure Cloud
Benefits of the Azure CloudCaserta
 
Azure Overview Arc
Azure Overview ArcAzure Overview Arc
Azure Overview Arcrajramab
 
Data Integration for Both Self-Service Analytics and IT Users
Data Integration for Both Self-Service Analytics and IT Users Data Integration for Both Self-Service Analytics and IT Users
Data Integration for Both Self-Service Analytics and IT Users Senturus
 
Cloud Modernization and Data as a Service Option
Cloud Modernization and Data as a Service OptionCloud Modernization and Data as a Service Option
Cloud Modernization and Data as a Service OptionDenodo
 
Cloud 12 08 V2
Cloud 12 08 V2Cloud 12 08 V2
Cloud 12 08 V2Pini Cohen
 
Serverless service adoption for Thailand
Serverless service adoption for ThailandServerless service adoption for Thailand
Serverless service adoption for ThailandWatcharin Yang-Ngam
 
Machine Learning for z/OS
Machine Learning for z/OSMachine Learning for z/OS
Machine Learning for z/OSCuneyt Goksu
 
Smart Integration to the Cloud - Kellton Tech Webinar
Smart Integration to the Cloud - Kellton Tech WebinarSmart Integration to the Cloud - Kellton Tech Webinar
Smart Integration to the Cloud - Kellton Tech WebinarKellton Tech Solutions Ltd
 
Denodo as the Core Pillar of your API Strategy
Denodo as the Core Pillar of your API StrategyDenodo as the Core Pillar of your API Strategy
Denodo as the Core Pillar of your API StrategyDenodo
 

Similar to Time to Talk about Data Mesh (20)

Accelerating a Path to Digital with a Cloud Data Strategy
Accelerating a Path to Digital with a Cloud Data StrategyAccelerating a Path to Digital with a Cloud Data Strategy
Accelerating a Path to Digital with a Cloud Data Strategy
 
Cloud what is the best model for vietnam
Cloud   what is the best model for vietnamCloud   what is the best model for vietnam
Cloud what is the best model for vietnam
 
Cloud computing adoption in sap technologies
Cloud computing adoption in sap technologiesCloud computing adoption in sap technologies
Cloud computing adoption in sap technologies
 
La creación de una capa operacional con MongoDB
La creación de una capa operacional con MongoDBLa creación de una capa operacional con MongoDB
La creación de una capa operacional con MongoDB
 
2011.11.22 - Cloud Infrastructure Provider - 8ème Forum du Club Cloud des Par...
2011.11.22 - Cloud Infrastructure Provider - 8ème Forum du Club Cloud des Par...2011.11.22 - Cloud Infrastructure Provider - 8ème Forum du Club Cloud des Par...
2011.11.22 - Cloud Infrastructure Provider - 8ème Forum du Club Cloud des Par...
 
Big Data: It’s all about the Use Cases
Big Data: It’s all about the Use CasesBig Data: It’s all about the Use Cases
Big Data: It’s all about the Use Cases
 
Technology Overview
Technology OverviewTechnology Overview
Technology Overview
 
Conquering Disaster Recovery Challenges and Out-of-Control Data with the Hybr...
Conquering Disaster Recovery Challenges and Out-of-Control Data with the Hybr...Conquering Disaster Recovery Challenges and Out-of-Control Data with the Hybr...
Conquering Disaster Recovery Challenges and Out-of-Control Data with the Hybr...
 
SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101
SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101
SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101
 
Forecast key1 0615_ak_evening
Forecast key1 0615_ak_eveningForecast key1 0615_ak_evening
Forecast key1 0615_ak_evening
 
Azure Overview Csco
Azure Overview CscoAzure Overview Csco
Azure Overview Csco
 
Benefits of the Azure Cloud
Benefits of the Azure CloudBenefits of the Azure Cloud
Benefits of the Azure Cloud
 
Azure Overview Arc
Azure Overview ArcAzure Overview Arc
Azure Overview Arc
 
Data Integration for Both Self-Service Analytics and IT Users
Data Integration for Both Self-Service Analytics and IT Users Data Integration for Both Self-Service Analytics and IT Users
Data Integration for Both Self-Service Analytics and IT Users
 
Cloud Modernization and Data as a Service Option
Cloud Modernization and Data as a Service OptionCloud Modernization and Data as a Service Option
Cloud Modernization and Data as a Service Option
 
Cloud 12 08 V2
Cloud 12 08 V2Cloud 12 08 V2
Cloud 12 08 V2
 
Serverless service adoption for Thailand
Serverless service adoption for ThailandServerless service adoption for Thailand
Serverless service adoption for Thailand
 
Machine Learning for z/OS
Machine Learning for z/OSMachine Learning for z/OS
Machine Learning for z/OS
 
Smart Integration to the Cloud - Kellton Tech Webinar
Smart Integration to the Cloud - Kellton Tech WebinarSmart Integration to the Cloud - Kellton Tech Webinar
Smart Integration to the Cloud - Kellton Tech Webinar
 
Denodo as the Core Pillar of your API Strategy
Denodo as the Core Pillar of your API StrategyDenodo as the Core Pillar of your API Strategy
Denodo as the Core Pillar of your API Strategy
 

More from LibbySchulze

Running distributed tests with k6.pdf
Running distributed tests with k6.pdfRunning distributed tests with k6.pdf
Running distributed tests with k6.pdfLibbySchulze
 
Extending Kubectl.pptx
Extending Kubectl.pptxExtending Kubectl.pptx
Extending Kubectl.pptxLibbySchulze
 
Enhancing Data Protection Workflows with Kanister And Argo Workflows
Enhancing Data Protection Workflows with Kanister And Argo WorkflowsEnhancing Data Protection Workflows with Kanister And Argo Workflows
Enhancing Data Protection Workflows with Kanister And Argo WorkflowsLibbySchulze
 
Fallacies in Platform Engineering.pdf
Fallacies in Platform Engineering.pdfFallacies in Platform Engineering.pdf
Fallacies in Platform Engineering.pdfLibbySchulze
 
Intro to Fluvio.pptx.pdf
Intro to Fluvio.pptx.pdfIntro to Fluvio.pptx.pdf
Intro to Fluvio.pptx.pdfLibbySchulze
 
CNCF On-Demand Webinar_ LitmusChaos Project Updates.pdf
CNCF On-Demand Webinar_ LitmusChaos Project Updates.pdfCNCF On-Demand Webinar_ LitmusChaos Project Updates.pdf
CNCF On-Demand Webinar_ LitmusChaos Project Updates.pdfLibbySchulze
 
Oh The Places You'll Sign.pdf
Oh The Places You'll Sign.pdfOh The Places You'll Sign.pdf
Oh The Places You'll Sign.pdfLibbySchulze
 
Rancher MasterClass - Avoiding-configuration-drift.pptx
Rancher  MasterClass - Avoiding-configuration-drift.pptxRancher  MasterClass - Avoiding-configuration-drift.pptx
Rancher MasterClass - Avoiding-configuration-drift.pptxLibbySchulze
 
vFunction Konveyor Meetup - Why App Modernization Projects Fail - Aug 2022.pptx
vFunction Konveyor Meetup - Why App Modernization Projects Fail - Aug 2022.pptxvFunction Konveyor Meetup - Why App Modernization Projects Fail - Aug 2022.pptx
vFunction Konveyor Meetup - Why App Modernization Projects Fail - Aug 2022.pptxLibbySchulze
 
CNCF Live Webinar: Low Footprint Java Containers with GraalVM
CNCF Live Webinar: Low Footprint Java Containers with GraalVMCNCF Live Webinar: Low Footprint Java Containers with GraalVM
CNCF Live Webinar: Low Footprint Java Containers with GraalVMLibbySchulze
 
EnRoute-OPA-Integration.pdf
EnRoute-OPA-Integration.pdfEnRoute-OPA-Integration.pdf
EnRoute-OPA-Integration.pdfLibbySchulze
 
AirGap_zusammen_neu.pdf
AirGap_zusammen_neu.pdfAirGap_zusammen_neu.pdf
AirGap_zusammen_neu.pdfLibbySchulze
 
Copy of OTel Me All About OpenTelemetry The Current & Future State, Navigatin...
Copy of OTel Me All About OpenTelemetry The Current & Future State, Navigatin...Copy of OTel Me All About OpenTelemetry The Current & Future State, Navigatin...
Copy of OTel Me All About OpenTelemetry The Current & Future State, Navigatin...LibbySchulze
 
OTel Me All About OpenTelemetry The Current & Future State, Navigating the Pr...
OTel Me All About OpenTelemetry The Current & Future State, Navigating the Pr...OTel Me All About OpenTelemetry The Current & Future State, Navigating the Pr...
OTel Me All About OpenTelemetry The Current & Future State, Navigating the Pr...LibbySchulze
 
CNCF_ A step to step guide to platforming your delivery setup.pdf
CNCF_ A step to step guide to platforming your delivery setup.pdfCNCF_ A step to step guide to platforming your delivery setup.pdf
CNCF_ A step to step guide to platforming your delivery setup.pdfLibbySchulze
 
CNCF Online - Data Protection Guardrails using Open Policy Agent (OPA).pdf
CNCF Online - Data Protection Guardrails using Open Policy Agent (OPA).pdfCNCF Online - Data Protection Guardrails using Open Policy Agent (OPA).pdf
CNCF Online - Data Protection Guardrails using Open Policy Agent (OPA).pdfLibbySchulze
 
Securing Windows workloads.pdf
Securing Windows workloads.pdfSecuring Windows workloads.pdf
Securing Windows workloads.pdfLibbySchulze
 
Securing Windows workloads.pdf
Securing Windows workloads.pdfSecuring Windows workloads.pdf
Securing Windows workloads.pdfLibbySchulze
 
Advancements in Kubernetes Workload Identity for Azure
Advancements in Kubernetes Workload Identity for AzureAdvancements in Kubernetes Workload Identity for Azure
Advancements in Kubernetes Workload Identity for AzureLibbySchulze
 
Containerized IDEs.pdf
Containerized IDEs.pdfContainerized IDEs.pdf
Containerized IDEs.pdfLibbySchulze
 

More from LibbySchulze (20)

Running distributed tests with k6.pdf
Running distributed tests with k6.pdfRunning distributed tests with k6.pdf
Running distributed tests with k6.pdf
 
Extending Kubectl.pptx
Extending Kubectl.pptxExtending Kubectl.pptx
Extending Kubectl.pptx
 
Enhancing Data Protection Workflows with Kanister And Argo Workflows
Enhancing Data Protection Workflows with Kanister And Argo WorkflowsEnhancing Data Protection Workflows with Kanister And Argo Workflows
Enhancing Data Protection Workflows with Kanister And Argo Workflows
 
Fallacies in Platform Engineering.pdf
Fallacies in Platform Engineering.pdfFallacies in Platform Engineering.pdf
Fallacies in Platform Engineering.pdf
 
Intro to Fluvio.pptx.pdf
Intro to Fluvio.pptx.pdfIntro to Fluvio.pptx.pdf
Intro to Fluvio.pptx.pdf
 
CNCF On-Demand Webinar_ LitmusChaos Project Updates.pdf
CNCF On-Demand Webinar_ LitmusChaos Project Updates.pdfCNCF On-Demand Webinar_ LitmusChaos Project Updates.pdf
CNCF On-Demand Webinar_ LitmusChaos Project Updates.pdf
 
Oh The Places You'll Sign.pdf
Oh The Places You'll Sign.pdfOh The Places You'll Sign.pdf
Oh The Places You'll Sign.pdf
 
Rancher MasterClass - Avoiding-configuration-drift.pptx
Rancher  MasterClass - Avoiding-configuration-drift.pptxRancher  MasterClass - Avoiding-configuration-drift.pptx
Rancher MasterClass - Avoiding-configuration-drift.pptx
 
vFunction Konveyor Meetup - Why App Modernization Projects Fail - Aug 2022.pptx
vFunction Konveyor Meetup - Why App Modernization Projects Fail - Aug 2022.pptxvFunction Konveyor Meetup - Why App Modernization Projects Fail - Aug 2022.pptx
vFunction Konveyor Meetup - Why App Modernization Projects Fail - Aug 2022.pptx
 
CNCF Live Webinar: Low Footprint Java Containers with GraalVM
CNCF Live Webinar: Low Footprint Java Containers with GraalVMCNCF Live Webinar: Low Footprint Java Containers with GraalVM
CNCF Live Webinar: Low Footprint Java Containers with GraalVM
 
EnRoute-OPA-Integration.pdf
EnRoute-OPA-Integration.pdfEnRoute-OPA-Integration.pdf
EnRoute-OPA-Integration.pdf
 
AirGap_zusammen_neu.pdf
AirGap_zusammen_neu.pdfAirGap_zusammen_neu.pdf
AirGap_zusammen_neu.pdf
 
Copy of OTel Me All About OpenTelemetry The Current & Future State, Navigatin...
Copy of OTel Me All About OpenTelemetry The Current & Future State, Navigatin...Copy of OTel Me All About OpenTelemetry The Current & Future State, Navigatin...
Copy of OTel Me All About OpenTelemetry The Current & Future State, Navigatin...
 
OTel Me All About OpenTelemetry The Current & Future State, Navigating the Pr...
OTel Me All About OpenTelemetry The Current & Future State, Navigating the Pr...OTel Me All About OpenTelemetry The Current & Future State, Navigating the Pr...
OTel Me All About OpenTelemetry The Current & Future State, Navigating the Pr...
 
CNCF_ A step to step guide to platforming your delivery setup.pdf
CNCF_ A step to step guide to platforming your delivery setup.pdfCNCF_ A step to step guide to platforming your delivery setup.pdf
CNCF_ A step to step guide to platforming your delivery setup.pdf
 
CNCF Online - Data Protection Guardrails using Open Policy Agent (OPA).pdf
CNCF Online - Data Protection Guardrails using Open Policy Agent (OPA).pdfCNCF Online - Data Protection Guardrails using Open Policy Agent (OPA).pdf
CNCF Online - Data Protection Guardrails using Open Policy Agent (OPA).pdf
 
Securing Windows workloads.pdf
Securing Windows workloads.pdfSecuring Windows workloads.pdf
Securing Windows workloads.pdf
 
Securing Windows workloads.pdf
Securing Windows workloads.pdfSecuring Windows workloads.pdf
Securing Windows workloads.pdf
 
Advancements in Kubernetes Workload Identity for Azure
Advancements in Kubernetes Workload Identity for AzureAdvancements in Kubernetes Workload Identity for Azure
Advancements in Kubernetes Workload Identity for Azure
 
Containerized IDEs.pdf
Containerized IDEs.pdfContainerized IDEs.pdf
Containerized IDEs.pdf
 

Recently uploaded

NANOG 90: 'BGP in 2023' presented by Geoff Huston
NANOG 90: 'BGP in 2023' presented by Geoff HustonNANOG 90: 'BGP in 2023' presented by Geoff Huston
NANOG 90: 'BGP in 2023' presented by Geoff HustonAPNIC
 
Elevate Your Business: Unleashing Collaboration and Efficiency through Expert...
Elevate Your Business: Unleashing Collaboration and Efficiency through Expert...Elevate Your Business: Unleashing Collaboration and Efficiency through Expert...
Elevate Your Business: Unleashing Collaboration and Efficiency through Expert...Prometix Pty Ltd
 
DNS-OARC 42: Is the DNS ready for IPv6? presentation by Geoff Huston
DNS-OARC 42: Is the DNS ready for IPv6? presentation by Geoff HustonDNS-OARC 42: Is the DNS ready for IPv6? presentation by Geoff Huston
DNS-OARC 42: Is the DNS ready for IPv6? presentation by Geoff HustonAPNIC
 
Regulation is Coming - Trusted Media Summit 2023
Regulation is Coming - Trusted Media Summit 2023Regulation is Coming - Trusted Media Summit 2023
Regulation is Coming - Trusted Media Summit 2023Damar Juniarto
 
Model Jaringan network jaringan komputer.pdf
Model Jaringan network jaringan komputer.pdfModel Jaringan network jaringan komputer.pdf
Model Jaringan network jaringan komputer.pdfgalfinprihardiputra0
 
Biometrics Technology Intresting PPT
Biometrics Technology Intresting PPTBiometrics Technology Intresting PPT
Biometrics Technology Intresting PPTPraveenKumarThota7
 

Recently uploaded (6)

NANOG 90: 'BGP in 2023' presented by Geoff Huston
NANOG 90: 'BGP in 2023' presented by Geoff HustonNANOG 90: 'BGP in 2023' presented by Geoff Huston
NANOG 90: 'BGP in 2023' presented by Geoff Huston
 
Elevate Your Business: Unleashing Collaboration and Efficiency through Expert...
Elevate Your Business: Unleashing Collaboration and Efficiency through Expert...Elevate Your Business: Unleashing Collaboration and Efficiency through Expert...
Elevate Your Business: Unleashing Collaboration and Efficiency through Expert...
 
DNS-OARC 42: Is the DNS ready for IPv6? presentation by Geoff Huston
DNS-OARC 42: Is the DNS ready for IPv6? presentation by Geoff HustonDNS-OARC 42: Is the DNS ready for IPv6? presentation by Geoff Huston
DNS-OARC 42: Is the DNS ready for IPv6? presentation by Geoff Huston
 
Regulation is Coming - Trusted Media Summit 2023
Regulation is Coming - Trusted Media Summit 2023Regulation is Coming - Trusted Media Summit 2023
Regulation is Coming - Trusted Media Summit 2023
 
Model Jaringan network jaringan komputer.pdf
Model Jaringan network jaringan komputer.pdfModel Jaringan network jaringan komputer.pdf
Model Jaringan network jaringan komputer.pdf
 
Biometrics Technology Intresting PPT
Biometrics Technology Intresting PPTBiometrics Technology Intresting PPT
Biometrics Technology Intresting PPT
 

Time to Talk about Data Mesh

  • 1. It is time to talk about DATA MESH Distributed Data Management for Microservices
  • 3. 3 THE CHALLENGES OF MICROSERVICE DESIGN TRANSFORM BUSINESS LOGIC AND DATA INTO MICROSERVICE ARCHITECTURE Business flow combing, Function and service separating Data decoupling and fragmentation management ◼ DDD (Domain-Driven Design) ◼ Split services by business flow ◼ EDA (Event-driven Architecture) ◼ Microservice component design ◼ Microservice horizontal scaling strategies ◼ Separating data by business flow ◼ Sharing data across domains ◼ Vast data deployment and caching strategies ◼ Data consistency(Data fragmentation management) ◼ The final data consistent ◼ Strong data consistent ◼ Absolute data consistent CHALLENGE DONE WELL Business Logic Data CELL GATEWAY Database Service Service Analyze Decouple Aggregate To implement the microservice architecture with decision of design patterns, deployment and maintenance strategies, integration of technical components, planning and actual software architectures. Implement API SERVICES/ EDGE SERVICES COMPOSITE SERVICES/ INTEGRATION SERVICES CORE SERVICES/ ATOMIC SERVICES CLIENT APPLICATIONS To implement the design and choose an immediately available solution to implement various microservice design patterns and basic architectures. Based on stable software components and platforms, determine service roles and implement business logic. Special design with stateful atomic services
  • 4. 4 TRANSFORM TO DATABASE PER SERVICE For Microservice Architecture App service App service App service DATABASE App service App service App service Database per service Shared Database How? Risk? Performance? Plan?
  • 5. 5 PROBLEMS WITH DATABASE PER SERVICE WHEN WE IMPLEMENT IT App service App service App service Round-trips Problem Busy App service App service App service Anti-Pattern App service App service App service Confusion OR OR How do we solve these problems ?
  • 6. 6 PROVISION ONCE, SNAPSHOT MANY CACHEING FOR MULTIPLE SNAPSHOTS CLIENT APPLICATIONS Service API SNAPSHOT Service API DATA FLOW Source VIRTIAL TABLE VIRTUAL TABLE BUSINESS BUSINESS 100M Records 100M Records 100M Records Service Owner According to requirements of business task, selecting database, table and fields to make a copy into a virtual table. Update Database per service 1 Provision 2 3 Program A Program B
  • 7. 7 The scale of the supply pipelines and the degree of deployment efficiency THE KEY TO THE SMOOTH IMPLEMENTATION OF MICROSERVICE ARCHITECTURE
  • 8. 8 MANAGE DATA SUPPLY CHAIN WITH DATA PLATFORM FAST TO BUILD DATA SUPPLY CHAIN FOR APPLICATIONS CLIENT APPLICATIONS Service API SNAPSHOT Service API DATA PLATFORM Source VIRTIAL TABLE VIRTUAL TABLE BUSINESS BUSINESS DATA PROVISION AND CACHING 100M Records 100M Records 100M Records Service Owner According to requirements of business task, selecting database, table and fields to make a copy into a virtual table. Update Database per service 1 Provision 2 4 Caching 3
  • 9. 9 App App APPLICATION-DRIVEN DATA SUPPLY DISTRIBUTED DATA INTEGRATION VIRTUAL TABLE App App App App DATA MESH VIRTUAL TABLE VIRTUAL TABLE VIRTUAL TABLE VIRTUAL TABLE VIRTUAL TABLE DATABASE DATA SOURCE CUSTOMIZATION DATA SOURCE FILES DATA SOURCE
  • 10. 10 On-Premise APPLICATIONS WHY NOT THE TRADITIONAL SOLUTIONS? CENTRALIZED DATA MANAGEMENT CANNOT FIT TO MICROSERVICE ARCHITECTURE Public FILES DATABASE CUSTOMIZATION DATA SOURCE DATA SOURCE DATA SOURCE Cloud Private Cloud Cloud VIRTUAL TABLE Centralized Data Process VIRTUAL TABLE VIRTUAL TABLE Traditional ETL, Data Virtualization and Data Warehouse encounter problems with centralized system design. ETL Batch Polling
  • 11. 11 On-Premise APPLICATIONS DATA MESH CONCEPT DISTRIBUTED DATA MANAGEMENT WITH DATA MESH Public FILES DATABASE DATA SOURCE CUSTOMIZATION DATA SOURCE DATA SOURCE Cloud Private Cloud Cloud VIRTUAL TABLE VIRTUAL TABLE VIRTUAL TABLE DATA MESH Data Mesh focusing on supplying data to applications as fast as possible with distributed system design. Provision Provision Provision
  • 12. HOW TO BUILD DATA MESH INFRASTRUCTURE 02
  • 13. 13 Data Fabric HOW TO BUILD DATA MESH INFRASTRUCTURE Data Virtualization and Relay Data Caching Presentation Deployment and Management Data Source Data Mesh Data Provision Infrastructure Services Business ◼ Provision and manage data with pipelines ◼ Establish data relay mechanism(Caching) ◼ Rapidly deployment and management data pipeline clusters(Container) ◼ Manage vast multiplexed data channels and pipelines (Multiplexer) Objective Applications Legacy and others Service BUSINESS Service BUSINESS Service BUSINESS Service BUSINESS Service BUSINESS Service BUSINESS Service BUSINESS Service BUSINESS Last Mile Infra needs to be able to quickly deploy all data management mechanisms based on application requirements.
  • 14. 14 BUILD DATA NODE BASED ON KUBERNETES YAML WAY TO DEPLOY DATA NODE AND PIPELINE INFRSTRUCTURE Data Mesh Controller YAML Relay Database Relay Database Relay Database YAML YAML Service BUSINESS Service BUSINESS Service BUSINESS Service BUSINESS Service BUSINESS Service BUSINESS Container Platform Orchestrator Kubernetes ◼ Declarative pipeline deployment with integrated container platform ◼ Meet the demand for a large number of microservice data supply ◼ Manage and maintain a large number of provision pipelines Objective DATA PLATFORM Hybrid Clouds
  • 15. 15 Using YAML SCENARIO: Starting from event data provision, caching, building virtual table, and quickly generate APIs. No need to write a single line of program, build the entire framework in a few minutes, from Oracle to MongoDB and generate usable data API SERVICE CDC Protocol Data Platform Kubernetes cluster VM / Container Presenter component Event changed Intercept database change events The data source can be an application or a database Select the required data fields Other applications Customized Query from cached data Deploy a pipeline mechanisms Caching API Auto generated Imported from templates 1 Apply API settings 2 3 CLIENT APPLICATIONS DATA SOURCE INFRA AND PLATFORM DATA PROVISION AND CACHING 11g R2
  • 16. 16 DATABASE RESULTS Legacy Program USE CASE 1: Improve the efficiency of data interface to the legacy system SOLVE PROBLEM THAT LEGACY SYSTEM BRINGS ON LOW LATENCY AND INTEGRATION ISSUES DATABASE DATA SOURCE Query Storage Program DATABASE RESULTS Polling Export Insert File DATABASE DATA SOURCE DATA MESH Event Update Real-time and Low Latency Provision and Caching
  • 17. 17 B Batch DATABASE RESULTS USE CASE 2: Solve the inefficiency and performance impact of various batch processing SOLVE PROBLEM THAT INTEGRATION ISSUES DATABASE DATA SOURCE Query A Program DATABASE RESULTS Insert DATABASE DATA SOURCE DATA MESH NODE Provision Supply to Multiple Applications without Performance Impact A Service DATA MESH PIPELINE Push DATA MESH PIPELINE B Service Push Parallel Caching A Batch Query and Extract B Program A B Subscribe Subscribe Query and Extract Insert
  • 18. 18 Program SNAPSHOT USE CASE 3: High-speed parallel processing of the aggregation and correlation of multiple data sources USE CASE: INTEGRATION WITH MULTIPLE SOURCES IT DATA SOURCES OT DATA SOURCES Program Output Aggregate and Transform Round-Trip Problem Round-Trip Problem IT DATA SOURCES OT DATA SOURCES DATA MESH NODE Event DATA MESH NODE Event DATA MESH PIPELINE Present Aggregate Snapshot Snapshot Output Transform Aggregation in Parallel for High Performance Parallel Caching Caching
  • 19. 19 Program SNAPSHOT USE CASE 4: Integration of hybrid cloud, cross-site and cross- environment USE CASE: INTEGRATION FOR HYBRID CLOUD IT DATA SOURCES OT DATA SOURCES Program Output Aggregate and Transform Round-Trip Problem Round-Trip Problem IT DATA SOURCES OT DATA SOURCES DATA MESH NODE Event DATA MESH NODE Event DATA MESH NODE Present Aggregate Snapshot Snapshot Output Transform Aggregation Across Clouds With Parallel Caching Caching On-Premise Public Cloud External Site On-Premise Public Cloud External Site
  • 21. About Brobridge Brobridge refers to Broad Bridge Co., Ltd. (a limited company formed under the laws of Taiwan). Please refer to www.brobridge.com for a detailed description. Brobridge Co., Ltd. provides information technology consulting, application development, IT infrastructure design, consulting services and system maintenance services to listed and unlisted clients in various industries. Brobridge Co., Ltd. recruits the most professionals in the industry, is committed to the pursuit of excellence and setting an example. Brobridge Co., Ltd. Brobridge refers to Brobridge Co., Ltd.. Brobridge Co., Ltd. enjoys a good reputation in the industry for its excellent customer service, excellent talents, perfect training and rigorous development technology. Through the resources of Brobridge Co., Ltd., we provide a full range of services for customer information systems, including virtual desktop systems, multi-cloud integration, IT infrastructure planning and design, and advanced cutting-edge program development. This publication is compiled based on general information and is for readers' reference only. Brobridge Co., Ltd. is not deemed to provide professional advice or services to anyone because of this publication. Any individual of Brobridge Co., Ltd. shall not be held responsible for any loss caused by reliance on this publication. © 2016 Broad Bridge Co., Ltd. All rights reserved. 寬橋 THANK YOU!