With Industry 4.0, several technologies are used to have data analysis in real-time, maintaining, organizing, and building this on the other hand is a complex and complicated job. Over the past 30 years, we saw several ideas to centralize the database in a single place as the united and true source of data has been implemented in companies, such as Data wareHouse, NoSQL, Data Lake, Lambda & Kappa Architecture.
On the other hand, Software Engineering has been applying ideas to separate applications to facilitate and improve application performance, such as microservices.
The idea is to use the MicroService patterns on the date and divide the model into several smaller ones. And a good way to split it up is to use the model using the DDD principles. And that's how I try to explain and define DataMesh & Data Fabric.
4. During2020/ 2021the
world continuesto go
throughaParadigmShift
into afuture where“Cyber-
PhysicalSystems”arethe
newnormal.
“Digital Transformation”
requires mindset shift:
1.Sharingdatais more
effective thanaccumulating
2.Decentralizing,distributing,
andcopyingis more
powerful than stockpiling
3.Connectivityandflow of
datais the starting point for
innovation andsocializing.
16. What is a Data Mesh?
16
Microservice
Patterns
Log-based
Integrations
Polyglot Data
Movement
Data Mesh is a data-tier architecture to integrate and
govern enterprise data assets across distributed multi-cloud
environments – two defining characteristics are:
(1) De-centralized data processing; no ETL/Hubs/Lake monoliths
(2) Event-driven; real-time where possible, batch only when necessary
Microservices-centric:
• For the administration, deployment and monitoring of the core
frameworks of data movement and governance
• “Sidecar Proxy” style pattern for Events and Data; Aligns with
Service Mesh frameworks (Kubernetes, Istio, etc)
Immutable event-logs for data integrations:
• Messaging and data store events are globally accessible via
immutable event logs
• Logs may be used to drive Streaming or Batch integrations
Distributed data movement of all types of data
• A data mesh moves data: Relational, NoSQL, JSON, Graph…
• Relational data consistency (ACID) during data movement
• Must work reliably with enterprise OLTP data sets
Data
Mesh
Event
Streaming
Immutable
Logs
Data
Replication
Polyglot
Persistence
Edge / 5G
Frameworks
Domain
Driven
Design
Service Mesh
“Sidecars”
Data
Mesh
21. Microservice Design Patterns for Data
Patterns for MicroservicesInherent to the Microservice Architecture is the developer
using specific patterns, sometimes the patterns are partially
embodied in a Programming Framework, but typically the
developers must choose to follow certain heuristics while
programming.
This presentation’s focus:
• “Database Patterns” & “Integration Patterns” …using DBEvent
Replication (AKA: Change Data Capture) to improvethem
• Simplify the pattern, make the microservice application more resilient
and provide better data consistency guarantees
DB Patterns for Discussion:
• Database per Service (coveredearlier)
• CQRS – Command Query Responsibility Segregation
• Event Sourcing
• Saga Pattern
• Transactional Outbox
• Aggregates (AKA: Domain Events)
Transaction
Outbox
21
24. Stream Processing/CEP for Event Driven Architectures
There has been a widespread
awakening to the benefits of Event
Drive Architecture (EDA) for
increasing the scalability and agility of
business systems. […] Stream
analytics is based on the mathematics
of complex-event processing (CEP).
CEP is a computing technique in
which incoming data about what is
happening (event data) is processed
as it arrives (data in motion or
recently in motion) to generate
higher level, more useful, summary
information (complex events).
W. Roy Schulte (of Gartner), March 2020:
EDA is Suddenly Popular Will Stream Analytics be Next?
Event Stream Analytics (& CEP)
Data & Microservice Events
Event/Data
Pipelines
Time-Series
Analysis
Geospatial
Analysis
Real-time
AI/ML
Continious
ETL
Use Cases:
28. Critiques of Event Sourcing
Exposing the Persistence Tier:
• Taken too far (Why Event Sourcing is an Anti-Pattern), developers wind up usingthe
Event Store as a Shared Persistence model, and other microservice now have hard-
coupled binding to the message formats of the originatingservice
Whole System Fallacy:
• Some microservices leaders (Udi and Greg Reach CQRS Agreement) sayto narrow
the aperture on when to use CQRS + Event Sourcing → only within a Business
Component and a Single Bounded Context
• Minimizes utility of pattern for Communications
Forcing Eventual Consistency on Developers:
• The propensity to over-use CQRS & Event Sourcing at the at the whole systemlevel
forces developers to manage eventual consistency in the Application tiers (What
they don’t tell you about eventsourcing)
• “…they will make your life a living hell” doing DevOps, debugging and
system recovery when a “Mesh” of services are interacting via Event Store and
message signatures can lead todisaster
41. This is not a Metamorphosis, it is a Paradigm Shift
Data success factors that did wellin
Industry 3.0will not be the factors that
create success in Industry4.0
The Success Paradox Next Gen DataArchitecture
ETL Vendors
1990 –2010’s Gen1:
• Replication
• Messaging
• Streaming
• Pipelines
Next-Genhas
newDNAnot
tiedto oldETL tools
Itis impossible to evolve older Batch Processing
tools into a modern Event- Centric Stream
Processing solution; the underlying paradigms
arefundamentally different
41