SlideShare a Scribd company logo
© 2023 Thoughtworks
Data Mesh 101
Chris Ford & Pablo Porto
© 2023 Thoughtworks
I am Head of Technology for Thoughtworks
Spain. I help clients with architecture, agile
development and organisational
effectiveness.
I was a technical reviewer for Zhamak
Dehghani’s 2022 book 'Data Mesh'.
2
I am a Lead Developer for Thoughtworks
Spain's Data and AI Service Line. I help
clients build distributed systems, platforms
and data architectures.
I am currently working as part of one of the
most mature Data Mesh implementations in
the healthcare industry.
2
© 2023 Thoughtworks
Chris Ford Pablo Porto
© 2023 Thoughtworks
Part zero: Introduction
3
3
© 2023 Thoughtworks 4
Promise
What is the value
proposition of Data
Mesh?
Principles
What are the core
elements of Data
Mesh?
Practicalities
Why is it important
and how can you get
started?
Structure of this talk
Table of contents
© 2023 Thoughtworks
“All models are wrong but
some are useful.”
5
George Box, Statistician
Image from Wikipedia
5
© 2022 Thoughtworks
© 2023 Thoughtworks
(domain-driven design + microservices)
× data
= Data Mesh
sort of...
6
© 2023 Thoughtworks
Part one: Promise
7
7
© 2023 Thoughtworks 8
If we fulfill these conditions
Architectural paradigms
We can obtain these benefits
So long as we don’t mind these costs
8
© 2022 Thoughtworks
What is the cost/benefit?
In combination, a paradigm’s elements define a kind of promise:
The promise only pays off when the conditions are satisfied, the benefits are valued
and the costs are acceptable.
© 2023 Thoughtworks
It’s usually more valuable to consider
“Does this paradigm apply in this context?”
than to try and decide
“Is this paradigm good?”
9
9
© 2023 Thoughtworks
© 2023 Thoughtworks 10
Microservices promise
10
© 2022 Thoughtworks
If we break things into small pieces
We can change them independently
So long as we don’t mind the added
integration complexity
© 2023 Thoughtworks 11
Domain-driven design promise
11
© 2022 Thoughtworks
If we align our architecture with our
business domain
We can represent our business
accurately, even as it changes
So long as we don’t mind adapting the
technology to the business domain
© 2023 Thoughtworks 12
Data warehouse promise
12
© 2022 Thoughtworks
If we model our data up front in a central
schema
We can run analytical queries across our
business
So long as we don’t mind the effort of
aggregating and reconciling it
E
T
L
© 2023 Thoughtworks 13
Data lake promise
13
© 2022 Thoughtworks
If we collect our raw data into a central
location
We can post-hoc run queries about
anything we like
So long as we don’t mind dependencies
being implicit
E
T
L
E
T
L
© 2023 Thoughtworks 14
1.
Data mesh promise
14
© 2022 Thoughtworks
If we give responsibility for data to the
people who produce it
We can rapidly incorporate new data
sources and use cases
So long as we don’t mind distributing
skills and investing in self-service
infrastructure
© 2023 Thoughtworks
Part two: Principles
15
15
© 2023 Thoughtworks
Data mesh
16
Principles
Domain
ownership
Self-serve data
platform
Data as a
product
Federated
computational
governance
© 2023 Thoughtworks
Domain ownership
17
Principles
Domain
ownership
What is it?
You give the originators of data authority over it and the responsibility
to make it easily and usefully available.
Where’s the value?
This reduces the distance between producer and consumer (in terms of
handoffs), enabling quicker, easier and richer consumption.
© 2023 Thoughtworks
Data as a product
18
Principles
Data as a
product
What is it?
You make your data available as products that are designed around the
needs of its consumers.
Where’s the value?
By framing things in terms of product, you put the emphasis on use and
give clear responsibility on the owners to fix anything that interferes
with use (and credit for working on anything that promotes use).
© 2023 Thoughtworks
Self-serve data platform
19
Principles
Self-serve
data
platform
What is it?
You have the ability to provision infrastructure for the creation and
consumption of data products without a human in the loop.
Where’s the value?
If you want to decrease lead time and enable on-demand changes, you
need self-service.
© 2023 Thoughtworks
Federated computational governance
20
Principles
Federated
computational
governance
What is it?
You have general guidance that make the requirements of good
citizenship clear to everyone in the mesh, leaning on automation
whenever possible.
Where’s the value?
By managing with constraints, rather than inspecting individual items,
we empower teams. By using automation, we scale, reduce handoffs
and encourage interoperability.
© 2023 Thoughtworks
Data mesh
21
Principles
Domain
ownership
Self-serve data
platform
Data as a
product
Federated
computational
governance
© 2023 Thoughtworks
Data mesh
22
Principles
Domain
ownership
Self-serve data
platform
Data as a
product
Federated
computational
governance
Team
Organisation
© 2023 Thoughtworks
Part three: Practicalities
23
23
© 2023 Thoughtworks
Why is now a time to care about data mesh?
Industry context
Rapidly developing new data use cases is
increasingly important to businesses.
The upside of Data Mesh is going up.
24
Data skills and infrastructure are
increasingly accessible as technology
advances.
The downside of Data Mesh is going
down.
© 2023 Thoughtworks
Glovo is a delivery service facing
strong competition in a market where
innovation and operational efficiency
is key to success.
With the pandemic, consumer
behaviours and expectations
changed.
Understanding customers’ needs and
the capability to quickly react to them
are crucial to increase customer
loyalty and satisfaction.
An execution
plan on how to
deliver on the
vision
A clear vision for
data in the
organisation
25
© 2023 Thoughtworks
Lesson learned #2
Introduce clear rules about what aspects
of data quality to measure and
communicate.
26
“A data platform that
supports data products
and business insights”
Lesson learned #1
Ensure that every system that produces
or transforms important data has a clear
owner who is taking care of it.
© 2023 Thoughtworks
© 2023 Thoughtworks
Roche is one of the biggest life
sciences and healthcare companies
in the world with over 100k
employees and operating in multiple
markets.
Roche wanted to unlock its potential
on the market leveraging the rich and
abundant data it possesses.
They chose Data Mesh as the
approach and Thoughtworks as a
partner to achieve their vision.
Access to data
is hard due to
technical and
organisational
constraints
Data
interoperability
becomes hard at
scale with
multiple data
platforms
27
© 2023 Thoughtworks
Lesson learned #2
Show value frequently, both to business
stakeholders and developers using your
platform.
28
Focus on incremental value
Lesson learned #1
Start your journey with your existing
organisational boundaries.
© 2023 Thoughtworks
© 2023 Thoughtworks
How to get started
What combination of data products is needed to
serve these use cases?
What is the thinnest viable platform that is
practical to support these data products?
What is the minimum viable cooperation we
need to work successfully together?
Use cases
Data products
Self-serve data platform
Governance
What are good candidates use case with real
and achievable value?
© 2023 Thoughtworks 30
Questions?
(We are hiring Senior and Lead Data Engineers!)
(We are remote-friendly and ordinary friendly too)
© 2023 Thoughtworks
Resources
31
31
© 2023 Thoughtworks
References
● Original How to Move Beyond a Monolithic Data Lake to a
Distributed Data Mesh article that introduced the concept
by Zhamak Dehghani
● Follow-up Data Mesh Principles and Logical Architecture
article by Zhamak Dehghani
● Data Mesh in practice: Getting off to the right start article
series about Roche’s Data Mesh journey by Ammara Gafoor,
Ian Murdoch and Kiran Prakash
● Data mesh: it's not just about tech, it's about ownership and
communication article series about Glovo’s Data Mesh
journey by Jorge Agudo, Narek Verdian, Óscar Torres
Fernández, Pablo Giner, Diana Pinto and Javier García.
● Data Mesh Accelerate workshop description by Steve Upton
and Paulo Caroli
32
WHERE TO GO NEXT
Data Mesh book by
Zhamak Dehghani
© 2023 Thoughtworks
Chris Ford
Head of Technology, Thoughtworks Spain
linkedin.com/in/ctford
twitter.com/ctford
chris.ford@thoughtworks.com
33
Thank you!
Pablo Porto
Lead Developer, Thoughtworks Spain
linkedin.com/in/pabloportoveloso
twitter.com/portovep
pablo.porto@thoughtworks.com
© 2023 Thoughtworks
“We’ve only got three teams”
Threat to applicability 1: no scale
Data Mesh gives responsibility for
data to its originators.
34
This is great for reducing the
distance between data producer
and data consumer.
BUT...
If your organisation is so small that distance between data producer and data consumer
is naturally short, there’s not much point investing in reducing it.
© 2023 Thoughtworks
“Data’s not key to our business right now”
Threat to applicability 2: no value
Data Mesh makes new data use
cases quicker and easier.
35
This is great for experimentation
and for getting new data use cases
to market quickly.
BUT...
If data use cases are not of value to your business, there’s not much point investing in
enabling them.
© 2023 Thoughtworks
“We’re not into flow and autonomy”
Threat to applicability 3: no flow
Data Mesh enables autonomous
change within domains.
36
This is great for achieving fast
flow, though it requires
investment to pull it off.
BUT...
If your culture or context is not prepared to take advantage of team autonomy, there’s
not much point investing in enabling it.
© 2023 Thoughtworks
Myth 1: infra explosion
37
“The infrastructure will be too expensive”
● Data Mesh emphasises that domains should own their own data.
● Some people interpret that as meaning that they need to provision independent
infrastructure for each domain, which could lead to a cost blowout.
● However, Data Mesh only requires that infrastructure be logically separate and
self-service, so multi tenant infrastructure used by multiple domains is fine.
Data Mesh is opinionated about how you organise infrastructure, not how you
provision it.
© 2023 Thoughtworks
Myth 2: data castle
38
“Data quality will suffer”
● Many organisations have data quality assurance processes where experts own the
data lake or data warehouse and inspect changes to datasets.
● In Data Mesh, governance happens via policy, not by inspection of individual
additions, alterations or consumptions of datasets / data products.
● This works by reducing distance between data producer and data consumer and
giving experts greater leverage by governing the process.
Data Mesh has an alternative way of ensuring data quality, delivering potentially
richer and deeper quality to consumers.
© 2023 Thoughtworks
Myth 3: over enthusiasm
39
“Data mesh will solve everything”
● Operational systems manage transactional data that changes in real time.
● Analytical systems manage a view of the facts of the business over time.
● Data Mesh is an architectural paradigm specifically aimed at analytical problems.
● Organisations that will benefit from Data Mesh likely have other problems in their
operational systems, but they have other solutions.
Data Mesh aims to bring operational and analytical systems into harmony, not to
conflate them.

More Related Content

What's hot

Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
Databricks
 
Data Architecture Strategies: Data Architecture for Digital Transformation
Data Architecture Strategies: Data Architecture for Digital TransformationData Architecture Strategies: Data Architecture for Digital Transformation
Data Architecture Strategies: Data Architecture for Digital Transformation
DATAVERSITY
 
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Dr. Arif Wider
 
Enabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationEnabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data Virtualization
Denodo
 
Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2
Databricks
 
Building Lakehouses on Delta Lake with SQL Analytics Primer
Building Lakehouses on Delta Lake with SQL Analytics PrimerBuilding Lakehouses on Delta Lake with SQL Analytics Primer
Building Lakehouses on Delta Lake with SQL Analytics Primer
Databricks
 
Data Catalog as the Platform for Data Intelligence
Data Catalog as the Platform for Data IntelligenceData Catalog as the Platform for Data Intelligence
Data Catalog as the Platform for Data Intelligence
Alation
 
Best Practices in DataOps: How to Create Agile, Automated Data Pipelines
Best Practices in DataOps: How to Create Agile, Automated Data PipelinesBest Practices in DataOps: How to Create Agile, Automated Data Pipelines
Best Practices in DataOps: How to Create Agile, Automated Data Pipelines
Eric Kavanagh
 
Data Governance Trends - A Look Backwards and Forwards
Data Governance Trends - A Look Backwards and ForwardsData Governance Trends - A Look Backwards and Forwards
Data Governance Trends - A Look Backwards and Forwards
DATAVERSITY
 
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
DataScienceConferenc1
 
Data as a Product by Wayne Eckerson
Data as a Product by Wayne EckersonData as a Product by Wayne Eckerson
Data as a Product by Wayne Eckerson
Zoomdata
 
Data mesh
Data meshData mesh
Data mesh
ManojKumarR41
 
Data Governance and Metadata Management
Data Governance and Metadata ManagementData Governance and Metadata Management
Data Governance and Metadata Management
DATAVERSITY
 
DW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptxDW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptx
Databricks
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)
James Serra
 
Learn to Use Databricks for Data Science
Learn to Use Databricks for Data ScienceLearn to Use Databricks for Data Science
Learn to Use Databricks for Data Science
Databricks
 
How to govern and secure a Data Mesh?
How to govern and secure a Data Mesh?How to govern and secure a Data Mesh?
How to govern and secure a Data Mesh?
confluent
 
Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3
Jeffrey T. Pollock
 
Data Catalogs Are the Answer – What is the Question?
Data Catalogs Are the Answer – What is the Question?Data Catalogs Are the Answer – What is the Question?
Data Catalogs Are the Answer – What is the Question?
DATAVERSITY
 
How a Semantic Layer Makes Data Mesh Work at Scale
How a Semantic Layer Makes  Data Mesh Work at ScaleHow a Semantic Layer Makes  Data Mesh Work at Scale
How a Semantic Layer Makes Data Mesh Work at Scale
DATAVERSITY
 

What's hot (20)

Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
 
Data Architecture Strategies: Data Architecture for Digital Transformation
Data Architecture Strategies: Data Architecture for Digital TransformationData Architecture Strategies: Data Architecture for Digital Transformation
Data Architecture Strategies: Data Architecture for Digital Transformation
 
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
 
Enabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationEnabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data Virtualization
 
Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2
 
Building Lakehouses on Delta Lake with SQL Analytics Primer
Building Lakehouses on Delta Lake with SQL Analytics PrimerBuilding Lakehouses on Delta Lake with SQL Analytics Primer
Building Lakehouses on Delta Lake with SQL Analytics Primer
 
Data Catalog as the Platform for Data Intelligence
Data Catalog as the Platform for Data IntelligenceData Catalog as the Platform for Data Intelligence
Data Catalog as the Platform for Data Intelligence
 
Best Practices in DataOps: How to Create Agile, Automated Data Pipelines
Best Practices in DataOps: How to Create Agile, Automated Data PipelinesBest Practices in DataOps: How to Create Agile, Automated Data Pipelines
Best Practices in DataOps: How to Create Agile, Automated Data Pipelines
 
Data Governance Trends - A Look Backwards and Forwards
Data Governance Trends - A Look Backwards and ForwardsData Governance Trends - A Look Backwards and Forwards
Data Governance Trends - A Look Backwards and Forwards
 
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
 
Data as a Product by Wayne Eckerson
Data as a Product by Wayne EckersonData as a Product by Wayne Eckerson
Data as a Product by Wayne Eckerson
 
Data mesh
Data meshData mesh
Data mesh
 
Data Governance and Metadata Management
Data Governance and Metadata ManagementData Governance and Metadata Management
Data Governance and Metadata Management
 
DW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptxDW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptx
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)
 
Learn to Use Databricks for Data Science
Learn to Use Databricks for Data ScienceLearn to Use Databricks for Data Science
Learn to Use Databricks for Data Science
 
How to govern and secure a Data Mesh?
How to govern and secure a Data Mesh?How to govern and secure a Data Mesh?
How to govern and secure a Data Mesh?
 
Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3
 
Data Catalogs Are the Answer – What is the Question?
Data Catalogs Are the Answer – What is the Question?Data Catalogs Are the Answer – What is the Question?
Data Catalogs Are the Answer – What is the Question?
 
How a Semantic Layer Makes Data Mesh Work at Scale
How a Semantic Layer Makes  Data Mesh Work at ScaleHow a Semantic Layer Makes  Data Mesh Work at Scale
How a Semantic Layer Makes Data Mesh Work at Scale
 

Similar to Data Mesh 101

Keynote: Art of the Possible - Moore
Keynote: Art of the Possible - MooreKeynote: Art of the Possible - Moore
Keynote: Art of the Possible - Moore
Neo4j
 
DCD Big Discussion Guide
DCD Big Discussion GuideDCD Big Discussion Guide
DCD Big Discussion Guide
James Laker
 
Keynote: Art of the Possible - Chandra Rangan
Keynote: Art of the Possible - Chandra RanganKeynote: Art of the Possible - Chandra Rangan
Keynote: Art of the Possible - Chandra Rangan
Neo4j
 
The art of the possible with graph technology_Neo4j GraphSummit Dublin 2023.pptx
The art of the possible with graph technology_Neo4j GraphSummit Dublin 2023.pptxThe art of the possible with graph technology_Neo4j GraphSummit Dublin 2023.pptx
The art of the possible with graph technology_Neo4j GraphSummit Dublin 2023.pptx
Neo4j
 
New ways to apply infrastructure data for better business outcomes
New ways to apply infrastructure data for better business outcomesNew ways to apply infrastructure data for better business outcomes
New ways to apply infrastructure data for better business outcomes
accenture
 
Open Source Ecosystem Future of Enterprise IT
Open Source Ecosystem Future of Enterprise ITOpen Source Ecosystem Future of Enterprise IT
Open Source Ecosystem Future of Enterprise IT
andreas kuncoro
 
The Art of the Possible with Graph Technology
The Art of the Possible with Graph TechnologyThe Art of the Possible with Graph Technology
The Art of the Possible with Graph Technology
Neo4j
 
Thailand Business with the Cloud Service
Thailand Business with  the Cloud ServiceThailand Business with  the Cloud Service
Thailand Business with the Cloud Service
IMC Institute
 
Hybrid Architecture - Is Cloud the Inevitable Best Practice?
Hybrid Architecture - Is Cloud the Inevitable Best Practice?Hybrid Architecture - Is Cloud the Inevitable Best Practice?
Hybrid Architecture - Is Cloud the Inevitable Best Practice?
Christopher Reece
 
Webinar: Cutting through the cloud adoption complexity
Webinar: Cutting through the cloud adoption complexityWebinar: Cutting through the cloud adoption complexity
Webinar: Cutting through the cloud adoption complexity
Interxion
 
Qlik’s CTO on Why the Cloud Data Diaspora Forces Businesses To Rethink their ...
Qlik’s CTO on Why the Cloud Data Diaspora Forces Businesses To Rethink their ...Qlik’s CTO on Why the Cloud Data Diaspora Forces Businesses To Rethink their ...
Qlik’s CTO on Why the Cloud Data Diaspora Forces Businesses To Rethink their ...
Dana Gardner
 
Overcoming Business Challenges with Azure
Overcoming Business Challenges with AzureOvercoming Business Challenges with Azure
Overcoming Business Challenges with Azure
run_frictionless
 
What the future holds for the hybrid cloud
What the future holds for the hybrid cloudWhat the future holds for the hybrid cloud
What the future holds for the hybrid cloud
Netmagic Solutions Pvt. Ltd.
 
7th cloud computing & big data 2013 Summit - 2013
7th cloud computing & big data 2013 Summit - 2013 7th cloud computing & big data 2013 Summit - 2013
7th cloud computing & big data 2013 Summit - 2013
Deepak Raj (2,000+Connections)
 
The 4 Biggest Trends In Big Data and Analytics Right For 2021
The 4 Biggest Trends In Big Data and Analytics Right For 2021The 4 Biggest Trends In Big Data and Analytics Right For 2021
The 4 Biggest Trends In Big Data and Analytics Right For 2021
Bernard Marr
 
Neo4j GraphSummit London March 2023 Emil Eifrem Keynote.pptx
Neo4j GraphSummit London March 2023 Emil Eifrem Keynote.pptxNeo4j GraphSummit London March 2023 Emil Eifrem Keynote.pptx
Neo4j GraphSummit London March 2023 Emil Eifrem Keynote.pptx
Neo4j
 
Buying Into The Cloud
Buying Into The CloudBuying Into The Cloud
Buying Into The Cloud
myhosting
 
QuickView #5 - Cloud
QuickView #5 - CloudQuickView #5 - Cloud
QuickView #5 - Cloud
Sonovate
 
A blueprint for data in a multicloud world
A blueprint for data in a multicloud worldA blueprint for data in a multicloud world
A blueprint for data in a multicloud world
Mehdi Charafeddine
 
Neo4j : L’art des Possibles avec la Technologie des Graphes
Neo4j : L’art des Possibles avec la Technologie des GraphesNeo4j : L’art des Possibles avec la Technologie des Graphes
Neo4j : L’art des Possibles avec la Technologie des Graphes
Neo4j
 

Similar to Data Mesh 101 (20)

Keynote: Art of the Possible - Moore
Keynote: Art of the Possible - MooreKeynote: Art of the Possible - Moore
Keynote: Art of the Possible - Moore
 
DCD Big Discussion Guide
DCD Big Discussion GuideDCD Big Discussion Guide
DCD Big Discussion Guide
 
Keynote: Art of the Possible - Chandra Rangan
Keynote: Art of the Possible - Chandra RanganKeynote: Art of the Possible - Chandra Rangan
Keynote: Art of the Possible - Chandra Rangan
 
The art of the possible with graph technology_Neo4j GraphSummit Dublin 2023.pptx
The art of the possible with graph technology_Neo4j GraphSummit Dublin 2023.pptxThe art of the possible with graph technology_Neo4j GraphSummit Dublin 2023.pptx
The art of the possible with graph technology_Neo4j GraphSummit Dublin 2023.pptx
 
New ways to apply infrastructure data for better business outcomes
New ways to apply infrastructure data for better business outcomesNew ways to apply infrastructure data for better business outcomes
New ways to apply infrastructure data for better business outcomes
 
Open Source Ecosystem Future of Enterprise IT
Open Source Ecosystem Future of Enterprise ITOpen Source Ecosystem Future of Enterprise IT
Open Source Ecosystem Future of Enterprise IT
 
The Art of the Possible with Graph Technology
The Art of the Possible with Graph TechnologyThe Art of the Possible with Graph Technology
The Art of the Possible with Graph Technology
 
Thailand Business with the Cloud Service
Thailand Business with  the Cloud ServiceThailand Business with  the Cloud Service
Thailand Business with the Cloud Service
 
Hybrid Architecture - Is Cloud the Inevitable Best Practice?
Hybrid Architecture - Is Cloud the Inevitable Best Practice?Hybrid Architecture - Is Cloud the Inevitable Best Practice?
Hybrid Architecture - Is Cloud the Inevitable Best Practice?
 
Webinar: Cutting through the cloud adoption complexity
Webinar: Cutting through the cloud adoption complexityWebinar: Cutting through the cloud adoption complexity
Webinar: Cutting through the cloud adoption complexity
 
Qlik’s CTO on Why the Cloud Data Diaspora Forces Businesses To Rethink their ...
Qlik’s CTO on Why the Cloud Data Diaspora Forces Businesses To Rethink their ...Qlik’s CTO on Why the Cloud Data Diaspora Forces Businesses To Rethink their ...
Qlik’s CTO on Why the Cloud Data Diaspora Forces Businesses To Rethink their ...
 
Overcoming Business Challenges with Azure
Overcoming Business Challenges with AzureOvercoming Business Challenges with Azure
Overcoming Business Challenges with Azure
 
What the future holds for the hybrid cloud
What the future holds for the hybrid cloudWhat the future holds for the hybrid cloud
What the future holds for the hybrid cloud
 
7th cloud computing & big data 2013 Summit - 2013
7th cloud computing & big data 2013 Summit - 2013 7th cloud computing & big data 2013 Summit - 2013
7th cloud computing & big data 2013 Summit - 2013
 
The 4 Biggest Trends In Big Data and Analytics Right For 2021
The 4 Biggest Trends In Big Data and Analytics Right For 2021The 4 Biggest Trends In Big Data and Analytics Right For 2021
The 4 Biggest Trends In Big Data and Analytics Right For 2021
 
Neo4j GraphSummit London March 2023 Emil Eifrem Keynote.pptx
Neo4j GraphSummit London March 2023 Emil Eifrem Keynote.pptxNeo4j GraphSummit London March 2023 Emil Eifrem Keynote.pptx
Neo4j GraphSummit London March 2023 Emil Eifrem Keynote.pptx
 
Buying Into The Cloud
Buying Into The CloudBuying Into The Cloud
Buying Into The Cloud
 
QuickView #5 - Cloud
QuickView #5 - CloudQuickView #5 - Cloud
QuickView #5 - Cloud
 
A blueprint for data in a multicloud world
A blueprint for data in a multicloud worldA blueprint for data in a multicloud world
A blueprint for data in a multicloud world
 
Neo4j : L’art des Possibles avec la Technologie des Graphes
Neo4j : L’art des Possibles avec la Technologie des GraphesNeo4j : L’art des Possibles avec la Technologie des Graphes
Neo4j : L’art des Possibles avec la Technologie des Graphes
 

Recently uploaded

Orion Context Broker introduction 20240604
Orion Context Broker introduction 20240604Orion Context Broker introduction 20240604
Orion Context Broker introduction 20240604
Fermin Galan
 
Revolutionizing Visual Effects Mastering AI Face Swaps.pdf
Revolutionizing Visual Effects Mastering AI Face Swaps.pdfRevolutionizing Visual Effects Mastering AI Face Swaps.pdf
Revolutionizing Visual Effects Mastering AI Face Swaps.pdf
Undress Baby
 
OpenMetadata Community Meeting - 5th June 2024
OpenMetadata Community Meeting - 5th June 2024OpenMetadata Community Meeting - 5th June 2024
OpenMetadata Community Meeting - 5th June 2024
OpenMetadata
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024
Hironori Washizaki
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
Green Software Development
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
Rakesh Kumar R
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
Drona Infotech
 
2024 eCommerceDays Toulouse - Sylius 2.0.pdf
2024 eCommerceDays Toulouse - Sylius 2.0.pdf2024 eCommerceDays Toulouse - Sylius 2.0.pdf
2024 eCommerceDays Toulouse - Sylius 2.0.pdf
Łukasz Chruściel
 
socradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdfsocradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdf
SOCRadar
 
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Crescat
 
Hand Rolled Applicative User Validation Code Kata
Hand Rolled Applicative User ValidationCode KataHand Rolled Applicative User ValidationCode Kata
Hand Rolled Applicative User Validation Code Kata
Philip Schwarz
 
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise EditionWhy Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
Envertis Software Solutions
 
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppAI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
Google
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOMLORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
lorraineandreiamcidl
 
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR  SPRING BOOT (SPRING IO)GOING AOT WITH GRAALVM FOR  SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
Alina Yurenko
 
DDS-Security 1.2 - What's New? Stronger security for long-running systems
DDS-Security 1.2 - What's New? Stronger security for long-running systemsDDS-Security 1.2 - What's New? Stronger security for long-running systems
DDS-Security 1.2 - What's New? Stronger security for long-running systems
Gerardo Pardo-Castellote
 
May Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdfMay Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdf
Adele Miller
 
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
kalichargn70th171
 
Enterprise Resource Planning System in Telangana
Enterprise Resource Planning System in TelanganaEnterprise Resource Planning System in Telangana
Enterprise Resource Planning System in Telangana
NYGGS Automation Suite
 

Recently uploaded (20)

Orion Context Broker introduction 20240604
Orion Context Broker introduction 20240604Orion Context Broker introduction 20240604
Orion Context Broker introduction 20240604
 
Revolutionizing Visual Effects Mastering AI Face Swaps.pdf
Revolutionizing Visual Effects Mastering AI Face Swaps.pdfRevolutionizing Visual Effects Mastering AI Face Swaps.pdf
Revolutionizing Visual Effects Mastering AI Face Swaps.pdf
 
OpenMetadata Community Meeting - 5th June 2024
OpenMetadata Community Meeting - 5th June 2024OpenMetadata Community Meeting - 5th June 2024
OpenMetadata Community Meeting - 5th June 2024
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
 
2024 eCommerceDays Toulouse - Sylius 2.0.pdf
2024 eCommerceDays Toulouse - Sylius 2.0.pdf2024 eCommerceDays Toulouse - Sylius 2.0.pdf
2024 eCommerceDays Toulouse - Sylius 2.0.pdf
 
socradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdfsocradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdf
 
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
 
Hand Rolled Applicative User Validation Code Kata
Hand Rolled Applicative User ValidationCode KataHand Rolled Applicative User ValidationCode Kata
Hand Rolled Applicative User Validation Code Kata
 
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise EditionWhy Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
 
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppAI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOMLORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
 
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR  SPRING BOOT (SPRING IO)GOING AOT WITH GRAALVM FOR  SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
 
DDS-Security 1.2 - What's New? Stronger security for long-running systems
DDS-Security 1.2 - What's New? Stronger security for long-running systemsDDS-Security 1.2 - What's New? Stronger security for long-running systems
DDS-Security 1.2 - What's New? Stronger security for long-running systems
 
May Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdfMay Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdf
 
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
 
Enterprise Resource Planning System in Telangana
Enterprise Resource Planning System in TelanganaEnterprise Resource Planning System in Telangana
Enterprise Resource Planning System in Telangana
 

Data Mesh 101

  • 1. © 2023 Thoughtworks Data Mesh 101 Chris Ford & Pablo Porto
  • 2. © 2023 Thoughtworks I am Head of Technology for Thoughtworks Spain. I help clients with architecture, agile development and organisational effectiveness. I was a technical reviewer for Zhamak Dehghani’s 2022 book 'Data Mesh'. 2 I am a Lead Developer for Thoughtworks Spain's Data and AI Service Line. I help clients build distributed systems, platforms and data architectures. I am currently working as part of one of the most mature Data Mesh implementations in the healthcare industry. 2 © 2023 Thoughtworks Chris Ford Pablo Porto
  • 3. © 2023 Thoughtworks Part zero: Introduction 3 3
  • 4. © 2023 Thoughtworks 4 Promise What is the value proposition of Data Mesh? Principles What are the core elements of Data Mesh? Practicalities Why is it important and how can you get started? Structure of this talk Table of contents
  • 5. © 2023 Thoughtworks “All models are wrong but some are useful.” 5 George Box, Statistician Image from Wikipedia 5 © 2022 Thoughtworks
  • 6. © 2023 Thoughtworks (domain-driven design + microservices) × data = Data Mesh sort of... 6
  • 7. © 2023 Thoughtworks Part one: Promise 7 7
  • 8. © 2023 Thoughtworks 8 If we fulfill these conditions Architectural paradigms We can obtain these benefits So long as we don’t mind these costs 8 © 2022 Thoughtworks What is the cost/benefit? In combination, a paradigm’s elements define a kind of promise: The promise only pays off when the conditions are satisfied, the benefits are valued and the costs are acceptable.
  • 9. © 2023 Thoughtworks It’s usually more valuable to consider “Does this paradigm apply in this context?” than to try and decide “Is this paradigm good?” 9 9 © 2023 Thoughtworks
  • 10. © 2023 Thoughtworks 10 Microservices promise 10 © 2022 Thoughtworks If we break things into small pieces We can change them independently So long as we don’t mind the added integration complexity
  • 11. © 2023 Thoughtworks 11 Domain-driven design promise 11 © 2022 Thoughtworks If we align our architecture with our business domain We can represent our business accurately, even as it changes So long as we don’t mind adapting the technology to the business domain
  • 12. © 2023 Thoughtworks 12 Data warehouse promise 12 © 2022 Thoughtworks If we model our data up front in a central schema We can run analytical queries across our business So long as we don’t mind the effort of aggregating and reconciling it E T L
  • 13. © 2023 Thoughtworks 13 Data lake promise 13 © 2022 Thoughtworks If we collect our raw data into a central location We can post-hoc run queries about anything we like So long as we don’t mind dependencies being implicit E T L E T L
  • 14. © 2023 Thoughtworks 14 1. Data mesh promise 14 © 2022 Thoughtworks If we give responsibility for data to the people who produce it We can rapidly incorporate new data sources and use cases So long as we don’t mind distributing skills and investing in self-service infrastructure
  • 15. © 2023 Thoughtworks Part two: Principles 15 15
  • 16. © 2023 Thoughtworks Data mesh 16 Principles Domain ownership Self-serve data platform Data as a product Federated computational governance
  • 17. © 2023 Thoughtworks Domain ownership 17 Principles Domain ownership What is it? You give the originators of data authority over it and the responsibility to make it easily and usefully available. Where’s the value? This reduces the distance between producer and consumer (in terms of handoffs), enabling quicker, easier and richer consumption.
  • 18. © 2023 Thoughtworks Data as a product 18 Principles Data as a product What is it? You make your data available as products that are designed around the needs of its consumers. Where’s the value? By framing things in terms of product, you put the emphasis on use and give clear responsibility on the owners to fix anything that interferes with use (and credit for working on anything that promotes use).
  • 19. © 2023 Thoughtworks Self-serve data platform 19 Principles Self-serve data platform What is it? You have the ability to provision infrastructure for the creation and consumption of data products without a human in the loop. Where’s the value? If you want to decrease lead time and enable on-demand changes, you need self-service.
  • 20. © 2023 Thoughtworks Federated computational governance 20 Principles Federated computational governance What is it? You have general guidance that make the requirements of good citizenship clear to everyone in the mesh, leaning on automation whenever possible. Where’s the value? By managing with constraints, rather than inspecting individual items, we empower teams. By using automation, we scale, reduce handoffs and encourage interoperability.
  • 21. © 2023 Thoughtworks Data mesh 21 Principles Domain ownership Self-serve data platform Data as a product Federated computational governance
  • 22. © 2023 Thoughtworks Data mesh 22 Principles Domain ownership Self-serve data platform Data as a product Federated computational governance Team Organisation
  • 23. © 2023 Thoughtworks Part three: Practicalities 23 23
  • 24. © 2023 Thoughtworks Why is now a time to care about data mesh? Industry context Rapidly developing new data use cases is increasingly important to businesses. The upside of Data Mesh is going up. 24 Data skills and infrastructure are increasingly accessible as technology advances. The downside of Data Mesh is going down.
  • 25. © 2023 Thoughtworks Glovo is a delivery service facing strong competition in a market where innovation and operational efficiency is key to success. With the pandemic, consumer behaviours and expectations changed. Understanding customers’ needs and the capability to quickly react to them are crucial to increase customer loyalty and satisfaction. An execution plan on how to deliver on the vision A clear vision for data in the organisation 25
  • 26. © 2023 Thoughtworks Lesson learned #2 Introduce clear rules about what aspects of data quality to measure and communicate. 26 “A data platform that supports data products and business insights” Lesson learned #1 Ensure that every system that produces or transforms important data has a clear owner who is taking care of it. © 2023 Thoughtworks
  • 27. © 2023 Thoughtworks Roche is one of the biggest life sciences and healthcare companies in the world with over 100k employees and operating in multiple markets. Roche wanted to unlock its potential on the market leveraging the rich and abundant data it possesses. They chose Data Mesh as the approach and Thoughtworks as a partner to achieve their vision. Access to data is hard due to technical and organisational constraints Data interoperability becomes hard at scale with multiple data platforms 27
  • 28. © 2023 Thoughtworks Lesson learned #2 Show value frequently, both to business stakeholders and developers using your platform. 28 Focus on incremental value Lesson learned #1 Start your journey with your existing organisational boundaries. © 2023 Thoughtworks
  • 29. © 2023 Thoughtworks How to get started What combination of data products is needed to serve these use cases? What is the thinnest viable platform that is practical to support these data products? What is the minimum viable cooperation we need to work successfully together? Use cases Data products Self-serve data platform Governance What are good candidates use case with real and achievable value?
  • 30. © 2023 Thoughtworks 30 Questions? (We are hiring Senior and Lead Data Engineers!) (We are remote-friendly and ordinary friendly too)
  • 32. © 2023 Thoughtworks References ● Original How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh article that introduced the concept by Zhamak Dehghani ● Follow-up Data Mesh Principles and Logical Architecture article by Zhamak Dehghani ● Data Mesh in practice: Getting off to the right start article series about Roche’s Data Mesh journey by Ammara Gafoor, Ian Murdoch and Kiran Prakash ● Data mesh: it's not just about tech, it's about ownership and communication article series about Glovo’s Data Mesh journey by Jorge Agudo, Narek Verdian, Óscar Torres Fernández, Pablo Giner, Diana Pinto and Javier García. ● Data Mesh Accelerate workshop description by Steve Upton and Paulo Caroli 32 WHERE TO GO NEXT Data Mesh book by Zhamak Dehghani
  • 33. © 2023 Thoughtworks Chris Ford Head of Technology, Thoughtworks Spain linkedin.com/in/ctford twitter.com/ctford chris.ford@thoughtworks.com 33 Thank you! Pablo Porto Lead Developer, Thoughtworks Spain linkedin.com/in/pabloportoveloso twitter.com/portovep pablo.porto@thoughtworks.com
  • 34. © 2023 Thoughtworks “We’ve only got three teams” Threat to applicability 1: no scale Data Mesh gives responsibility for data to its originators. 34 This is great for reducing the distance between data producer and data consumer. BUT... If your organisation is so small that distance between data producer and data consumer is naturally short, there’s not much point investing in reducing it.
  • 35. © 2023 Thoughtworks “Data’s not key to our business right now” Threat to applicability 2: no value Data Mesh makes new data use cases quicker and easier. 35 This is great for experimentation and for getting new data use cases to market quickly. BUT... If data use cases are not of value to your business, there’s not much point investing in enabling them.
  • 36. © 2023 Thoughtworks “We’re not into flow and autonomy” Threat to applicability 3: no flow Data Mesh enables autonomous change within domains. 36 This is great for achieving fast flow, though it requires investment to pull it off. BUT... If your culture or context is not prepared to take advantage of team autonomy, there’s not much point investing in enabling it.
  • 37. © 2023 Thoughtworks Myth 1: infra explosion 37 “The infrastructure will be too expensive” ● Data Mesh emphasises that domains should own their own data. ● Some people interpret that as meaning that they need to provision independent infrastructure for each domain, which could lead to a cost blowout. ● However, Data Mesh only requires that infrastructure be logically separate and self-service, so multi tenant infrastructure used by multiple domains is fine. Data Mesh is opinionated about how you organise infrastructure, not how you provision it.
  • 38. © 2023 Thoughtworks Myth 2: data castle 38 “Data quality will suffer” ● Many organisations have data quality assurance processes where experts own the data lake or data warehouse and inspect changes to datasets. ● In Data Mesh, governance happens via policy, not by inspection of individual additions, alterations or consumptions of datasets / data products. ● This works by reducing distance between data producer and data consumer and giving experts greater leverage by governing the process. Data Mesh has an alternative way of ensuring data quality, delivering potentially richer and deeper quality to consumers.
  • 39. © 2023 Thoughtworks Myth 3: over enthusiasm 39 “Data mesh will solve everything” ● Operational systems manage transactional data that changes in real time. ● Analytical systems manage a view of the facts of the business over time. ● Data Mesh is an architectural paradigm specifically aimed at analytical problems. ● Organisations that will benefit from Data Mesh likely have other problems in their operational systems, but they have other solutions. Data Mesh aims to bring operational and analytical systems into harmony, not to conflate them.