Data Mesh 101

© 2023 Thoughtworks
Data Mesh 101
Chris Ford & Pablo Porto

I am Head of Technology for Thoughtworks
Spain. I help clients with architecture, agile
development and organisational
effectiveness.
I was a technical reviewer for Zhamak
Dehghani’s 2022 book 'Data Mesh'.
2
I am a Lead Developer for Thoughtworks
Spain's Data and AI Service Line. I help
clients build distributed systems, platforms
and data architectures.
I am currently working as part of one of the
most mature Data Mesh implementations in
the healthcare industry.
2
Chris Ford Pablo Porto

Part zero: Introduction
3
3

© 2023 Thoughtworks 4
Promise
What is the value
proposition of Data
Mesh?
Principles
What are the core
elements of Data
Mesh?
Practicalities
Why is it important
and how can you get
started?
Structure of this talk
Table of contents

“All models are wrong but
some are useful.”
5
George Box, Statistician
Image from Wikipedia
5

(domain-driven design + microservices)
× data
= Data Mesh
sort of...
6

Part one: Promise
7
7

If we fulfill these conditions
Architectural paradigms
We can obtain these benefits
So long as we don’t mind these costs
8
What is the cost/benefit?
In combination, a paradigm’s elements define a kind of promise:
The promise only pays off when the conditions are satisfied, the benefits are valued
and the costs are acceptable.

It’s usually more valuable to consider
“Does this paradigm apply in this context?”
than to try and decide
“Is this paradigm good?”
9
9

Microservices promise
10
If we break things into small pieces
We can change them independently
So long as we don’t mind the added
integration complexity

Domain-driven design promise
11
If we align our architecture with our
business domain
We can represent our business
accurately, even as it changes
So long as we don’t mind adapting the
technology to the business domain

Data warehouse promise
12
If we model our data up front in a central
schema
We can run analytical queries across our
business
So long as we don’t mind the effort of
aggregating and reconciling it
E
T
L

Data lake promise
13
If we collect our raw data into a central
location
We can post-hoc run queries about
anything we like
So long as we don’t mind dependencies
being implicit
E
T
L
E
T
L

1.
Data mesh promise
14
If we give responsibility for data to the
people who produce it
We can rapidly incorporate new data
sources and use cases
So long as we don’t mind distributing
skills and investing in self-service
infrastructure

Part two: Principles
15
15

Data mesh
16
Principles
Domain
ownership
Self-serve data
platform
Data as a
product
Federated
computational
governance

Domain ownership
17
Principles
Domain
ownership
What is it?
You give the originators of data authority over it and the responsibility
to make it easily and usefully available.
Where’s the value?
This reduces the distance between producer and consumer (in terms of
handoffs), enabling quicker, easier and richer consumption.

Data as a product
18
Principles
Data as a
product
What is it?
You make your data available as products that are designed around the
needs of its consumers.
By framing things in terms of product, you put the emphasis on use and
give clear responsibility on the owners to fix anything that interferes
with use (and credit for working on anything that promotes use).

Self-serve data platform
19
Principles
Self-serve
data
platform
What is it?
You have the ability to provision infrastructure for the creation and
consumption of data products without a human in the loop.
If you want to decrease lead time and enable on-demand changes, you
need self-service.

Federated computational governance
20
Principles
Federated
computational
governance
What is it?
You have general guidance that make the requirements of good
citizenship clear to everyone in the mesh, leaning on automation
whenever possible.
By managing with constraints, rather than inspecting individual items,
we empower teams. By using automation, we scale, reduce handoffs
and encourage interoperability.

Data mesh
21
Principles
Domain
ownership
Self-serve data
platform
Data as a
product
Federated
computational
governance

Data mesh
22
Principles
Domain
ownership
Self-serve data
platform
Data as a
product
Federated
computational
governance
Team
Organisation

Part three: Practicalities
23
23

Why is now a time to care about data mesh?
Industry context
Rapidly developing new data use cases is
increasingly important to businesses.
The upside of Data Mesh is going up.
24
Data skills and infrastructure are
increasingly accessible as technology
advances.
The downside of Data Mesh is going
down.

Glovo is a delivery service facing
strong competition in a market where
innovation and operational efficiency
is key to success.
With the pandemic, consumer
behaviours and expectations
changed.
Understanding customers’ needs and
the capability to quickly react to them
are crucial to increase customer
loyalty and satisfaction.
An execution
plan on how to
deliver on the
vision
A clear vision for
data in the
organisation
25

Lesson learned #2
Introduce clear rules about what aspects
of data quality to measure and
communicate.
26
“A data platform that
supports data products
and business insights”
Lesson learned #1
Ensure that every system that produces
or transforms important data has a clear
owner who is taking care of it.

Roche is one of the biggest life
sciences and healthcare companies
in the world with over 100k
employees and operating in multiple
markets.
Roche wanted to unlock its potential
on the market leveraging the rich and
abundant data it possesses.
They chose Data Mesh as the
approach and Thoughtworks as a
partner to achieve their vision.
Access to data
is hard due to
technical and
organisational
constraints
Data
interoperability
becomes hard at
scale with
multiple data
platforms
27

Lesson learned #2
Show value frequently, both to business
stakeholders and developers using your
platform.
28
Focus on incremental value
Lesson learned #1
Start your journey with your existing
organisational boundaries.

How to get started
What combination of data products is needed to
serve these use cases?
What is the thinnest viable platform that is
practical to support these data products?
What is the minimum viable cooperation we
need to work successfully together?
Use cases
Data products
Self-serve data platform
Governance
What are good candidates use case with real
and achievable value?

Questions?
(We are hiring Senior and Lead Data Engineers!)
(We are remote-friendly and ordinary friendly too)

Resources
31
31

References
● Original How to Move Beyond a Monolithic Data Lake to a
Distributed Data Mesh article that introduced the concept
by Zhamak Dehghani
● Follow-up Data Mesh Principles and Logical Architecture
article by Zhamak Dehghani
● Data Mesh in practice: Getting off to the right start article
series about Roche’s Data Mesh journey by Ammara Gafoor,
Ian Murdoch and Kiran Prakash
● Data mesh: it's not just about tech, it's about ownership and
communication article series about Glovo’s Data Mesh
journey by Jorge Agudo, Narek Verdian, Óscar Torres
Fernández, Pablo Giner, Diana Pinto and Javier García.
● Data Mesh Accelerate workshop description by Steve Upton
and Paulo Caroli
32
WHERE TO GO NEXT
Data Mesh book by
Zhamak Dehghani

Chris Ford
Head of Technology, Thoughtworks Spain
linkedin.com/in/ctford
twitter.com/ctford
chris.ford@thoughtworks.com
33
Thank you!
Pablo Porto
Lead Developer, Thoughtworks Spain
linkedin.com/in/pabloportoveloso
twitter.com/portovep
pablo.porto@thoughtworks.com

“We’ve only got three teams”
Threat to applicability 1: no scale
Data Mesh gives responsibility for
data to its originators.
34
This is great for reducing the
distance between data producer
and data consumer.
BUT...
If your organisation is so small that distance between data producer and data consumer
is naturally short, there’s not much point investing in reducing it.

“Data’s not key to our business right now”
Threat to applicability 2: no value
Data Mesh makes new data use
cases quicker and easier.
35
This is great for experimentation
and for getting new data use cases
to market quickly.
BUT...
If data use cases are not of value to your business, there’s not much point investing in
enabling them.

“We’re not into flow and autonomy”
Threat to applicability 3: no ﬂow
Data Mesh enables autonomous
change within domains.
36
This is great for achieving fast
ﬂow, though it requires
investment to pull it off.
BUT...
If your culture or context is not prepared to take advantage of team autonomy, there’s
not much point investing in enabling it.

Myth 1: infra explosion
37
“The infrastructure will be too expensive”
● Data Mesh emphasises that domains should own their own data.
● Some people interpret that as meaning that they need to provision independent
infrastructure for each domain, which could lead to a cost blowout.
● However, Data Mesh only requires that infrastructure be logically separate and
self-service, so multi tenant infrastructure used by multiple domains is fine.
Data Mesh is opinionated about how you organise infrastructure, not how you
provision it.

Myth 2: data castle
38
“Data quality will suffer”
● Many organisations have data quality assurance processes where experts own the
data lake or data warehouse and inspect changes to datasets.
● In Data Mesh, governance happens via policy, not by inspection of individual
additions, alterations or consumptions of datasets / data products.
● This works by reducing distance between data producer and data consumer and
giving experts greater leverage by governing the process.
Data Mesh has an alternative way of ensuring data quality, delivering potentially
richer and deeper quality to consumers.

Myth 3: over enthusiasm
39
“Data mesh will solve everything”
● Operational systems manage transactional data that changes in real time.
● Analytical systems manage a view of the facts of the business over time.
● Data Mesh is an architectural paradigm specifically aimed at analytical problems.
● Organisations that will benefit from Data Mesh likely have other problems in their
operational systems, but they have other solutions.
Data Mesh aims to bring operational and analytical systems into harmony, not to
conflate them.

Data Mesh 101

More Related Content

What's hot

Similar to Data Mesh 101

Recently uploaded

Data Mesh 101