SlideShare a Scribd company logo
Data Mesh
at CMC Markets:
Past, Present and Future
Data Mesh Learning Meetup - 29/7/2021
About us
Principal Core Data Engineers at
Tareq Abedrabbo Lorenzo Nicora Michal Stypik
Agenda
◎ Context and background
◎ Data Mesh at CMC Markets
1. What Data Goes Inside the Data Mesh?
2. Cloud Infrastructure and Self-service
3. Data Discovery: an Essential Ingredient
4. Natively Accessible Data in a Hybrid Environment
◎ Adoption: challenges & enablers
◎ The future
CMC Markets
◎ FTSE 250 Company
◎ Online financial trading platform (+10k instruments)
◎ Trade in CFD, Forex, shares and more
◎ Successfully launching multiple products over 30 years
◎ Data-centric business
◎ Transformation: enable rapid innovation
○ Public cloud adoption (AWS)
○ Cross-functional/Squad delivery model
○ Self-service
○ Autonomy and enablement
Data & Transformation
◎ Data is at the core of transformation
○ New cloud-based products
○ Valuable data is on premise
◎ Existing data is decentralised/siloed
⇒ Coupling of “knowledge” and “work” in a single team
○ Find where the data you need might be
○ Then wait for work to be done for you
○ Results in queuing, additional work and other inefficiencies
◎ Limited conventions, standards and norms shared across teams
Context
◎ Common questions
○ How do you find the data you need?
○ How can you understand and trust the data?
○ How do you make data available for new products?
○ How can you allow the business to innovate at scale?
Data Mesh at CMC Markets
1. What Data Goes Inside
the Data Mesh?
What Data Goes Inside the Data Mesh?
◎ What data? All data!
○ “Data Neutrality”
◎ Data is neither analytical nor operational, use cases are
○ Analytical and operational planes describe systems
(people + software), not the data itself
○ Not a useful criterion to decide the scope of the Data Mesh
What Data Goes Inside the Data Mesh?
◎ Key ideas
○ Data on the inside vs Data on the outside
(Pat Helland, 2005)
○ Fundamental data sources
What Data Goes Inside the Data Mesh?
◎ Data on the inside vs Data on the outside
○ Data internal to domains vs Data shared with other domains
○ We borrowed and adapted Pat Helland’s concept
○ Identify data of common interest ⇒ Data on the Outside
○ Data in the Data Mesh needs to be discoverable, consumable,
managed, etc. It’s hard work!
○ Allows to maintain teams autonomy by clarifying the boundaries of
the data domains and the interface between them
Refs:
Pat Helland - 2005 http://cidrdb.org/cidr2005/papers/P12.pdf
Adrian Colyer, 2016 https://blog.acolyer.org/2016/09/13/data-on-the-outside-versus-data-on-the-inside/
Pat Helland - 2020 https://queue.acm.org/detail.cfm?id=3415014
What Data Goes Inside the Data Mesh?
◎ Find fundamental data sources
○ Fundamental = authoritative, and primary
○ Belongs to a central business domain
○ With clear ownership
○ e.g. tradable prices, trades (vs P&L)
Our Approach
◎ What we do
○ Data-centric and heuristic approach - not “ivory tower”
○ Emerged from attempting to map all data flows within the business
○ We are building on top of a strong platform baseline
◎ What we don’t do
○ We are not starting from technology
○ We are not consolidating or linking up existing data silos
○ We are not building point-to-point data integrations
2. Cloud infrastructure
and Self-service
Cloud Infrastructure
◎ Cloud enablement and shared services
○ Collaboration, experience and skills
◎ Data Infrastructure
○ General infrastructure patterns and solutions
○ Common way of building
○ Reusable components
○ Self-serve and real life
◎ Tools we use: AWS, Terraform, GitHub Actions, containers
◎ Hybrid platform model (cloud and on-prem)
3. Data Discovery:
an essential ingredient
Data Discovery
◎ Essential capability to “see through” the Data Mesh
◎ For data consumers
○ Starting point to find the data they need
○ Enhances self-service
○ Improves the interaction between data consumers and data owners
◎ For data owners
○ Clear entry point to onboard new data sources
○ Centralises metadata, while data remains decentralised
○ Data Discovery is a subset of Metadata Management
(data governance, protection, etc...)
◎ Not just a data catalogue
Data Discovery
◎ We are trialling Amundsen (amundsen.io)
○ Still emerging but has a great community and traction
○ Open source: extendable and customisable
○ Simple architecture, cloud-friendly
○ Backed by a graph database
○ Deployed on our cloud infrastructure
4. Natively Accessible Data
in a hybrid environment
Data Natively Accessible
◎ Goal: data natively accessible on the cloud and on-prem
○ New products are built on the cloud
○ Many fundamental data sources are on-prem
Example
◎ Make tradable prices natively available on the cloud
○ Low-latency data stream generated from CMC based on market prices
○ Gap: available on-prem through low-latency messaging but not on cloud
◎ Solution: Aeron-based pricing bridge
○ Low-latency and reliable messaging on-premise and on the cloud
○ High fidelity: preserving semantics to enable a variety of use cases
◎ Multiple views, based on the same data, to serve different types of use cases
○ Latest prices, Timeseries, Event log (streaming)...
Data Mesh Adoption
Data Mesh + Transformation
◎ Data Mesh needs a broad scope to be effective
○ Not just a change of technology
○ Multiple parts of the business are involved
○ Shift in mindset
◎ Data Mesh & Transformation must share the same ethos
○ Decentralised
○ Collaborative
○ Allows scale and autonomy
◎ DDD foundation
⇒ business needs to reorganise around business domains
Challenges
◎ Data Mesh is a longer term investment
◎ Buy-in and commitment from the broader business
◎ Business priorities can shift quickly
◎ Established structures and ways of working
◎ Change generates friction
◎ Hybrid technology landscape
Enablers
◎ Transformation programme
◎ Cloud adoption and self-service
◎ Squad delivery model
◎ Small expert team: can think, do and respond to change
quickly and autonomously
◎ Attack the problem from different angles:
○ Find manageable use cases to validate the approach and create traction
○ Build new capabilities e.g. Data Discovery
○ Make fundamental data sources consumable
○ Share and educate
Future
◎ Gradual growing of the Data Mesh
◎ Onboarding more fundamental data sources
◎ Data discovery adoption
◎ The cloud becomes a viable environment for more
data use cases
Q & A

More Related Content

What's hot

Enabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationEnabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data Virtualization
Denodo
 
Data Mesh for Dinner
Data Mesh for DinnerData Mesh for Dinner
Data Mesh for Dinner
Kent Graziano
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
Databricks
 
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
HostedbyConfluent
 
Data Governance and Metadata Management
Data Governance and Metadata ManagementData Governance and Metadata Management
Data Governance and Metadata Management
DATAVERSITY
 
Data Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and GovernanceData Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and Governance
Denodo
 
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data PipelinesPutting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines
DATAVERSITY
 
Data Quality Best Practices
Data Quality Best PracticesData Quality Best Practices
Data Quality Best Practices
DATAVERSITY
 
Activate Data Governance Using the Data Catalog
Activate Data Governance Using the Data CatalogActivate Data Governance Using the Data Catalog
Activate Data Governance Using the Data Catalog
DATAVERSITY
 
Data Architecture Strategies: Data Architecture for Digital Transformation
Data Architecture Strategies: Data Architecture for Digital TransformationData Architecture Strategies: Data Architecture for Digital Transformation
Data Architecture Strategies: Data Architecture for Digital Transformation
DATAVERSITY
 
Data Governance Best Practices, Assessments, and Roadmaps
Data Governance Best Practices, Assessments, and RoadmapsData Governance Best Practices, Assessments, and Roadmaps
Data Governance Best Practices, Assessments, and Roadmaps
DATAVERSITY
 
Slides: Taking an Active Approach to Data Governance
Slides: Taking an Active Approach to Data GovernanceSlides: Taking an Active Approach to Data Governance
Slides: Taking an Active Approach to Data Governance
DATAVERSITY
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4
Databricks
 
DataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven OrganizationsDataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven Organizations
Ellen Friedman
 
Data mesh
Data meshData mesh
Data mesh
ManojKumarR41
 
Best Practices in Metadata Management
Best Practices in Metadata ManagementBest Practices in Metadata Management
Best Practices in Metadata Management
DATAVERSITY
 
Improving Data Literacy Around Data Architecture
Improving Data Literacy Around Data ArchitectureImproving Data Literacy Around Data Architecture
Improving Data Literacy Around Data Architecture
DATAVERSITY
 
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Cathrine Wilhelmsen
 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data Architecture
Databricks
 
Introdution to Dataops and AIOps (or MLOps)
Introdution to Dataops and AIOps (or MLOps)Introdution to Dataops and AIOps (or MLOps)
Introdution to Dataops and AIOps (or MLOps)
Adrien Blind
 

What's hot (20)

Enabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationEnabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data Virtualization
 
Data Mesh for Dinner
Data Mesh for DinnerData Mesh for Dinner
Data Mesh for Dinner
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
 
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
 
Data Governance and Metadata Management
Data Governance and Metadata ManagementData Governance and Metadata Management
Data Governance and Metadata Management
 
Data Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and GovernanceData Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and Governance
 
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data PipelinesPutting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines
 
Data Quality Best Practices
Data Quality Best PracticesData Quality Best Practices
Data Quality Best Practices
 
Activate Data Governance Using the Data Catalog
Activate Data Governance Using the Data CatalogActivate Data Governance Using the Data Catalog
Activate Data Governance Using the Data Catalog
 
Data Architecture Strategies: Data Architecture for Digital Transformation
Data Architecture Strategies: Data Architecture for Digital TransformationData Architecture Strategies: Data Architecture for Digital Transformation
Data Architecture Strategies: Data Architecture for Digital Transformation
 
Data Governance Best Practices, Assessments, and Roadmaps
Data Governance Best Practices, Assessments, and RoadmapsData Governance Best Practices, Assessments, and Roadmaps
Data Governance Best Practices, Assessments, and Roadmaps
 
Slides: Taking an Active Approach to Data Governance
Slides: Taking an Active Approach to Data GovernanceSlides: Taking an Active Approach to Data Governance
Slides: Taking an Active Approach to Data Governance
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4
 
DataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven OrganizationsDataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven Organizations
 
Data mesh
Data meshData mesh
Data mesh
 
Best Practices in Metadata Management
Best Practices in Metadata ManagementBest Practices in Metadata Management
Best Practices in Metadata Management
 
Improving Data Literacy Around Data Architecture
Improving Data Literacy Around Data ArchitectureImproving Data Literacy Around Data Architecture
Improving Data Literacy Around Data Architecture
 
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data Architecture
 
Introdution to Dataops and AIOps (or MLOps)
Introdution to Dataops and AIOps (or MLOps)Introdution to Dataops and AIOps (or MLOps)
Introdution to Dataops and AIOps (or MLOps)
 

Similar to Data Mesh at CMC Markets: Past, Present and Future

Myth Busters VII: I’m building a data mesh, so I don’t need data virtualization
Myth Busters VII: I’m building a data mesh, so I don’t need data virtualizationMyth Busters VII: I’m building a data mesh, so I don’t need data virtualization
Myth Busters VII: I’m building a data mesh, so I don’t need data virtualization
Denodo
 
BPM and SOA Are Going Mobile: An Architectural Perspective
BPM and SOA Are Going Mobile: An Architectural PerspectiveBPM and SOA Are Going Mobile: An Architectural Perspective
BPM and SOA Are Going Mobile: An Architectural Perspective
Guido Schmutz
 
Reinventing and Simplifying Data Management for a Successful Hybrid and Multi...
Reinventing and Simplifying Data Management for a Successful Hybrid and Multi...Reinventing and Simplifying Data Management for a Successful Hybrid and Multi...
Reinventing and Simplifying Data Management for a Successful Hybrid and Multi...
Denodo
 
Dell hans timmerman v1.1
Dell hans timmerman v1.1Dell hans timmerman v1.1
Dell hans timmerman v1.1
BigDataExpo
 
Accelerate Migration to the Cloud using Data Virtualization (APAC)
Accelerate Migration to the Cloud using Data Virtualization (APAC)Accelerate Migration to the Cloud using Data Virtualization (APAC)
Accelerate Migration to the Cloud using Data Virtualization (APAC)
Denodo
 
Data Mesh 101
Data Mesh 101Data Mesh 101
Data Mesh 101
ChrisFord803185
 
Multi-Cloud Integration with Data Virtualization (ASEAN)
Multi-Cloud Integration with Data Virtualization (ASEAN)Multi-Cloud Integration with Data Virtualization (ASEAN)
Multi-Cloud Integration with Data Virtualization (ASEAN)
Denodo
 
The rise of “Big Data” on cloud computing
The rise of “Big Data” on cloud computingThe rise of “Big Data” on cloud computing
The rise of “Big Data” on cloud computing
Minhazul Arefin
 
Speak to Your Data
Speak to Your DataSpeak to Your Data
Speak to Your Data
Amer Radwan , PMP , CSM
 
A Logical Architecture is Always a Flexible Architecture (ASEAN)
A Logical Architecture is Always a Flexible Architecture (ASEAN)A Logical Architecture is Always a Flexible Architecture (ASEAN)
A Logical Architecture is Always a Flexible Architecture (ASEAN)
Denodo
 
Hybrid Clouds: “Silver Bullet” of the Cloud Computing?
Hybrid Clouds: “Silver Bullet” of the Cloud Computing?Hybrid Clouds: “Silver Bullet” of the Cloud Computing?
Hybrid Clouds: “Silver Bullet” of the Cloud Computing?
Fabrizio Volpe
 
Cloud and Data Analytics Architecture: Data Everywhere for Everyone
Cloud and Data Analytics Architecture: Data Everywhere for EveryoneCloud and Data Analytics Architecture: Data Everywhere for Everyone
Cloud and Data Analytics Architecture: Data Everywhere for Everyone
Michal Hodinka
 
Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)
Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)
Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)
Denodo
 
Data Virtualization to Survive a Multi and Hybrid Cloud World
Data Virtualization to Survive a Multi and Hybrid Cloud WorldData Virtualization to Survive a Multi and Hybrid Cloud World
Data Virtualization to Survive a Multi and Hybrid Cloud World
Denodo
 
To mesh or mess up your data organisation - Jochem van Grondelle (Prosus/OLX ...
To mesh or mess up your data organisation - Jochem van Grondelle (Prosus/OLX ...To mesh or mess up your data organisation - Jochem van Grondelle (Prosus/OLX ...
To mesh or mess up your data organisation - Jochem van Grondelle (Prosus/OLX ...
Jochem van Grondelle
 
Partner Engagement Webinar Series: Highlights from DataFest North America
Partner Engagement Webinar Series: Highlights from DataFest North AmericaPartner Engagement Webinar Series: Highlights from DataFest North America
Partner Engagement Webinar Series: Highlights from DataFest North America
Denodo
 
Multi-Cloud-Datenintegration mit Datenvirtualisierung
Multi-Cloud-Datenintegration mit DatenvirtualisierungMulti-Cloud-Datenintegration mit Datenvirtualisierung
Multi-Cloud-Datenintegration mit Datenvirtualisierung
Denodo
 
Your Data is Waiting. What are the Top 5 Trends for Data in 2022? (ASEAN)
Your Data is Waiting. What are the Top 5 Trends for Data in 2022? (ASEAN)Your Data is Waiting. What are the Top 5 Trends for Data in 2022? (ASEAN)
Your Data is Waiting. What are the Top 5 Trends for Data in 2022? (ASEAN)
Denodo
 
techbrief-enterprisedatameshandgoldengate.pdf
techbrief-enterprisedatameshandgoldengate.pdftechbrief-enterprisedatameshandgoldengate.pdf
techbrief-enterprisedatameshandgoldengate.pdf
aliramezani30
 
cloud-20deployments-20model-131226165813-phpapp01.pptx
cloud-20deployments-20model-131226165813-phpapp01.pptxcloud-20deployments-20model-131226165813-phpapp01.pptx
cloud-20deployments-20model-131226165813-phpapp01.pptx
20DC11NOUFALN
 

Similar to Data Mesh at CMC Markets: Past, Present and Future (20)

Myth Busters VII: I’m building a data mesh, so I don’t need data virtualization
Myth Busters VII: I’m building a data mesh, so I don’t need data virtualizationMyth Busters VII: I’m building a data mesh, so I don’t need data virtualization
Myth Busters VII: I’m building a data mesh, so I don’t need data virtualization
 
BPM and SOA Are Going Mobile: An Architectural Perspective
BPM and SOA Are Going Mobile: An Architectural PerspectiveBPM and SOA Are Going Mobile: An Architectural Perspective
BPM and SOA Are Going Mobile: An Architectural Perspective
 
Reinventing and Simplifying Data Management for a Successful Hybrid and Multi...
Reinventing and Simplifying Data Management for a Successful Hybrid and Multi...Reinventing and Simplifying Data Management for a Successful Hybrid and Multi...
Reinventing and Simplifying Data Management for a Successful Hybrid and Multi...
 
Dell hans timmerman v1.1
Dell hans timmerman v1.1Dell hans timmerman v1.1
Dell hans timmerman v1.1
 
Accelerate Migration to the Cloud using Data Virtualization (APAC)
Accelerate Migration to the Cloud using Data Virtualization (APAC)Accelerate Migration to the Cloud using Data Virtualization (APAC)
Accelerate Migration to the Cloud using Data Virtualization (APAC)
 
Data Mesh 101
Data Mesh 101Data Mesh 101
Data Mesh 101
 
Multi-Cloud Integration with Data Virtualization (ASEAN)
Multi-Cloud Integration with Data Virtualization (ASEAN)Multi-Cloud Integration with Data Virtualization (ASEAN)
Multi-Cloud Integration with Data Virtualization (ASEAN)
 
The rise of “Big Data” on cloud computing
The rise of “Big Data” on cloud computingThe rise of “Big Data” on cloud computing
The rise of “Big Data” on cloud computing
 
Speak to Your Data
Speak to Your DataSpeak to Your Data
Speak to Your Data
 
A Logical Architecture is Always a Flexible Architecture (ASEAN)
A Logical Architecture is Always a Flexible Architecture (ASEAN)A Logical Architecture is Always a Flexible Architecture (ASEAN)
A Logical Architecture is Always a Flexible Architecture (ASEAN)
 
Hybrid Clouds: “Silver Bullet” of the Cloud Computing?
Hybrid Clouds: “Silver Bullet” of the Cloud Computing?Hybrid Clouds: “Silver Bullet” of the Cloud Computing?
Hybrid Clouds: “Silver Bullet” of the Cloud Computing?
 
Cloud and Data Analytics Architecture: Data Everywhere for Everyone
Cloud and Data Analytics Architecture: Data Everywhere for EveryoneCloud and Data Analytics Architecture: Data Everywhere for Everyone
Cloud and Data Analytics Architecture: Data Everywhere for Everyone
 
Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)
Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)
Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)
 
Data Virtualization to Survive a Multi and Hybrid Cloud World
Data Virtualization to Survive a Multi and Hybrid Cloud WorldData Virtualization to Survive a Multi and Hybrid Cloud World
Data Virtualization to Survive a Multi and Hybrid Cloud World
 
To mesh or mess up your data organisation - Jochem van Grondelle (Prosus/OLX ...
To mesh or mess up your data organisation - Jochem van Grondelle (Prosus/OLX ...To mesh or mess up your data organisation - Jochem van Grondelle (Prosus/OLX ...
To mesh or mess up your data organisation - Jochem van Grondelle (Prosus/OLX ...
 
Partner Engagement Webinar Series: Highlights from DataFest North America
Partner Engagement Webinar Series: Highlights from DataFest North AmericaPartner Engagement Webinar Series: Highlights from DataFest North America
Partner Engagement Webinar Series: Highlights from DataFest North America
 
Multi-Cloud-Datenintegration mit Datenvirtualisierung
Multi-Cloud-Datenintegration mit DatenvirtualisierungMulti-Cloud-Datenintegration mit Datenvirtualisierung
Multi-Cloud-Datenintegration mit Datenvirtualisierung
 
Your Data is Waiting. What are the Top 5 Trends for Data in 2022? (ASEAN)
Your Data is Waiting. What are the Top 5 Trends for Data in 2022? (ASEAN)Your Data is Waiting. What are the Top 5 Trends for Data in 2022? (ASEAN)
Your Data is Waiting. What are the Top 5 Trends for Data in 2022? (ASEAN)
 
techbrief-enterprisedatameshandgoldengate.pdf
techbrief-enterprisedatameshandgoldengate.pdftechbrief-enterprisedatameshandgoldengate.pdf
techbrief-enterprisedatameshandgoldengate.pdf
 
cloud-20deployments-20model-131226165813-phpapp01.pptx
cloud-20deployments-20model-131226165813-phpapp01.pptxcloud-20deployments-20model-131226165813-phpapp01.pptx
cloud-20deployments-20model-131226165813-phpapp01.pptx
 

More from Lorenzo Nicora

Write fast, think later - Event sourcing for IoT and Mobile
Write fast, think later  - Event sourcing for IoT and MobileWrite fast, think later  - Event sourcing for IoT and Mobile
Write fast, think later - Event sourcing for IoT and Mobile
Lorenzo Nicora
 
Event Sourcing in an Eventually Consistent World
Event Sourcing in an Eventually Consistent WorldEvent Sourcing in an Eventually Consistent World
Event Sourcing in an Eventually Consistent World
Lorenzo Nicora
 
Event sourcing for IoT and mobile - JAX London 2017
Event sourcing for IoT and mobile - JAX London 2017Event sourcing for IoT and mobile - JAX London 2017
Event sourcing for IoT and mobile - JAX London 2017
Lorenzo Nicora
 
From C to Q, one event at the time: Event sourcing illustrated [Voxxed Ticino...
From C to Q, one event at the time: Event sourcing illustrated [Voxxed Ticino...From C to Q, one event at the time: Event sourcing illustrated [Voxxed Ticino...
From C to Q, one event at the time: Event sourcing illustrated [Voxxed Ticino...
Lorenzo Nicora
 
From C to Q one event at a time: Event Sourcing illustrated
From C to Q one event at a time: Event Sourcing illustratedFrom C to Q one event at a time: Event Sourcing illustrated
From C to Q one event at a time: Event Sourcing illustrated
Lorenzo Nicora
 
Reactive Principles and Microservices
Reactive Principles and MicroservicesReactive Principles and Microservices
Reactive Principles and Microservices
Lorenzo Nicora
 
The Actor model: an alternative approach to concurrency
The Actor model: an alternative approach to concurrencyThe Actor model: an alternative approach to concurrency
The Actor model: an alternative approach to concurrency
Lorenzo Nicora
 
A visual introduction to Event Sourcing and CQRS
A visual introduction to Event Sourcing and CQRSA visual introduction to Event Sourcing and CQRS
A visual introduction to Event Sourcing and CQRS
Lorenzo Nicora
 

More from Lorenzo Nicora (8)

Write fast, think later - Event sourcing for IoT and Mobile
Write fast, think later  - Event sourcing for IoT and MobileWrite fast, think later  - Event sourcing for IoT and Mobile
Write fast, think later - Event sourcing for IoT and Mobile
 
Event Sourcing in an Eventually Consistent World
Event Sourcing in an Eventually Consistent WorldEvent Sourcing in an Eventually Consistent World
Event Sourcing in an Eventually Consistent World
 
Event sourcing for IoT and mobile - JAX London 2017
Event sourcing for IoT and mobile - JAX London 2017Event sourcing for IoT and mobile - JAX London 2017
Event sourcing for IoT and mobile - JAX London 2017
 
From C to Q, one event at the time: Event sourcing illustrated [Voxxed Ticino...
From C to Q, one event at the time: Event sourcing illustrated [Voxxed Ticino...From C to Q, one event at the time: Event sourcing illustrated [Voxxed Ticino...
From C to Q, one event at the time: Event sourcing illustrated [Voxxed Ticino...
 
From C to Q one event at a time: Event Sourcing illustrated
From C to Q one event at a time: Event Sourcing illustratedFrom C to Q one event at a time: Event Sourcing illustrated
From C to Q one event at a time: Event Sourcing illustrated
 
Reactive Principles and Microservices
Reactive Principles and MicroservicesReactive Principles and Microservices
Reactive Principles and Microservices
 
The Actor model: an alternative approach to concurrency
The Actor model: an alternative approach to concurrencyThe Actor model: an alternative approach to concurrency
The Actor model: an alternative approach to concurrency
 
A visual introduction to Event Sourcing and CQRS
A visual introduction to Event Sourcing and CQRSA visual introduction to Event Sourcing and CQRS
A visual introduction to Event Sourcing and CQRS
 

Recently uploaded

Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
Remote DBA Services
 
Modelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - AmsterdamModelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - Amsterdam
Alberto Brandolini
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
Rakesh Kumar R
 
zOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL DifferenceszOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL Differences
YousufSait3
 
SQL Accounting Software Brochure Malaysia
SQL Accounting Software Brochure MalaysiaSQL Accounting Software Brochure Malaysia
SQL Accounting Software Brochure Malaysia
GohKiangHock
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
Peter Muessig
 
Energy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina JonuziEnergy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina Jonuzi
Green Software Development
 
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
mz5nrf0n
 
Webinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for EmbeddedWebinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for Embedded
ICS
 
WWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders AustinWWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders Austin
Patrick Weigel
 
UI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design SystemUI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design System
Peter Muessig
 
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsUI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
Peter Muessig
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
dakas1
 
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension FunctionsArtificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
Octavian Nadolu
 
Lecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptxLecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptx
TaghreedAltamimi
 
YAML crash COURSE how to write yaml file for adding configuring details
YAML crash COURSE how to write yaml file for adding configuring detailsYAML crash COURSE how to write yaml file for adding configuring details
YAML crash COURSE how to write yaml file for adding configuring details
NishanthaBulumulla1
 
Project Management: The Role of Project Dashboards.pdf
Project Management: The Role of Project Dashboards.pdfProject Management: The Role of Project Dashboards.pdf
Project Management: The Role of Project Dashboards.pdf
Karya Keeper
 
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptxOracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
Remote DBA Services
 
Hand Rolled Applicative User Validation Code Kata
Hand Rolled Applicative User ValidationCode KataHand Rolled Applicative User ValidationCode Kata
Hand Rolled Applicative User Validation Code Kata
Philip Schwarz
 
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language ProcessorsFundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
Rakesh Kumar R
 

Recently uploaded (20)

Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
 
Modelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - AmsterdamModelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - Amsterdam
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
 
zOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL DifferenceszOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL Differences
 
SQL Accounting Software Brochure Malaysia
SQL Accounting Software Brochure MalaysiaSQL Accounting Software Brochure Malaysia
SQL Accounting Software Brochure Malaysia
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
 
Energy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina JonuziEnergy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina Jonuzi
 
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
 
Webinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for EmbeddedWebinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for Embedded
 
WWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders AustinWWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders Austin
 
UI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design SystemUI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design System
 
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsUI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
 
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension FunctionsArtificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
 
Lecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptxLecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptx
 
YAML crash COURSE how to write yaml file for adding configuring details
YAML crash COURSE how to write yaml file for adding configuring detailsYAML crash COURSE how to write yaml file for adding configuring details
YAML crash COURSE how to write yaml file for adding configuring details
 
Project Management: The Role of Project Dashboards.pdf
Project Management: The Role of Project Dashboards.pdfProject Management: The Role of Project Dashboards.pdf
Project Management: The Role of Project Dashboards.pdf
 
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptxOracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
 
Hand Rolled Applicative User Validation Code Kata
Hand Rolled Applicative User ValidationCode KataHand Rolled Applicative User ValidationCode Kata
Hand Rolled Applicative User Validation Code Kata
 
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language ProcessorsFundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
 

Data Mesh at CMC Markets: Past, Present and Future

  • 1. Data Mesh at CMC Markets: Past, Present and Future Data Mesh Learning Meetup - 29/7/2021
  • 2. About us Principal Core Data Engineers at Tareq Abedrabbo Lorenzo Nicora Michal Stypik
  • 3. Agenda ◎ Context and background ◎ Data Mesh at CMC Markets 1. What Data Goes Inside the Data Mesh? 2. Cloud Infrastructure and Self-service 3. Data Discovery: an Essential Ingredient 4. Natively Accessible Data in a Hybrid Environment ◎ Adoption: challenges & enablers ◎ The future
  • 4. CMC Markets ◎ FTSE 250 Company ◎ Online financial trading platform (+10k instruments) ◎ Trade in CFD, Forex, shares and more ◎ Successfully launching multiple products over 30 years ◎ Data-centric business ◎ Transformation: enable rapid innovation ○ Public cloud adoption (AWS) ○ Cross-functional/Squad delivery model ○ Self-service ○ Autonomy and enablement
  • 5. Data & Transformation ◎ Data is at the core of transformation ○ New cloud-based products ○ Valuable data is on premise ◎ Existing data is decentralised/siloed ⇒ Coupling of “knowledge” and “work” in a single team ○ Find where the data you need might be ○ Then wait for work to be done for you ○ Results in queuing, additional work and other inefficiencies ◎ Limited conventions, standards and norms shared across teams
  • 6. Context ◎ Common questions ○ How do you find the data you need? ○ How can you understand and trust the data? ○ How do you make data available for new products? ○ How can you allow the business to innovate at scale?
  • 7.
  • 8.
  • 9. Data Mesh at CMC Markets
  • 10. 1. What Data Goes Inside the Data Mesh?
  • 11. What Data Goes Inside the Data Mesh? ◎ What data? All data! ○ “Data Neutrality” ◎ Data is neither analytical nor operational, use cases are ○ Analytical and operational planes describe systems (people + software), not the data itself ○ Not a useful criterion to decide the scope of the Data Mesh
  • 12. What Data Goes Inside the Data Mesh? ◎ Key ideas ○ Data on the inside vs Data on the outside (Pat Helland, 2005) ○ Fundamental data sources
  • 13. What Data Goes Inside the Data Mesh? ◎ Data on the inside vs Data on the outside ○ Data internal to domains vs Data shared with other domains ○ We borrowed and adapted Pat Helland’s concept ○ Identify data of common interest ⇒ Data on the Outside ○ Data in the Data Mesh needs to be discoverable, consumable, managed, etc. It’s hard work! ○ Allows to maintain teams autonomy by clarifying the boundaries of the data domains and the interface between them Refs: Pat Helland - 2005 http://cidrdb.org/cidr2005/papers/P12.pdf Adrian Colyer, 2016 https://blog.acolyer.org/2016/09/13/data-on-the-outside-versus-data-on-the-inside/ Pat Helland - 2020 https://queue.acm.org/detail.cfm?id=3415014
  • 14. What Data Goes Inside the Data Mesh? ◎ Find fundamental data sources ○ Fundamental = authoritative, and primary ○ Belongs to a central business domain ○ With clear ownership ○ e.g. tradable prices, trades (vs P&L)
  • 15. Our Approach ◎ What we do ○ Data-centric and heuristic approach - not “ivory tower” ○ Emerged from attempting to map all data flows within the business ○ We are building on top of a strong platform baseline ◎ What we don’t do ○ We are not starting from technology ○ We are not consolidating or linking up existing data silos ○ We are not building point-to-point data integrations
  • 17. Cloud Infrastructure ◎ Cloud enablement and shared services ○ Collaboration, experience and skills ◎ Data Infrastructure ○ General infrastructure patterns and solutions ○ Common way of building ○ Reusable components ○ Self-serve and real life ◎ Tools we use: AWS, Terraform, GitHub Actions, containers ◎ Hybrid platform model (cloud and on-prem)
  • 18. 3. Data Discovery: an essential ingredient
  • 19. Data Discovery ◎ Essential capability to “see through” the Data Mesh ◎ For data consumers ○ Starting point to find the data they need ○ Enhances self-service ○ Improves the interaction between data consumers and data owners ◎ For data owners ○ Clear entry point to onboard new data sources ○ Centralises metadata, while data remains decentralised ○ Data Discovery is a subset of Metadata Management (data governance, protection, etc...) ◎ Not just a data catalogue
  • 20. Data Discovery ◎ We are trialling Amundsen (amundsen.io) ○ Still emerging but has a great community and traction ○ Open source: extendable and customisable ○ Simple architecture, cloud-friendly ○ Backed by a graph database ○ Deployed on our cloud infrastructure
  • 21.
  • 22. 4. Natively Accessible Data in a hybrid environment
  • 23. Data Natively Accessible ◎ Goal: data natively accessible on the cloud and on-prem ○ New products are built on the cloud ○ Many fundamental data sources are on-prem Example ◎ Make tradable prices natively available on the cloud ○ Low-latency data stream generated from CMC based on market prices ○ Gap: available on-prem through low-latency messaging but not on cloud ◎ Solution: Aeron-based pricing bridge ○ Low-latency and reliable messaging on-premise and on the cloud ○ High fidelity: preserving semantics to enable a variety of use cases ◎ Multiple views, based on the same data, to serve different types of use cases ○ Latest prices, Timeseries, Event log (streaming)...
  • 24.
  • 25.
  • 27.
  • 28. Data Mesh + Transformation ◎ Data Mesh needs a broad scope to be effective ○ Not just a change of technology ○ Multiple parts of the business are involved ○ Shift in mindset ◎ Data Mesh & Transformation must share the same ethos ○ Decentralised ○ Collaborative ○ Allows scale and autonomy ◎ DDD foundation ⇒ business needs to reorganise around business domains
  • 29. Challenges ◎ Data Mesh is a longer term investment ◎ Buy-in and commitment from the broader business ◎ Business priorities can shift quickly ◎ Established structures and ways of working ◎ Change generates friction ◎ Hybrid technology landscape
  • 30. Enablers ◎ Transformation programme ◎ Cloud adoption and self-service ◎ Squad delivery model ◎ Small expert team: can think, do and respond to change quickly and autonomously ◎ Attack the problem from different angles: ○ Find manageable use cases to validate the approach and create traction ○ Build new capabilities e.g. Data Discovery ○ Make fundamental data sources consumable ○ Share and educate
  • 31. Future ◎ Gradual growing of the Data Mesh ◎ Onboarding more fundamental data sources ◎ Data discovery adoption ◎ The cloud becomes a viable environment for more data use cases
  • 32. Q & A