SlideShare a Scribd company logo
Nathan Bijnens
Manager, Belux CSU Data Team
Data Mesh in Azure
using Cloud Scale Analytics
What we’ve heard
To spend less time preparing
data
Robust data governance
Platform to actionable Insights to
the business
Ability to increase the value of
hidden data
Improve Operational Efficiency
Ideally, organizations
want to have…..
Reduce cost of data engineering
Need for Frictionless
Data Governance
Difficult to balance
access and data
protection
Data and Analytics
Operationalization
Enable Lines of Businesses
Poor data quality
Disparate systems
and data silos
Too slow moving
from data to decision
Barriers
to
achieve
business
outcomes
Unified ecosystem
Project prioritization
Every application that creates data, needs and will have a database
Application A Application B
Consequently, when we have two applications, we hypothesize that each application has its own ‘database’.
When there is interoperability between these two applications, we expect data to be transferred from one
application to the other.
Every application, at least in the context of data management, that creates data, needs and will have a
database. Even stateless applications that create data have “databases”. In these scenarios the database
typically sits in the RAM or in a temp file.
We can’t escape from data integration
Application A Application B
The ‘always’ required data transformation lies in the fact that an application database schema is designed to
meet the application’s specific requirements. Since the requirements differ from application to application,
the schemas are expected to be different and data integration is always required when moving data around.
A crucial aspect when it comes to data transfer is that data integration is always right around the corner.
Whether you do ETL or ELT, virtual or physical, batch or real-time, there’s no escape from the data
integration* dilemma.
Data integration
Business Drivers
•Lack of data
ownership
Lack of data quality
Difficult to see
interdependencies
Model conflicts
across business
concerns
Tremendous effort
for integration and
coordination leads
to bypasses
Business and IT
work in silos
Disconnect
between the data
producers and data
consumers
Central team
becomes the
bottleneck
Difficult to apply
policy and
governance
Hard to see
technical
dependencies
Small changes
become risky due
to unexpected
consequences
Technical
ownership rather
than data
ownership
Many Enterprises are saddled with outdated Data Architectures that do not scale to the needs of large multi-
disciplinary organizations.
Problems with Existing Architectures
There’s a deep assumption that centralization is the solution to data management. This includes
centralizing all data and management activities into one central team, building one data platform,
using one ETL framework, using one canonical model, etc.
Transactional
Sources
Analytical
Consumers
Centralized Architecture
• Single team with centralized knowledge and book of work
• Centralized pipelines for all extraction / ingestion activities
• Centralized transformations to create harmonized data
• Central platform serves as large integration database: all
execution and analysis is done on the same platform
Data providers Data consumers
Central engineering team
Transactional
Sources
Transactional
Sources
Analytical
Consumers
Analytical
Consumers
Transformational Trends in the Data Landscape
Massive increase of computing power, driven
by hardware innovation (SSD storage, in-
memory storage, GPU advances) lets us move
data to compute faster.
Cloud and APIs make it easier to integrate.
Software & Platform as a Service (SaaS, PaaS)
offerings push the connectivity and API usage
even further.
Explosion of tools
New (open source) concepts are introduced,
such as NoSQL database types, block chain,
new database designs, distributed models
(Hadoop), new analytical methods, etc.
Exponential growth of data, especially external
data sources like open and social data.
Internal, external, structured, and unstructured
data are all used to deliver additional insights.
Eco-system connectivity
Exponential growth of data
Increase of computing power
Stronger regulatory requirements, such as
GDPR and BCBS 239, are coming into effect
worldwide. Data quality and lineage become
more important every day.
Increased regulatory attention
The read/write ratio has changed due to more
intensive data consumption: data is read more
often, there is increased real-time consumption
and more searches are performed.
Increase of read/write ratio
Data as a Product
Data as a Product
Data is no
longer a
side-effect,
it’s a product.
Who are my
"customers"?
What do my
"customers"
need?
Are they
happy with
the data? Are
they using it?
How do I let
my
"customers"
know my
data exists?
What is in it
for the
"customer"?
Data Product Owner
Domain
Data
Product
Owner
Data
Engineer
Software
Developer
Infra
Engineer
How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh (martinfowler.com)
Zhamak Dehghani
Data Product Properties
How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh (martinfowler.com)
Zhamak Dehghani
• Overview of product in central data catalog
• Provide easy discoverability
Discoverable
• Help users access the product
programmatically
Addressable
• Data Product Owners provide monitored SLOs
• Data is cleansed and up to standard
Trustworthy
• Minimal friction for data engineers and
scientists to use the data
Self-describing
• Open standards for harmonization
• Field type formatting
Interoperable
• Access control policies
• Use SSO and RBAC
Secure
Data Mesh
Data Mesh
Data Mesh is a new decentralized
socio-technical approach to
managing data, designed to work
with organizational complexity and
continuous growth. It enables large
organizations to get value from their
data, at scale, through reusability,
analytics and ML. It is building on the
Domain Driven Design methodology.
Data
Mesh
Domain
Driven
Design
Domain
Zones
Data
Products
Consumed
by other
Domains
How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh (martinfowler.com)
Zhamak Dehghani
Centralized Implementation is not working!
GSR
Finance
HR
Travel
Sales
Clinical Ops
Centralized Platform
LOBs are the SMEs and Shared
Service team is not able to cope
up with the projects
Datasets sprawls
Competing needs within the
organization
• IT needs to standardize
• LOBs need to implement analytics
Primitive Data Strategy
Introduction to Data Domains
Search
Keywords
Promotions
Top
Selling
Products
Orders
Customer
Profiles
Data Products
Integration
Services
Operational
Systems
Marketing
Domain
Customer Services
Domain
Order Management
Domain
• A domain is a collection of people, typically organized around a common business purpose.
• Create and serve data products to other domains and end users, independently from other domains.
• Ensure data is accessible, usable, available, and meets the quality criteria defined.
• Evolve data products based on user feedback and retire data products when they become irrelevant.
Domain Zones
Engineering
Finance HR Innovation
Program 1 Operations
Management zone
Data products
Data Domains
Microsoft Enterprise Data Mesh
Domain Zone
Domain Zone
Environment for each LOB
LOBs: Implement Data Services
• ex: Exploration Service, Data Order System
LOBs: Build and Share Data Products
• ex: Sales Forecast, Clean Room Performance
Automated using templates
• security, integration, monitoring, etc
E N T E R P R I S E
R E Q U I R E M E N T S
Security & Privacy
Governance & Compliance
Availability & Recovery
Performance & Scalability
Skills & Training
Licensing & Usage
Observation & Monitoring
Domain Architecture
Shift towards Domain Ownership
A new type of eco-system architecture, which shifts to the left towards a modern distributed
architecture that enables domain-specific data and data products, empowering each domain to handle
their own data pipelines.
Supporting governance and domain-agnostic platform infrastructure
Data Providers Data Product
Data Providers Data Product
Data Providers Data Product
Source-oriented
Domains
Consumer-
specific
Transformation
Data Consumer
Consumer-
specific
Transformation
Data Consumer
Consumer-
specific
Transformation
Data Consumer
Consumption-oriented
Domains
Domain Zones
Data Products
Domain Zone
HR
Recruitment
Time Tracking
Employee Value
And Performance
Training and
Development
Engagement and
Retention
Engineering Operations
New Project :
Digital Twin
Clean Room
Personnel
• Map your data domains organically, during the onboarding of data
providers and consumers.
• Reference your business capabilities (e.g., strategy and processes) while
mapping your data domains.
• Isolate your data domains and enable communication through data
products like APIs or events.
• Create and document a shared, ubiquitous language that different domains
can use to communicate.
• Determine boundaries for both business and technical granularity.
Data Domain Considerations from the Field
Enterprise Scale for
Analytics
Cloud-scale Analytics Framework
Enterprise Scale: Azure Landing Zones
The main purpose of a “Landing Zone”
is to ensure that when a workload
lands on Azure, the required
“plumbing” is already in place,
providing greater agility and
compliance with enterprise security
and governance requirements.
Data Management Landing Zone
Data Management Landing Zone
Business Glossary
Data Discovery
SLAs Business Rules Ref. Data Mgmt.
Master Record Mgmt.
Data Policy
Access Governance
Loss Prevention
Privacy Operations
Risk Assessment
Repository for Data
Models
Integration
API Documentation
Automation for provisioning landing zones, data
integrations, and products
Pre-configured network and monitoring setup Standard images for deploying analytics and AI services
Azure Subscription Azure Policy
Data Landing Zone
Core
Networking Shared
Products
Ingest and
Processing
Upload
Data Lake
Services
Metadata
Services
Preconfigured
network and
monitoring setup
Data lake configured
with layers and
connectivity
Spark and
scheduling
engines
Blobs where 3rd parties
can upload their data
Scanners for data
governance/metadata
required by landing
zone
Analytics engines for
exploratory analytics
Data
Integration
Data
Integration #
Data Integration Teams are responsible for the ingestion of data to a
read data source. The data shouldn’t have any data transformation
applied apart from data quality checks and data type verification.
Data
Integration #
Pull SAP Data into
Landing Zone #
Streaming interface
to pull data from
heat sensors
Data
Products
Data Product
#
Data Product #
Financial Reporting
pulling Customers and
Sales together
Streaming Machine
Data from Read Data
Source
Data products fulfil a specific business need using data. Data products
manage, organize, and make sense of data across domains and
present the insights gained from the data products.
A data product is a result from one or many data integrations and/or
other data products.
Infrastructure
as
Code
Azure Event
Hubs
Azure Data
Lake Store Gen2
Storing read-optimized
domain data
Data
Product
Team
Data
Product
Team
Data
Product
Team
Data
Product
Team Data Onboarding Team
Data Integration
Synapse
Analytics
Data
Product
Team
Data
Product
Team
Data
Product
Team
Real-time applications,
operational systems
Self-service BI,
semantic models
Analytical applications
Data
Engineering
Team
Data Management
Landing Zone
Data Governance
Team
Azure Purview
Data Lake Services
Azure Data
Factory
Transforming into read-
optimized data products
Data Integration
Data Integration
Data Landing Zone
Azure Databricks
Shared Service
Data-driven
applications
Data Product
Data Product
Example Reference Architecture for Data Mesh in a Small Company
Data Product
Optimize Existing Implementation Patterns
Take a new approach to data management that supports and evolves with your strategy.
The data management and analytics scenario supports a range of patterns to
build on your current data infrastructure, to help you modernize and scale from where you are.
Data Warehouse Data Lake Data Lakehouse Data Mesh
Data Fabric
Integrating your DWH in a Data Mesh
 From be-all end-all to yet another Data Product in your mesh
 Ownership based on your preference
 DWH is data product on its own: managed by one data product team
 DWH serves as "wrapper" for multiple data products: managed by multiple teams
 DWH consumes data from multiple Data Products
 Multiple Data Products consume data from DWH
Agile Data Management
Enforce data governance and security.

Serve data as a product rather than a byproduct.

Provide an ecosystem of data products.

Create data domains to serve lines of business.

Empower teams to drive analytics solutions that deliver value to the business.

Modernize your teams and operations.

Prepare your company to:
Multi Organization
Data Mesh
Contoso
Managem
ent zone
Data products
Data Domains
Multi Organization Data Mesh
Finance
HR
Organization
Contoso
Managem
ent zone
Finance
HR
Contoso
Managem
ent zone
Finance
HR
Interested in
learning more?
Reach out to
Nathan.Bijnens@microsoft.com
Links
DDD
Best Practice - An Introduction To Domain-Driven Design | Microsoft Docs
Introduction into Domain-Driven Design (DDD) (jannikwempe.com)
IBM Automation Event-Driven Reference Architecture – Domain Driven Design (ibm-
cloud-architecture.github.io)
Data Mesh
How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh
(martinfowler.com)
Data Mesh in Practice: How Europe's Leading Online Platform for Fashion Goes
Beyond the Data Lake - Databricks

More Related Content

What's hot

Activate Data Governance Using the Data Catalog
Activate Data Governance Using the Data CatalogActivate Data Governance Using the Data Catalog
Activate Data Governance Using the Data Catalog
DATAVERSITY
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4
Databricks
 
Enabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationEnabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data Virtualization
Denodo
 
Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3
Jeffrey T. Pollock
 
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
DATAVERSITY
 
Microsoft Information Protection demystified Albert Hoitingh
Microsoft Information Protection demystified Albert HoitinghMicrosoft Information Protection demystified Albert Hoitingh
Microsoft Information Protection demystified Albert Hoitingh
Albert Hoitingh
 
Azure purview
Azure purviewAzure purview
Azure purview
Shafqat Turza
 
Microsoft cloud big data strategy
Microsoft cloud big data strategyMicrosoft cloud big data strategy
Microsoft cloud big data strategy
James Serra
 
Azure Purview Data Toboggan Erwin de Kreuk
Azure Purview Data Toboggan Erwin de KreukAzure Purview Data Toboggan Erwin de Kreuk
Azure Purview Data Toboggan Erwin de Kreuk
Erwin de Kreuk
 
How to govern and secure a Data Mesh?
How to govern and secure a Data Mesh?How to govern and secure a Data Mesh?
How to govern and secure a Data Mesh?
confluent
 
Data mesh
Data meshData mesh
Data mesh
ManojKumarR41
 
Building the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake AnalyticsBuilding the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake Analytics
Khalid Salama
 
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Dr. Arif Wider
 
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Cathrine Wilhelmsen
 
Building Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft AzureBuilding Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft Azure
Dmitry Anoshin
 
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Tristan Baker
 
Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?
DATAVERSITY
 
Modern Data architecture Design
Modern Data architecture DesignModern Data architecture Design
Modern Data architecture Design
Kujambu Murugesan
 
ExpertsLive NL 2022 - Microsoft Purview - What's in it for my organization?
ExpertsLive NL 2022 - Microsoft Purview - What's in it for my organization?ExpertsLive NL 2022 - Microsoft Purview - What's in it for my organization?
ExpertsLive NL 2022 - Microsoft Purview - What's in it for my organization?
Albert Hoitingh
 
Data Architecture - The Foundation for Enterprise Architecture and Governance
Data Architecture - The Foundation for Enterprise Architecture and GovernanceData Architecture - The Foundation for Enterprise Architecture and Governance
Data Architecture - The Foundation for Enterprise Architecture and Governance
DATAVERSITY
 

What's hot (20)

Activate Data Governance Using the Data Catalog
Activate Data Governance Using the Data CatalogActivate Data Governance Using the Data Catalog
Activate Data Governance Using the Data Catalog
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4
 
Enabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationEnabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data Virtualization
 
Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3
 
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
 
Microsoft Information Protection demystified Albert Hoitingh
Microsoft Information Protection demystified Albert HoitinghMicrosoft Information Protection demystified Albert Hoitingh
Microsoft Information Protection demystified Albert Hoitingh
 
Azure purview
Azure purviewAzure purview
Azure purview
 
Microsoft cloud big data strategy
Microsoft cloud big data strategyMicrosoft cloud big data strategy
Microsoft cloud big data strategy
 
Azure Purview Data Toboggan Erwin de Kreuk
Azure Purview Data Toboggan Erwin de KreukAzure Purview Data Toboggan Erwin de Kreuk
Azure Purview Data Toboggan Erwin de Kreuk
 
How to govern and secure a Data Mesh?
How to govern and secure a Data Mesh?How to govern and secure a Data Mesh?
How to govern and secure a Data Mesh?
 
Data mesh
Data meshData mesh
Data mesh
 
Building the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake AnalyticsBuilding the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake Analytics
 
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
 
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
 
Building Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft AzureBuilding Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft Azure
 
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
 
Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?
 
Modern Data architecture Design
Modern Data architecture DesignModern Data architecture Design
Modern Data architecture Design
 
ExpertsLive NL 2022 - Microsoft Purview - What's in it for my organization?
ExpertsLive NL 2022 - Microsoft Purview - What's in it for my organization?ExpertsLive NL 2022 - Microsoft Purview - What's in it for my organization?
ExpertsLive NL 2022 - Microsoft Purview - What's in it for my organization?
 
Data Architecture - The Foundation for Enterprise Architecture and Governance
Data Architecture - The Foundation for Enterprise Architecture and GovernanceData Architecture - The Foundation for Enterprise Architecture and Governance
Data Architecture - The Foundation for Enterprise Architecture and Governance
 

Similar to Data Mesh in Azure using Cloud Scale Analytics (WAF)

Data Mesh using Microsoft Fabric
Data Mesh using Microsoft FabricData Mesh using Microsoft Fabric
Data Mesh using Microsoft Fabric
Nathan Bijnens
 
data-mesh_whitepaper_dec2021.pdf
data-mesh_whitepaper_dec2021.pdfdata-mesh_whitepaper_dec2021.pdf
data-mesh_whitepaper_dec2021.pdf
ssuser18927d
 
Data and Application Modernization in the Age of the Cloud
Data and Application Modernization in the Age of the CloudData and Application Modernization in the Age of the Cloud
Data and Application Modernization in the Age of the Cloud
redmondpulver
 
Modernize your Infrastructure and Mobilize Your Data
Modernize your Infrastructure and Mobilize Your DataModernize your Infrastructure and Mobilize Your Data
Modernize your Infrastructure and Mobilize Your Data
Precisely
 
When and How Data Lakes Fit into a Modern Data Architecture
When and How Data Lakes Fit into a Modern Data ArchitectureWhen and How Data Lakes Fit into a Modern Data Architecture
When and How Data Lakes Fit into a Modern Data Architecture
DATAVERSITY
 
ANIn Pune July 2024 | Bootstrapping Data Mesh for a Complex Enterprise by Bal...
ANIn Pune July 2024 | Bootstrapping Data Mesh for a Complex Enterprise by Bal...ANIn Pune July 2024 | Bootstrapping Data Mesh for a Complex Enterprise by Bal...
ANIn Pune July 2024 | Bootstrapping Data Mesh for a Complex Enterprise by Bal...
AgileNetwork
 
Is your big data journey stalling? Take the Leap with Capgemini and Cloudera
Is your big data journey stalling? Take the Leap with Capgemini and ClouderaIs your big data journey stalling? Take the Leap with Capgemini and Cloudera
Is your big data journey stalling? Take the Leap with Capgemini and Cloudera
Cloudera, Inc.
 
SMAC - Social, Mobile, Analytics and Cloud - An overview
SMAC - Social, Mobile, Analytics and Cloud - An overview SMAC - Social, Mobile, Analytics and Cloud - An overview
SMAC - Social, Mobile, Analytics and Cloud - An overview
Rajesh Menon
 
Evolving Big Data Strategies: Bringing Data Lake and Data Mesh Vision to Life
Evolving Big Data Strategies: Bringing Data Lake and Data Mesh Vision to LifeEvolving Big Data Strategies: Bringing Data Lake and Data Mesh Vision to Life
Evolving Big Data Strategies: Bringing Data Lake and Data Mesh Vision to Life
SG Analytics
 
How a Logical Data Fabric Enhances the Customer 360 View
How a Logical Data Fabric Enhances the Customer 360 ViewHow a Logical Data Fabric Enhances the Customer 360 View
How a Logical Data Fabric Enhances the Customer 360 View
Denodo
 
Your Data is Waiting. What are the Top 5 Trends for Data in 2022? (ASEAN)
Your Data is Waiting. What are the Top 5 Trends for Data in 2022? (ASEAN)Your Data is Waiting. What are the Top 5 Trends for Data in 2022? (ASEAN)
Your Data is Waiting. What are the Top 5 Trends for Data in 2022? (ASEAN)
Denodo
 
Accelerate Cloud Migrations and Architecture with Data Virtualization
Accelerate Cloud Migrations and Architecture with Data VirtualizationAccelerate Cloud Migrations and Architecture with Data Virtualization
Accelerate Cloud Migrations and Architecture with Data Virtualization
Denodo
 
Benefits of a data lake
Benefits of a data lake Benefits of a data lake
Benefits of a data lake
Sun Technologies
 
Big Data Evolution
Big Data EvolutionBig Data Evolution
Big Data Evolution
itnewsafrica
 
Data Virtualization: Introduction and Business Value (UK)
Data Virtualization: Introduction and Business Value (UK)Data Virtualization: Introduction and Business Value (UK)
Data Virtualization: Introduction and Business Value (UK)
Denodo
 
Data Fabric - Why Should Organizations Implement a Logical and Not a Physical...
Data Fabric - Why Should Organizations Implement a Logical and Not a Physical...Data Fabric - Why Should Organizations Implement a Logical and Not a Physical...
Data Fabric - Why Should Organizations Implement a Logical and Not a Physical...
Denodo
 
The Shifting Landscape of Data Integration
The Shifting Landscape of Data IntegrationThe Shifting Landscape of Data Integration
The Shifting Landscape of Data Integration
DATAVERSITY
 
IBM Relay 2015: Open for Data
IBM Relay 2015: Open for Data IBM Relay 2015: Open for Data
IBM Relay 2015: Open for Data
IBM
 
Data Virtualization, a Strategic IT Investment to Build Modern Enterprise Dat...
Data Virtualization, a Strategic IT Investment to Build Modern Enterprise Dat...Data Virtualization, a Strategic IT Investment to Build Modern Enterprise Dat...
Data Virtualization, a Strategic IT Investment to Build Modern Enterprise Dat...
Denodo
 
BAR360 open data platform presentation at DAMA, Sydney
BAR360 open data platform presentation at DAMA, SydneyBAR360 open data platform presentation at DAMA, Sydney
BAR360 open data platform presentation at DAMA, Sydney
Sai Paravastu
 

Similar to Data Mesh in Azure using Cloud Scale Analytics (WAF) (20)

Data Mesh using Microsoft Fabric
Data Mesh using Microsoft FabricData Mesh using Microsoft Fabric
Data Mesh using Microsoft Fabric
 
data-mesh_whitepaper_dec2021.pdf
data-mesh_whitepaper_dec2021.pdfdata-mesh_whitepaper_dec2021.pdf
data-mesh_whitepaper_dec2021.pdf
 
Data and Application Modernization in the Age of the Cloud
Data and Application Modernization in the Age of the CloudData and Application Modernization in the Age of the Cloud
Data and Application Modernization in the Age of the Cloud
 
Modernize your Infrastructure and Mobilize Your Data
Modernize your Infrastructure and Mobilize Your DataModernize your Infrastructure and Mobilize Your Data
Modernize your Infrastructure and Mobilize Your Data
 
When and How Data Lakes Fit into a Modern Data Architecture
When and How Data Lakes Fit into a Modern Data ArchitectureWhen and How Data Lakes Fit into a Modern Data Architecture
When and How Data Lakes Fit into a Modern Data Architecture
 
ANIn Pune July 2024 | Bootstrapping Data Mesh for a Complex Enterprise by Bal...
ANIn Pune July 2024 | Bootstrapping Data Mesh for a Complex Enterprise by Bal...ANIn Pune July 2024 | Bootstrapping Data Mesh for a Complex Enterprise by Bal...
ANIn Pune July 2024 | Bootstrapping Data Mesh for a Complex Enterprise by Bal...
 
Is your big data journey stalling? Take the Leap with Capgemini and Cloudera
Is your big data journey stalling? Take the Leap with Capgemini and ClouderaIs your big data journey stalling? Take the Leap with Capgemini and Cloudera
Is your big data journey stalling? Take the Leap with Capgemini and Cloudera
 
SMAC - Social, Mobile, Analytics and Cloud - An overview
SMAC - Social, Mobile, Analytics and Cloud - An overview SMAC - Social, Mobile, Analytics and Cloud - An overview
SMAC - Social, Mobile, Analytics and Cloud - An overview
 
Evolving Big Data Strategies: Bringing Data Lake and Data Mesh Vision to Life
Evolving Big Data Strategies: Bringing Data Lake and Data Mesh Vision to LifeEvolving Big Data Strategies: Bringing Data Lake and Data Mesh Vision to Life
Evolving Big Data Strategies: Bringing Data Lake and Data Mesh Vision to Life
 
How a Logical Data Fabric Enhances the Customer 360 View
How a Logical Data Fabric Enhances the Customer 360 ViewHow a Logical Data Fabric Enhances the Customer 360 View
How a Logical Data Fabric Enhances the Customer 360 View
 
Your Data is Waiting. What are the Top 5 Trends for Data in 2022? (ASEAN)
Your Data is Waiting. What are the Top 5 Trends for Data in 2022? (ASEAN)Your Data is Waiting. What are the Top 5 Trends for Data in 2022? (ASEAN)
Your Data is Waiting. What are the Top 5 Trends for Data in 2022? (ASEAN)
 
Accelerate Cloud Migrations and Architecture with Data Virtualization
Accelerate Cloud Migrations and Architecture with Data VirtualizationAccelerate Cloud Migrations and Architecture with Data Virtualization
Accelerate Cloud Migrations and Architecture with Data Virtualization
 
Benefits of a data lake
Benefits of a data lake Benefits of a data lake
Benefits of a data lake
 
Big Data Evolution
Big Data EvolutionBig Data Evolution
Big Data Evolution
 
Data Virtualization: Introduction and Business Value (UK)
Data Virtualization: Introduction and Business Value (UK)Data Virtualization: Introduction and Business Value (UK)
Data Virtualization: Introduction and Business Value (UK)
 
Data Fabric - Why Should Organizations Implement a Logical and Not a Physical...
Data Fabric - Why Should Organizations Implement a Logical and Not a Physical...Data Fabric - Why Should Organizations Implement a Logical and Not a Physical...
Data Fabric - Why Should Organizations Implement a Logical and Not a Physical...
 
The Shifting Landscape of Data Integration
The Shifting Landscape of Data IntegrationThe Shifting Landscape of Data Integration
The Shifting Landscape of Data Integration
 
IBM Relay 2015: Open for Data
IBM Relay 2015: Open for Data IBM Relay 2015: Open for Data
IBM Relay 2015: Open for Data
 
Data Virtualization, a Strategic IT Investment to Build Modern Enterprise Dat...
Data Virtualization, a Strategic IT Investment to Build Modern Enterprise Dat...Data Virtualization, a Strategic IT Investment to Build Modern Enterprise Dat...
Data Virtualization, a Strategic IT Investment to Build Modern Enterprise Dat...
 
BAR360 open data platform presentation at DAMA, Sydney
BAR360 open data platform presentation at DAMA, SydneyBAR360 open data platform presentation at DAMA, Sydney
BAR360 open data platform presentation at DAMA, Sydney
 

More from Nathan Bijnens

Dataminds - ML in Production
Dataminds - ML in ProductionDataminds - ML in Production
Dataminds - ML in Production
Nathan Bijnens
 
Azure Databricks & Spark @ Techorama 2018
Azure Databricks & Spark @ Techorama 2018Azure Databricks & Spark @ Techorama 2018
Azure Databricks & Spark @ Techorama 2018
Nathan Bijnens
 
Big Data Expo '18 - Microsoft AI
Big Data Expo '18 - Microsoft AIBig Data Expo '18 - Microsoft AI
Big Data Expo '18 - Microsoft AI
Nathan Bijnens
 
Spark on Azure, a gentle introduction (nov 2015)
Spark on Azure, a gentle introduction (nov 2015)Spark on Azure, a gentle introduction (nov 2015)
Spark on Azure, a gentle introduction (nov 2015)
Nathan Bijnens
 
Cloudera, Azure and Big Data at Cloudera Meetup '17
Cloudera, Azure and Big Data at Cloudera Meetup '17Cloudera, Azure and Big Data at Cloudera Meetup '17
Cloudera, Azure and Big Data at Cloudera Meetup '17
Nathan Bijnens
 
Microsoft AI at SAI '17
Microsoft AI at SAI '17Microsoft AI at SAI '17
Microsoft AI at SAI '17
Nathan Bijnens
 
Microsoft Advanced Analytics @ Data Science Ghent '16
Microsoft Advanced Analytics @ Data Science Ghent '16Microsoft Advanced Analytics @ Data Science Ghent '16
Microsoft Advanced Analytics @ Data Science Ghent '16
Nathan Bijnens
 
Virdata: lessons learned from the Internet of Things and M2M Cloud Services @...
Virdata: lessons learned from the Internet of Things and M2M Cloud Services @...Virdata: lessons learned from the Internet of Things and M2M Cloud Services @...
Virdata: lessons learned from the Internet of Things and M2M Cloud Services @...
Nathan Bijnens
 
A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...
A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...
A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...
Nathan Bijnens
 
a real-time architecture using Hadoop and Storm at Devoxx
a real-time architecture using Hadoop and Storm at Devoxxa real-time architecture using Hadoop and Storm at Devoxx
a real-time architecture using Hadoop and Storm at Devoxx
Nathan Bijnens
 
A real-time architecture using Hadoop and Storm @ JAX London
A real-time architecture using Hadoop and Storm @ JAX LondonA real-time architecture using Hadoop and Storm @ JAX London
A real-time architecture using Hadoop and Storm @ JAX London
Nathan Bijnens
 
A real-time architecture using Hadoop and Storm @ BigData.be
A real-time architecture using Hadoop and Storm @ BigData.beA real-time architecture using Hadoop and Storm @ BigData.be
A real-time architecture using Hadoop and Storm @ BigData.be
Nathan Bijnens
 
Microsoft Big Data @ SQLUG 2013
Microsoft Big Data @ SQLUG 2013Microsoft Big Data @ SQLUG 2013
Microsoft Big Data @ SQLUG 2013
Nathan Bijnens
 
A real time architecture using Hadoop and Storm @ FOSDEM 2013
A real time architecture using Hadoop and Storm @ FOSDEM 2013A real time architecture using Hadoop and Storm @ FOSDEM 2013
A real time architecture using Hadoop and Storm @ FOSDEM 2013
Nathan Bijnens
 
Getting more out of your big data
Getting more out of your big dataGetting more out of your big data
Getting more out of your big data
Nathan Bijnens
 
Hadoop Pig: MapReduce the easy way!
Hadoop Pig: MapReduce the easy way!Hadoop Pig: MapReduce the easy way!
Hadoop Pig: MapReduce the easy way!
Nathan Bijnens
 

More from Nathan Bijnens (16)

Dataminds - ML in Production
Dataminds - ML in ProductionDataminds - ML in Production
Dataminds - ML in Production
 
Azure Databricks & Spark @ Techorama 2018
Azure Databricks & Spark @ Techorama 2018Azure Databricks & Spark @ Techorama 2018
Azure Databricks & Spark @ Techorama 2018
 
Big Data Expo '18 - Microsoft AI
Big Data Expo '18 - Microsoft AIBig Data Expo '18 - Microsoft AI
Big Data Expo '18 - Microsoft AI
 
Spark on Azure, a gentle introduction (nov 2015)
Spark on Azure, a gentle introduction (nov 2015)Spark on Azure, a gentle introduction (nov 2015)
Spark on Azure, a gentle introduction (nov 2015)
 
Cloudera, Azure and Big Data at Cloudera Meetup '17
Cloudera, Azure and Big Data at Cloudera Meetup '17Cloudera, Azure and Big Data at Cloudera Meetup '17
Cloudera, Azure and Big Data at Cloudera Meetup '17
 
Microsoft AI at SAI '17
Microsoft AI at SAI '17Microsoft AI at SAI '17
Microsoft AI at SAI '17
 
Microsoft Advanced Analytics @ Data Science Ghent '16
Microsoft Advanced Analytics @ Data Science Ghent '16Microsoft Advanced Analytics @ Data Science Ghent '16
Microsoft Advanced Analytics @ Data Science Ghent '16
 
Virdata: lessons learned from the Internet of Things and M2M Cloud Services @...
Virdata: lessons learned from the Internet of Things and M2M Cloud Services @...Virdata: lessons learned from the Internet of Things and M2M Cloud Services @...
Virdata: lessons learned from the Internet of Things and M2M Cloud Services @...
 
A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...
A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...
A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...
 
a real-time architecture using Hadoop and Storm at Devoxx
a real-time architecture using Hadoop and Storm at Devoxxa real-time architecture using Hadoop and Storm at Devoxx
a real-time architecture using Hadoop and Storm at Devoxx
 
A real-time architecture using Hadoop and Storm @ JAX London
A real-time architecture using Hadoop and Storm @ JAX LondonA real-time architecture using Hadoop and Storm @ JAX London
A real-time architecture using Hadoop and Storm @ JAX London
 
A real-time architecture using Hadoop and Storm @ BigData.be
A real-time architecture using Hadoop and Storm @ BigData.beA real-time architecture using Hadoop and Storm @ BigData.be
A real-time architecture using Hadoop and Storm @ BigData.be
 
Microsoft Big Data @ SQLUG 2013
Microsoft Big Data @ SQLUG 2013Microsoft Big Data @ SQLUG 2013
Microsoft Big Data @ SQLUG 2013
 
A real time architecture using Hadoop and Storm @ FOSDEM 2013
A real time architecture using Hadoop and Storm @ FOSDEM 2013A real time architecture using Hadoop and Storm @ FOSDEM 2013
A real time architecture using Hadoop and Storm @ FOSDEM 2013
 
Getting more out of your big data
Getting more out of your big dataGetting more out of your big data
Getting more out of your big data
 
Hadoop Pig: MapReduce the easy way!
Hadoop Pig: MapReduce the easy way!Hadoop Pig: MapReduce the easy way!
Hadoop Pig: MapReduce the easy way!
 

Recently uploaded

VIP Kolkata Girls Call Kolkata 0X0000000X Doorstep High-Profile Girl Service ...
VIP Kolkata Girls Call Kolkata 0X0000000X Doorstep High-Profile Girl Service ...VIP Kolkata Girls Call Kolkata 0X0000000X Doorstep High-Profile Girl Service ...
VIP Kolkata Girls Call Kolkata 0X0000000X Doorstep High-Profile Girl Service ...
sukaniyasunnu
 
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDeliveryBDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
erynsouthern
 
New Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Avail...
New Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Avail...New Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Avail...
New Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Avail...
kinni singh$A17
 
Oracle Database Desupported Features on 23ai (Part B)
Oracle Database Desupported Features on 23ai (Part B)Oracle Database Desupported Features on 23ai (Part B)
Oracle Database Desupported Features on 23ai (Part B)
Alireza Kamrani
 
CHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptxCHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptx
girewiy968
 
🚂🚘 Premium Girls Call Nashik 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Nashik  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...🚂🚘 Premium Girls Call Nashik  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Nashik 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
kuldeepsharmaks8120
 
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
birajmohan012
 
社内勉強会資料_TransNeXt: Robust Foveal Visual Perception for Vision Transformers
社内勉強会資料_TransNeXt: Robust Foveal Visual Perception for Vision Transformers社内勉強会資料_TransNeXt: Robust Foveal Visual Perception for Vision Transformers
社内勉強会資料_TransNeXt: Robust Foveal Visual Perception for Vision Transformers
NABLAS株式会社
 
Lucknow Girls Call Indira Nagar Lucknow 08630512678 Provide Best And Top Girl...
Lucknow Girls Call Indira Nagar Lucknow 08630512678 Provide Best And Top Girl...Lucknow Girls Call Indira Nagar Lucknow 08630512678 Provide Best And Top Girl...
Lucknow Girls Call Indira Nagar Lucknow 08630512678 Provide Best And Top Girl...
bangaloreakshitakaus
 
Girls call in Hyderabad 000XX00000 Provide Best And Top Girl Service And No1 ...
Girls call in Hyderabad 000XX00000 Provide Best And Top Girl Service And No1 ...Girls call in Hyderabad 000XX00000 Provide Best And Top Girl Service And No1 ...
Girls call in Hyderabad 000XX00000 Provide Best And Top Girl Service And No1 ...
avanikakapoor
 
Histology of Muscle types histology o.ppt
Histology of Muscle types histology o.pptHistology of Muscle types histology o.ppt
Histology of Muscle types histology o.ppt
SamanArshad11
 
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
44annissa
 
DataScienceConcept_Kanchana_Weerasinghe.pptx
DataScienceConcept_Kanchana_Weerasinghe.pptxDataScienceConcept_Kanchana_Weerasinghe.pptx
DataScienceConcept_Kanchana_Weerasinghe.pptx
Kanchana Weerasinghe
 
Premium Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Premium Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Premium Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Premium Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
6459astrid
 
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
fatima shekh$A17
 
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
norina2645
 
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
kinni singh$A17
 
DU degree offer diploma Transcript
DU degree offer diploma TranscriptDU degree offer diploma Transcript
DU degree offer diploma Transcript
uapta
 
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
saadkhan1485265
 
Potential Uses of the Floyd-Warshall Algorithm as appropriate
Potential Uses of the Floyd-Warshall Algorithm as appropriatePotential Uses of the Floyd-Warshall Algorithm as appropriate
Potential Uses of the Floyd-Warshall Algorithm as appropriate
huseindihon
 

Recently uploaded (20)

VIP Kolkata Girls Call Kolkata 0X0000000X Doorstep High-Profile Girl Service ...
VIP Kolkata Girls Call Kolkata 0X0000000X Doorstep High-Profile Girl Service ...VIP Kolkata Girls Call Kolkata 0X0000000X Doorstep High-Profile Girl Service ...
VIP Kolkata Girls Call Kolkata 0X0000000X Doorstep High-Profile Girl Service ...
 
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDeliveryBDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
 
New Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Avail...
New Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Avail...New Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Avail...
New Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Avail...
 
Oracle Database Desupported Features on 23ai (Part B)
Oracle Database Desupported Features on 23ai (Part B)Oracle Database Desupported Features on 23ai (Part B)
Oracle Database Desupported Features on 23ai (Part B)
 
CHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptxCHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptx
 
🚂🚘 Premium Girls Call Nashik 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Nashik  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...🚂🚘 Premium Girls Call Nashik  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Nashik 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
 
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
 
社内勉強会資料_TransNeXt: Robust Foveal Visual Perception for Vision Transformers
社内勉強会資料_TransNeXt: Robust Foveal Visual Perception for Vision Transformers社内勉強会資料_TransNeXt: Robust Foveal Visual Perception for Vision Transformers
社内勉強会資料_TransNeXt: Robust Foveal Visual Perception for Vision Transformers
 
Lucknow Girls Call Indira Nagar Lucknow 08630512678 Provide Best And Top Girl...
Lucknow Girls Call Indira Nagar Lucknow 08630512678 Provide Best And Top Girl...Lucknow Girls Call Indira Nagar Lucknow 08630512678 Provide Best And Top Girl...
Lucknow Girls Call Indira Nagar Lucknow 08630512678 Provide Best And Top Girl...
 
Girls call in Hyderabad 000XX00000 Provide Best And Top Girl Service And No1 ...
Girls call in Hyderabad 000XX00000 Provide Best And Top Girl Service And No1 ...Girls call in Hyderabad 000XX00000 Provide Best And Top Girl Service And No1 ...
Girls call in Hyderabad 000XX00000 Provide Best And Top Girl Service And No1 ...
 
Histology of Muscle types histology o.ppt
Histology of Muscle types histology o.pptHistology of Muscle types histology o.ppt
Histology of Muscle types histology o.ppt
 
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
 
DataScienceConcept_Kanchana_Weerasinghe.pptx
DataScienceConcept_Kanchana_Weerasinghe.pptxDataScienceConcept_Kanchana_Weerasinghe.pptx
DataScienceConcept_Kanchana_Weerasinghe.pptx
 
Premium Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Premium Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Premium Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Premium Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
 
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
 
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
 
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
 
DU degree offer diploma Transcript
DU degree offer diploma TranscriptDU degree offer diploma Transcript
DU degree offer diploma Transcript
 
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
 
Potential Uses of the Floyd-Warshall Algorithm as appropriate
Potential Uses of the Floyd-Warshall Algorithm as appropriatePotential Uses of the Floyd-Warshall Algorithm as appropriate
Potential Uses of the Floyd-Warshall Algorithm as appropriate
 

Data Mesh in Azure using Cloud Scale Analytics (WAF)

  • 1. Nathan Bijnens Manager, Belux CSU Data Team Data Mesh in Azure using Cloud Scale Analytics
  • 2. What we’ve heard To spend less time preparing data Robust data governance Platform to actionable Insights to the business Ability to increase the value of hidden data Improve Operational Efficiency Ideally, organizations want to have….. Reduce cost of data engineering Need for Frictionless Data Governance Difficult to balance access and data protection Data and Analytics Operationalization Enable Lines of Businesses Poor data quality Disparate systems and data silos Too slow moving from data to decision Barriers to achieve business outcomes Unified ecosystem Project prioritization
  • 3. Every application that creates data, needs and will have a database Application A Application B Consequently, when we have two applications, we hypothesize that each application has its own ‘database’. When there is interoperability between these two applications, we expect data to be transferred from one application to the other. Every application, at least in the context of data management, that creates data, needs and will have a database. Even stateless applications that create data have “databases”. In these scenarios the database typically sits in the RAM or in a temp file.
  • 4. We can’t escape from data integration Application A Application B The ‘always’ required data transformation lies in the fact that an application database schema is designed to meet the application’s specific requirements. Since the requirements differ from application to application, the schemas are expected to be different and data integration is always required when moving data around. A crucial aspect when it comes to data transfer is that data integration is always right around the corner. Whether you do ETL or ELT, virtual or physical, batch or real-time, there’s no escape from the data integration* dilemma. Data integration
  • 5. Business Drivers •Lack of data ownership Lack of data quality Difficult to see interdependencies Model conflicts across business concerns Tremendous effort for integration and coordination leads to bypasses Business and IT work in silos Disconnect between the data producers and data consumers Central team becomes the bottleneck Difficult to apply policy and governance Hard to see technical dependencies Small changes become risky due to unexpected consequences Technical ownership rather than data ownership Many Enterprises are saddled with outdated Data Architectures that do not scale to the needs of large multi- disciplinary organizations.
  • 6. Problems with Existing Architectures There’s a deep assumption that centralization is the solution to data management. This includes centralizing all data and management activities into one central team, building one data platform, using one ETL framework, using one canonical model, etc. Transactional Sources Analytical Consumers Centralized Architecture • Single team with centralized knowledge and book of work • Centralized pipelines for all extraction / ingestion activities • Centralized transformations to create harmonized data • Central platform serves as large integration database: all execution and analysis is done on the same platform Data providers Data consumers Central engineering team Transactional Sources Transactional Sources Analytical Consumers Analytical Consumers
  • 7. Transformational Trends in the Data Landscape Massive increase of computing power, driven by hardware innovation (SSD storage, in- memory storage, GPU advances) lets us move data to compute faster. Cloud and APIs make it easier to integrate. Software & Platform as a Service (SaaS, PaaS) offerings push the connectivity and API usage even further. Explosion of tools New (open source) concepts are introduced, such as NoSQL database types, block chain, new database designs, distributed models (Hadoop), new analytical methods, etc. Exponential growth of data, especially external data sources like open and social data. Internal, external, structured, and unstructured data are all used to deliver additional insights. Eco-system connectivity Exponential growth of data Increase of computing power Stronger regulatory requirements, such as GDPR and BCBS 239, are coming into effect worldwide. Data quality and lineage become more important every day. Increased regulatory attention The read/write ratio has changed due to more intensive data consumption: data is read more often, there is increased real-time consumption and more searches are performed. Increase of read/write ratio
  • 8. Data as a Product
  • 9. Data as a Product Data is no longer a side-effect, it’s a product. Who are my "customers"? What do my "customers" need? Are they happy with the data? Are they using it? How do I let my "customers" know my data exists? What is in it for the "customer"?
  • 10. Data Product Owner Domain Data Product Owner Data Engineer Software Developer Infra Engineer How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh (martinfowler.com) Zhamak Dehghani
  • 11. Data Product Properties How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh (martinfowler.com) Zhamak Dehghani • Overview of product in central data catalog • Provide easy discoverability Discoverable • Help users access the product programmatically Addressable • Data Product Owners provide monitored SLOs • Data is cleansed and up to standard Trustworthy • Minimal friction for data engineers and scientists to use the data Self-describing • Open standards for harmonization • Field type formatting Interoperable • Access control policies • Use SSO and RBAC Secure
  • 13. Data Mesh Data Mesh is a new decentralized socio-technical approach to managing data, designed to work with organizational complexity and continuous growth. It enables large organizations to get value from their data, at scale, through reusability, analytics and ML. It is building on the Domain Driven Design methodology. Data Mesh Domain Driven Design Domain Zones Data Products Consumed by other Domains How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh (martinfowler.com) Zhamak Dehghani
  • 14. Centralized Implementation is not working! GSR Finance HR Travel Sales Clinical Ops Centralized Platform LOBs are the SMEs and Shared Service team is not able to cope up with the projects Datasets sprawls Competing needs within the organization • IT needs to standardize • LOBs need to implement analytics Primitive Data Strategy
  • 15. Introduction to Data Domains Search Keywords Promotions Top Selling Products Orders Customer Profiles Data Products Integration Services Operational Systems Marketing Domain Customer Services Domain Order Management Domain • A domain is a collection of people, typically organized around a common business purpose. • Create and serve data products to other domains and end users, independently from other domains. • Ensure data is accessible, usable, available, and meets the quality criteria defined. • Evolve data products based on user feedback and retire data products when they become irrelevant.
  • 16. Domain Zones Engineering Finance HR Innovation Program 1 Operations Management zone Data products Data Domains Microsoft Enterprise Data Mesh
  • 17. Domain Zone Domain Zone Environment for each LOB LOBs: Implement Data Services • ex: Exploration Service, Data Order System LOBs: Build and Share Data Products • ex: Sales Forecast, Clean Room Performance Automated using templates • security, integration, monitoring, etc
  • 18. E N T E R P R I S E R E Q U I R E M E N T S Security & Privacy Governance & Compliance Availability & Recovery Performance & Scalability Skills & Training Licensing & Usage Observation & Monitoring Domain Architecture
  • 19. Shift towards Domain Ownership A new type of eco-system architecture, which shifts to the left towards a modern distributed architecture that enables domain-specific data and data products, empowering each domain to handle their own data pipelines. Supporting governance and domain-agnostic platform infrastructure Data Providers Data Product Data Providers Data Product Data Providers Data Product Source-oriented Domains Consumer- specific Transformation Data Consumer Consumer- specific Transformation Data Consumer Consumer- specific Transformation Data Consumer Consumption-oriented Domains
  • 20. Domain Zones Data Products Domain Zone HR Recruitment Time Tracking Employee Value And Performance Training and Development Engagement and Retention Engineering Operations New Project : Digital Twin Clean Room Personnel
  • 21. • Map your data domains organically, during the onboarding of data providers and consumers. • Reference your business capabilities (e.g., strategy and processes) while mapping your data domains. • Isolate your data domains and enable communication through data products like APIs or events. • Create and document a shared, ubiquitous language that different domains can use to communicate. • Determine boundaries for both business and technical granularity. Data Domain Considerations from the Field
  • 23. Enterprise Scale: Azure Landing Zones The main purpose of a “Landing Zone” is to ensure that when a workload lands on Azure, the required “plumbing” is already in place, providing greater agility and compliance with enterprise security and governance requirements.
  • 24. Data Management Landing Zone Data Management Landing Zone Business Glossary Data Discovery SLAs Business Rules Ref. Data Mgmt. Master Record Mgmt. Data Policy Access Governance Loss Prevention Privacy Operations Risk Assessment Repository for Data Models Integration API Documentation Automation for provisioning landing zones, data integrations, and products Pre-configured network and monitoring setup Standard images for deploying analytics and AI services Azure Subscription Azure Policy
  • 25. Data Landing Zone Core Networking Shared Products Ingest and Processing Upload Data Lake Services Metadata Services Preconfigured network and monitoring setup Data lake configured with layers and connectivity Spark and scheduling engines Blobs where 3rd parties can upload their data Scanners for data governance/metadata required by landing zone Analytics engines for exploratory analytics Data Integration Data Integration # Data Integration Teams are responsible for the ingestion of data to a read data source. The data shouldn’t have any data transformation applied apart from data quality checks and data type verification. Data Integration # Pull SAP Data into Landing Zone # Streaming interface to pull data from heat sensors Data Products Data Product # Data Product # Financial Reporting pulling Customers and Sales together Streaming Machine Data from Read Data Source Data products fulfil a specific business need using data. Data products manage, organize, and make sense of data across domains and present the insights gained from the data products. A data product is a result from one or many data integrations and/or other data products. Infrastructure as Code
  • 26. Azure Event Hubs Azure Data Lake Store Gen2 Storing read-optimized domain data Data Product Team Data Product Team Data Product Team Data Product Team Data Onboarding Team Data Integration Synapse Analytics Data Product Team Data Product Team Data Product Team Real-time applications, operational systems Self-service BI, semantic models Analytical applications Data Engineering Team Data Management Landing Zone Data Governance Team Azure Purview Data Lake Services Azure Data Factory Transforming into read- optimized data products Data Integration Data Integration Data Landing Zone Azure Databricks Shared Service Data-driven applications Data Product Data Product Example Reference Architecture for Data Mesh in a Small Company Data Product
  • 27. Optimize Existing Implementation Patterns Take a new approach to data management that supports and evolves with your strategy. The data management and analytics scenario supports a range of patterns to build on your current data infrastructure, to help you modernize and scale from where you are. Data Warehouse Data Lake Data Lakehouse Data Mesh Data Fabric
  • 28. Integrating your DWH in a Data Mesh  From be-all end-all to yet another Data Product in your mesh  Ownership based on your preference  DWH is data product on its own: managed by one data product team  DWH serves as "wrapper" for multiple data products: managed by multiple teams  DWH consumes data from multiple Data Products  Multiple Data Products consume data from DWH
  • 29. Agile Data Management Enforce data governance and security.  Serve data as a product rather than a byproduct.  Provide an ecosystem of data products.  Create data domains to serve lines of business.  Empower teams to drive analytics solutions that deliver value to the business.  Modernize your teams and operations.  Prepare your company to:
  • 31. Contoso Managem ent zone Data products Data Domains Multi Organization Data Mesh Finance HR Organization Contoso Managem ent zone Finance HR Contoso Managem ent zone Finance HR
  • 32. Interested in learning more? Reach out to Nathan.Bijnens@microsoft.com
  • 33. Links DDD Best Practice - An Introduction To Domain-Driven Design | Microsoft Docs Introduction into Domain-Driven Design (DDD) (jannikwempe.com) IBM Automation Event-Driven Reference Architecture – Domain Driven Design (ibm- cloud-architecture.github.io) Data Mesh How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh (martinfowler.com) Data Mesh in Practice: How Europe's Leading Online Platform for Fashion Goes Beyond the Data Lake - Databricks