[DSC Europe 23] Predrag Ilic & Simeon Rilling - From Data Lakes to Data Mesh - The Evolution of redmesh

From Data Lakes to Data Mesh
The evolution of redmesh
DSC Europe 23
Predrag Ilić – Cloud Tech Lead
Rilling Simeon– Product Owner Enterprise Data Mesh

Intern | HRL-Bg | 10.11.2023
© Robert Bosch d.o.o. 2023. Alle Rechte vorbehalten, auch bzgl. jeder Verfügung, Verwertung, Reproduktion, Bearbeitung, Weitergabe sowie für den Fall von Schutzrechtsanmeldungen.
Together
we shape
tomorrow

Intern | HRL-Bg | 10.11.2023
Bosch Digital: customer centricity and business outcomes
Bosch Digital
Digital/IT Products, Services,
Solutions and Projects
Bosch divisions & central functions
For our customers we deliver outstanding digital products,
services and solutions for growth and efficiency.
We are the trusted partner for digital business and
enterprise IT for Bosch.
We empower our diverse talents that are united by the
passion to drive digital transformation in an agile
environment.

Intern | HRL-Bg | 10.11.2023
Data mesh as enabler for business
Data 1.0: Local Databases
Decentralized & Non-Standard
From databases…
Data 2.0: Data Lakes
Centralized & Standardized
…via data lakes…

Intern | HRL-Bg | 10.11.2023
Data mesh as enabler for business
Data 2.0: Data Lakes
Centralized & Standardized
…via data lakes…
Data 3.0: Data Mesh
Federated & Standardized
…to data mesh
„Data mesh is a decentralized sociotechnical approach to share, access and manage analytical data in complex and large-
scale environments – within or across organizations.“ Zhamak Dehghani

Intern | HRL-Bg | 10.11.2023
Data mesh principles
Domain Ownership
Responsibility and
ownership of data and
transformation pipelines lies
with those who know the
data best.
Data as a Product
Product thinking is applied
on data sets, and each data
product is described by a
set of properties and
possesses certain
capabilities.
Self-Serve
Technology, which is needed
by the business teams to
create, store, and offer their
data products, is owned by a
central data infrastructure
team, and provided in the
form of self-service
offerings.
Federated Governance
Interoperability between the
independently created data
products is ensured by a set
of global rules (e.g. roles),
which is defined by a
federation of domain data
product owners and data
platform product
owners.
Data mesh principles* by Zhamak Dehghani in redmesh: “The data mesh platform is an intentionally designed distributed data architecture, under
centralized governance and standardization for interoperability, enabled by a shared and harmonized self-serve data infrastructure.”

Intern | HRL-Bg | 10.11.2023
Data domains
Self-Serve Data Platform
Enabling
Enterprise IT User generated IoT Web
…
• Governance
• Consulting
• Data
marketplace
• Compliance
• DataOps
Toolsuite
• …
Clear ownership of data
and data products.
Fast and flexible
implementation within
domain teams.
Structured in matrix with
process and business
domains as dimensions.
Finance
Logistics
…
Automotive
Data Mesh Domains
Sustainability
Storage, Pipelines, Data Catalogue, Access Control, …

Intern | HRL-Bg | 10.11.2023
Data as a Product
Data
Product
1
2
3
4
5
Value
Data & Technology
Consumable & Interoperable
Contract
Data Marketplace
Analytical data should be
treated as a product and
consumers as customers.
Provides business context to
the data and ensures high
quality.
Example: Stock quantity in a
warehouse
Up to
40%
cost savings for
application
development
Up to
60%
faster ad-hoc
analysis using
data products

Intern | HRL-Bg | 10.11.2023
Federated Governance
Bosch Global Governance
Focus: Handling data as assets
Definition of roles for data management; Management of terminologies,
data models, and metadata; Security and compliance
redmesh Global Rules (Federated Governance Board)
Focus: Creating interoperability
Facilitating discovery and understandability; Ensuring compliance;
Deciding on cross-domain topics
Domain-Local Governance
Focus: Creation and offering of high-quality data products
Definition of domain-owned data products and data models; Decision on technologies
and methods; Definition of data sharing agreements and SLAs
Ensure interoperability by minimum
set of rules while respecting
autonomy of local domains.
Example: standards for data
consumption, architecture patterns,
product and technology catalog

Intern | HRL-Bg | 10.11.2023
Self-serve data platform
Sales
Logistics
Quality
Decentralized Data Products and Domains
… Mobility …
Finance
Regional
satellite
Data Visualization (excerpt)
Data Mesh
• Global and GE policies
• Data ownership
• Data catalogue
• Data models
Distributed Platforms
Federated &
standardized platform
increases speed, flexibility and
scalability.
Reporting &
Dashboarding
Exploration
Planning &
Simulation
Data / Process
Mining
on-premise data lake
cloud data lake
>800 TB
of data
stored on
distributed
technical
platform
>36k
tables
continuously
replicated in
near-realtime
from SAP
220%
increase of
data
consumptio
n in 2023

Intern | HRL-Bg | 10.11.2023
Product and service offering
Data Onboarding
Service
Exploration
zone
Data Provisioning
Service
Data Onboarding
Service
Business
Satellite
Data Onboarding
Service
Big Data &
AI Platforms
Relational data
lake
Data Marketplace
Data Modelling
redmesh
Data Catalogue
Enabling &
Consulting
Legend
on-premise
cloud
general
platform
service
Data lake

Intern | HRL-Bg | 10.11.2023
Business Satellite deep dive
• Flexible data consumption –
Consumption patterns
• Distributed architecture – Technology
agnostic
• Faster implementations – pattern
study, automation
• Stable environment – Maintenance by
reusable components
• Forerunner cloud data consumption for
analytics
• Trend: Cloud analytics
• Big workloads
• Autonomous data platform
• REDLake data – On cloud
• Distributed Architecture
• Data consumption
mechanisms

Intern | HRL-Bg | 10.11.2023
Business Satellite service map
On premise
Data lake
Customer Subscription
On Premise
sources
Cloud
sources
redmesh
Product Data Integration Azure subscription (redmesh)
Product development – data storage
– data consumption
Platform maintenance service
- Resource maintenance
- Role and IdM Integration
Solution Azure Subscription (Customer)
Application development
1 2 3
redmesh Azure
Security compliant with Bosch
directives
Big workloads, varied source and
data varieties
Synergize data access
Streamline data consumption
Support different data formats

Intern | HRL-Bg | 10.11.2023
redmesh Azure Subscription
On-premise
data lake
Sources on-premise / public
cloud integration
Proxy4server
Batch Ingest
MS
Integration
Runtime Azure Data Factory
Batch Storage
Data Lake Gen2
Shared Services
DevOps
Customer Application Subscription
Stream
Processing
Serving
Data
Analytics &
AI
https
Bastion Policy
Log
Analytics
Security
Center
Batch Data Sources
Stream Data
Sources
Creation &
Scheduling
of ETL jobs
Customizing
Metadata
Landing
Delivery
Refine
Harmonize
Raw
Batch
Processing
SQL/spark
Polybase
SQL
1
2
2.2
2.1
1
Replication setup to redmesh Azure
Integration Framework – consumption
oriented
Enterpris
e Data
IoT
Data
1.1
1.1
2.1 2.2
2
Blueprints:
Architecture Patterns: Workspace, Databases, schema – standardized for data analytics
Data Management: Data flow, resource, performance and security monitoring
Access Management: Roles, user assignments, IdM Integration (Access Management)
Monitoring and Support
Enterprise
Data
Replication
Data Lake
Management
Distributed redmesh Architecture
Solution & System Architecture Overview

Intern | HRL-Bg | 10.11.2023
redmesh Azure Subscription
On-premise
data lake
Sources on-premise / public
cloud integration
Proxy4server
Batch Ingest
MS
Integration
Runtime Azure Data Factory
Batch Storage
Data Lake Gen2
Shared Services
DevOps
DMS Azure Subscription(s)
Business Domain Scalability Options
< schema >
< pool >
< workspace >
< subscription >
Data
Analytics &
AI
https
Bastion Policy
Log
Analytics
Security
Center
Batch Data Sources
Stream Data
Sources
Creation &
Scheduling
of ETL jobs
Customizing
Metadata
Landing
Delivery
Refine
Harmonize
Raw
Batch
Processing
SQL/spark
1
2
2.2
1
Replication setup to redmesh Azure
Integration Framework – consumption
oriented
Enterpris
e Data
IoT
Data
1.1
1.1
2.1 2.2
2
Blueprints:
Architecture Patterns: Workspace, Databases, schema – standardized for data analytics
Data Management: Data flow, resource, performance and security monitoring
Access Management: Roles, user assignments, IdM Integration (Access Management)
Monitoring and Support
Enterprise
Data
Replication
Data Zone(s)
Management
Serving &
Modelling
Serverless
Dedicated
Spark
2.1
Delta Lake
redmesh & DMS Azure – Retrospective (2022)
Solution & System Architecture Overview
Exploration
Service
Business
Satellites

Intern | HRL-Bg | 10.11.2023
redmesh use-cases (excerpt)
SusAn (Sustainability Analytics Platform)
Driving sustainability as basis for future growth by integration of data
from various business processes of several entities.
Supply Chain Network Control Tower
Enabling product traceability through the full manufacturing life cycle,
specifically including serial numbers and SAP recordings.
Chatbots
Providing a central, modular and integrated data hub for chatbots based
on data products and API marketplace.
Mobility Data and Analytics
Providing easy and fast access to data to our automotive business to
improve our competitiveness through data driven solutions.
Compliance
with EU CSRD
Steet towards
sustainability
Competitive
advantage
Near-realtime
analysis
Simulation
product flow
Demand
forecast
Resource
management
EOS
prediction
Stock
reduction
Fast
development
API
enabled
Reusable
components

Intern | HRL-Bg | 10.11.2023
Key take-aways
1
Data mesh approach as evolution of data lake enables customers with flexibility and
autonomy.
2
Data mesh principles need to be implemented in accordance with business and IT
environment.
3
Self-serve data platform with high degree of automation is the technical backbone of a
data-driven enterprise.
4
Data mesh enables a variety of business use-cases and creates the foundation for
economical success.

Together
we shape
tomorrow
Thank You!

[DSC Europe 23] Predrag Ilic & Simeon Rilling - From Data Lakes to Data Mesh - The Evolution of redmesh

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to [DSC Europe 23] Predrag Ilic & Simeon Rilling - From Data Lakes to Data Mesh - The Evolution of redmesh

Similar to [DSC Europe 23] Predrag Ilic & Simeon Rilling - From Data Lakes to Data Mesh - The Evolution of redmesh (20)

More from DataScienceConferenc1

More from DataScienceConferenc1 (20)

Recently uploaded

Recently uploaded (20)

[DSC Europe 23] Predrag Ilic & Simeon Rilling - From Data Lakes to Data Mesh - The Evolution of redmesh