Transforming Data Management
for the Cloud
Transforming Data Management in the
Cloud with the Denodo Platform
Paul Moxon
SVP Data Architecture and Chief Evangelist
Denodo
Prologis
4
$2.7 TRILLION
is the economic value of goods flowing through
our distribution centers each year, representing:
4.
0 %
of GDP for the 19 countries where
we do business
%
2.8
of the World’s GDP
1983 100
GLOBA
L 1,200MSF
Founded Most sustainable corporations
$196B
Assets under management on four continents
MILLION
employees under Prologis’ roofs
1.1
Prologis – Global Industrial Real Estate Company
5
Prologis RFI DIRFT (Daventry, UK)
6
Prologis – Existing Architecture
7
Seamless Migration to Snowflake
• Large or critical Cloud migrations are
risky
• Big Bang approach is not advised
• Phased approach is recommended
• Select data set to migrate, copy to Cloud
• Test and tune data access, then go live
• Repeat for next data set and so on
• Use Denodo as abstraction layer during
migration process
• Isolate users from shift of data
8
Prologis – New Hybrid Architecture
9
DATA FLOW
• Create a virtual representation of the physical tables
from legacy DWH in Denodo Cloud Platform.
• Connect other on-prem data sources to Denodo. Build
the foundation for a logical data warehouse or logical
data lake.
• Start moving physical objects from on-prem Data
Warehouse to Snowflake in bite-sized chunks to avoid
any downtime and ensure proper testing procedures.
• Switch the connection from legacy DWH to Snowflake
Cloud Platform inside Denodo.
• Maintain one consistent business data model across all
consumers and reporting tools. Reuse analytical objects
across multiple tools and consuming applications.
1
2
3
4
5
Example – Zero-Downtime Migration to Snowflake
Landsbankinn
11
Landsbankinn
• Leading financial institution in Iceland
• 40% Market share Individual Banking
• 33% Market share Corporate Banking
• Best ESG risk ratings amongst European
banks (Sustainalytics 2021)
• Best bank at the Icelandic consumer
satisfaction ratings (Ánægjuvogin / Stjórnvísi 2021)
12
SAS environment
Year Zero - Before Data Virtualization
▪ Too many query points
▪ Heterogenous technologies
▪ Complex source systems
▪ Scattered business rules
▪ Semantic layers in BI
▪ Business logics in DB views
▪ Many points of access control
▪ Audit points all over the place
▪ Each system has its own access control
KPI DB Source DBs New DWH Old DWH Markets DB
Views
BO
reporting
Self-service
BI
PDF
statements
MS Office
Integration
Views
Views
Views
General Reporting
KPI
Self-Service
data
Analytics
Reports
Analytics
Server
Risk Reporting
Monitoring / Audit Business security
Business rules
Board
Other DBs
SAP BO Semantic Layer
Data
Sources
Semantic
Layer
13
Year 1 - The Logical Data Warehouse
▪ Unique point of query
▪ “Need data? LDW has the answer!”
▪ For reporting, analytics, APIs, …
▪ Unique point of truth
▪ Business logic repository
▪ Lineage available
▪ Unique point of access control
▪ Unified access to the data
▪ Unique point of auditing
KPI DB Source DBs New DWH Old DWH Markets DB
BO
reporting
Self-service
BI
MS Office
Integration
General Reporting
KPI Self-Service
data
Analytics
Reports
Analytics
Server
Risk Reporting
Board
Other DBs
Data
Sources
Logical Data Warehouse w/ Denodo
Monitoring / Audit Business security
Business rules
PDF
statements
14
Years 2 and 3 - Expansion and Modernization
▪ Addition of data consumers
▪ Tableau
▪ REST / Restful APIs
▪ Addition of more data sources
▪ Where ETL is not required
▪ When history is provided in source
▪ Logical data pipelines
▪ Reduces the number of ETL jobs
▪ EDW gets data from LDW
BO
Reporting Tableau
RestWS to
Excel
General Reporting
KPI
Self-Service
data
Analytics
Reports
Analytics
Server
Risk Reporting
Board
Data
Sources
Logical Data Warehouse w/ Denodo
KPI DB
Source
DBs
New
DWH
Old
DWH
Markets
DB
Other
DBs
Flat files
Excel
SaaS
REST
SOAP
WWW
Customers
Domains
Operational
systems
Monitoring / Audit Business security
Business rules
Customer
statements
Leading Global Bank
16
Leading Global Bank – Pain Points
Supporting
Multiple Data
Access Tools
Changing
Technologies
Data
Lifecycle
Management
Data
Discovery
17
Leading Global Bank – Data Marketplace Objectives
DATA DISCOVERY
SECURITY
CONSISTENT
ACCESS
INTERFACE
REDUCE BARRIERS TO ADOPTION
DATA ACCESS
AGILITY
02
03
04
05
06
01
18
Leading Global Bank – Data Marketplace
Data Virtualization Platform
Data Marketplace
Client & Account
Active Clients Client
Accounts
Party Summary
Positions & Holdings Securities & Pricing Market Data Hub Index & Benchmark
Systems of Record Data Lake Data Warehouse
with Business Semantic Layer
Virtual Data Lake
BHP
20
About BHP
Company Profile and Background
• Anglo-Australian multinational mining, metals and petroleum dual-listed public company headquartered in Melbourne, Victoria,
Australia.
• BHP ranked as the world's largest mining company, based on market capitalization, and as Melbourne's third-largest
company by revenue,
• BHP has mining operations in Australia, North America, and South America, and petroleum operations in the U.S., Australia,
Trinidad and Tobago, UK, and Algeria.
• The company has four primary operational units
• Coal
• Copper
• Iron ore
• Petroleum
• No of Employees : 80,000
• Revenue : US$65.098 billion (2022)
21
BHP – Globally Distributed Data and Users
Houston DC
Santiago DC
Perth DC Brisbane
AWS US East
Escondida
Jansen London
Singapore
Kuala Lumpor
Shanghai
AWS
APAC
22
BHP – Global Data Fabric
Houston DC
Santiago DC
Perth DC Brisbane
AWS US East
AWS
APAC
Escondida
Jansen London
Singapore
Kuala Lumpor
Shanghai
Every Data Virtualization cluster is connected to local
data sources, and is the access point for local
consumer apps such as BI and analytics tools. Each
Data Virtualization cluster has visibility of the datasets
available from all other clusters, and requests this data
from it's peer cluster as required by end users
23
BHP – Global Data Fabric Infrastructure
24
Denodo Platform: The Foundation of a Logical Data Architecture
Agile Data
Integration
Logical Data
Abstraction
Smart Query
Acceleration
Advanced
Semantics
Automation &
Recommendation
Unified Security
& Governance
Data Catalog
AI/ML
6 Key Capabilities of Logical Data Management Differentiated Use Cases
Hybrid/Multi-Cloud
Data Integration
Data Marketplace/
Self-Service Analytics
Governance &
Compliance
3600
View of Entities
(e.g., Customer)
Accelerated Integration
for M&A Activities
Data Democratization
Enterprise Data Services
Data Fabric/ Data Mesh
25
Benefits of a Logical Data Architecture
“Now, we can do weekly releases.
We’re able to add new data sources
within 2 to 3 hours. We’re about 60%
faster than we were in the old world.”
VP of data and analytics, real estate
“To me, it all boils down to speed to
insights. Not having to wait to get the
question that you have top-of-mind
answered with data is huge.”
VP of data and analytics, real estate
Thank you!

Transforming Data Management in the Cloud with the Denodo Platform

  • 1.
  • 2.
    Transforming Data Managementin the Cloud with the Denodo Platform Paul Moxon SVP Data Architecture and Chief Evangelist Denodo
  • 3.
  • 4.
    4 $2.7 TRILLION is theeconomic value of goods flowing through our distribution centers each year, representing: 4. 0 % of GDP for the 19 countries where we do business % 2.8 of the World’s GDP 1983 100 GLOBA L 1,200MSF Founded Most sustainable corporations $196B Assets under management on four continents MILLION employees under Prologis’ roofs 1.1 Prologis – Global Industrial Real Estate Company
  • 5.
    5 Prologis RFI DIRFT(Daventry, UK)
  • 6.
  • 7.
    7 Seamless Migration toSnowflake • Large or critical Cloud migrations are risky • Big Bang approach is not advised • Phased approach is recommended • Select data set to migrate, copy to Cloud • Test and tune data access, then go live • Repeat for next data set and so on • Use Denodo as abstraction layer during migration process • Isolate users from shift of data
  • 8.
    8 Prologis – NewHybrid Architecture
  • 9.
    9 DATA FLOW • Createa virtual representation of the physical tables from legacy DWH in Denodo Cloud Platform. • Connect other on-prem data sources to Denodo. Build the foundation for a logical data warehouse or logical data lake. • Start moving physical objects from on-prem Data Warehouse to Snowflake in bite-sized chunks to avoid any downtime and ensure proper testing procedures. • Switch the connection from legacy DWH to Snowflake Cloud Platform inside Denodo. • Maintain one consistent business data model across all consumers and reporting tools. Reuse analytical objects across multiple tools and consuming applications. 1 2 3 4 5 Example – Zero-Downtime Migration to Snowflake
  • 10.
  • 11.
    11 Landsbankinn • Leading financialinstitution in Iceland • 40% Market share Individual Banking • 33% Market share Corporate Banking • Best ESG risk ratings amongst European banks (Sustainalytics 2021) • Best bank at the Icelandic consumer satisfaction ratings (Ánægjuvogin / Stjórnvísi 2021)
  • 12.
    12 SAS environment Year Zero- Before Data Virtualization ▪ Too many query points ▪ Heterogenous technologies ▪ Complex source systems ▪ Scattered business rules ▪ Semantic layers in BI ▪ Business logics in DB views ▪ Many points of access control ▪ Audit points all over the place ▪ Each system has its own access control KPI DB Source DBs New DWH Old DWH Markets DB Views BO reporting Self-service BI PDF statements MS Office Integration Views Views Views General Reporting KPI Self-Service data Analytics Reports Analytics Server Risk Reporting Monitoring / Audit Business security Business rules Board Other DBs SAP BO Semantic Layer Data Sources Semantic Layer
  • 13.
    13 Year 1 -The Logical Data Warehouse ▪ Unique point of query ▪ “Need data? LDW has the answer!” ▪ For reporting, analytics, APIs, … ▪ Unique point of truth ▪ Business logic repository ▪ Lineage available ▪ Unique point of access control ▪ Unified access to the data ▪ Unique point of auditing KPI DB Source DBs New DWH Old DWH Markets DB BO reporting Self-service BI MS Office Integration General Reporting KPI Self-Service data Analytics Reports Analytics Server Risk Reporting Board Other DBs Data Sources Logical Data Warehouse w/ Denodo Monitoring / Audit Business security Business rules PDF statements
  • 14.
    14 Years 2 and3 - Expansion and Modernization ▪ Addition of data consumers ▪ Tableau ▪ REST / Restful APIs ▪ Addition of more data sources ▪ Where ETL is not required ▪ When history is provided in source ▪ Logical data pipelines ▪ Reduces the number of ETL jobs ▪ EDW gets data from LDW BO Reporting Tableau RestWS to Excel General Reporting KPI Self-Service data Analytics Reports Analytics Server Risk Reporting Board Data Sources Logical Data Warehouse w/ Denodo KPI DB Source DBs New DWH Old DWH Markets DB Other DBs Flat files Excel SaaS REST SOAP WWW Customers Domains Operational systems Monitoring / Audit Business security Business rules Customer statements
  • 15.
  • 16.
    16 Leading Global Bank– Pain Points Supporting Multiple Data Access Tools Changing Technologies Data Lifecycle Management Data Discovery
  • 17.
    17 Leading Global Bank– Data Marketplace Objectives DATA DISCOVERY SECURITY CONSISTENT ACCESS INTERFACE REDUCE BARRIERS TO ADOPTION DATA ACCESS AGILITY 02 03 04 05 06 01
  • 18.
    18 Leading Global Bank– Data Marketplace Data Virtualization Platform Data Marketplace Client & Account Active Clients Client Accounts Party Summary Positions & Holdings Securities & Pricing Market Data Hub Index & Benchmark Systems of Record Data Lake Data Warehouse with Business Semantic Layer Virtual Data Lake
  • 19.
  • 20.
    20 About BHP Company Profileand Background • Anglo-Australian multinational mining, metals and petroleum dual-listed public company headquartered in Melbourne, Victoria, Australia. • BHP ranked as the world's largest mining company, based on market capitalization, and as Melbourne's third-largest company by revenue, • BHP has mining operations in Australia, North America, and South America, and petroleum operations in the U.S., Australia, Trinidad and Tobago, UK, and Algeria. • The company has four primary operational units • Coal • Copper • Iron ore • Petroleum • No of Employees : 80,000 • Revenue : US$65.098 billion (2022)
  • 21.
    21 BHP – GloballyDistributed Data and Users Houston DC Santiago DC Perth DC Brisbane AWS US East Escondida Jansen London Singapore Kuala Lumpor Shanghai AWS APAC
  • 22.
    22 BHP – GlobalData Fabric Houston DC Santiago DC Perth DC Brisbane AWS US East AWS APAC Escondida Jansen London Singapore Kuala Lumpor Shanghai Every Data Virtualization cluster is connected to local data sources, and is the access point for local consumer apps such as BI and analytics tools. Each Data Virtualization cluster has visibility of the datasets available from all other clusters, and requests this data from it's peer cluster as required by end users
  • 23.
    23 BHP – GlobalData Fabric Infrastructure
  • 24.
    24 Denodo Platform: TheFoundation of a Logical Data Architecture Agile Data Integration Logical Data Abstraction Smart Query Acceleration Advanced Semantics Automation & Recommendation Unified Security & Governance Data Catalog AI/ML 6 Key Capabilities of Logical Data Management Differentiated Use Cases Hybrid/Multi-Cloud Data Integration Data Marketplace/ Self-Service Analytics Governance & Compliance 3600 View of Entities (e.g., Customer) Accelerated Integration for M&A Activities Data Democratization Enterprise Data Services Data Fabric/ Data Mesh
  • 25.
    25 Benefits of aLogical Data Architecture “Now, we can do weekly releases. We’re able to add new data sources within 2 to 3 hours. We’re about 60% faster than we were in the old world.” VP of data and analytics, real estate “To me, it all boils down to speed to insights. Not having to wait to get the question that you have top-of-mind answered with data is huge.” VP of data and analytics, real estate
  • 26.