SlideShare a Scribd company logo
1 of 22
Download to read offline
1
Real life customer cases using Data Vault and
Data Warehouse automation
Dirk Vermeiren – Partner Tripwire Solutions
Historical – Milestone projects
Health Sector1
2009
Data Vault
Rule 1 – Do not implement a Data Vault DWH
without DWH Automation
¡ Why ?
¡ You have 3 to 4 times more objects than 3NF, meaning
much more manual development work.
¡ Data Vault objects have generic logic per type (HUB,
SATELLITE & LINK) and there are lots of them.
¡ Therefore code generation can be used to deliver
higher development speeds.
Rule 2 – Do not create a DV model that holds
the single version of the Truth in your first layer
of your DWH
¡ Why ?
¡ Single version of the Truth :
¡ Is defined by business and changes faster over
time than the source systems
¡ The single version of the truth is a myth.
¡ As business definitions changes over time you will
also get multiple versions of the truth over time
Rule 3 – Do not limit what you record in the DV
based on user requirements
¡ Why ?
¡ End users can not predict everything they need and
they want to be able change their mind on what is
needed.
Historical – Milestone projects
Health Sector1
2009
Data Vault
Health Sector2
2010
Data Vault
DWH Automation 1.0
Healthcare Sector Project 2
¡  Create a foundation layer the holds :
¡  the Single version of the Facts = Stores data in Source system format
¡  Atomic level data
¡  All data from source except for Interface or other technical tables
¡  All History of change
¡  Integrates data across sources
¡  Use a data Vault modeling which is flexible and resilient to change.
¡  Use etl-generation = OWB OMB-code generation
¡  Important : Reuse of investment of existing ETL-tool is important so
the automation tool should generate Mappings, not replace the
existing tool.
¡  Create a presentation layer
¡  Generate the incremental logic from foundation to presentation layer.
¡  Manual development.
¡  That structures the data in a way end users understands it.
Data Flow – HC Sector Project 2
Rule 4 – Do not automate incremental logic
towards PL
¡ Why ?
¡ Generic increment logic can not take in account that
there are driving tables, which means all tables are
driving tables in the load logic and this has a huge
impact on the performance.
¡ Exception :
¡ Use Engineered systems to run this logic
¡ Use Engineered systems & in Memory technology
to virtualize the Presentation Layer.
Rule 5 – Do not implement all business logic
From Foundation layer to Presentation layer
¡ Atomic level objects that do not exist in the source
should not be placed in the presentation layer
¡ Why ?
¡ They are typically used in business logic to build
multiple Presentation Layer object
¡ Best to persist them before the PL, otherwise you
have to implement the logic to load them,
multiple times
Historical – Milestone projects
Health Sector1
2009
Data Vault
Health Sector2
2010
Data Vault
DWH Automation 1.0
Bank project
¡ Introduced new DV features in DWH Automation tool
¡ Transactional Links
¡ Same as Logic
¡ Splitting Satellites over Multiple Satellites
¡ More customization so more customer standards
could be supported
Historical – Milestone projects
Health Sector1
2009
Data Vault
Health Sector2
2010
Data Vault
DWH Automation 1.0
The Agile Information Factory
i
Architecture & Approach
q  Innovation
ü  Supports all new
concepts in
Information
management
q  Delivers Value
ü  Agility
ü  Cost Reduction
q  Best Practices
ü  Reuse of approach &
solutions
q  Oracle Platform
ü  Uses Integrated
Software/Hardware
stack of Oracle
DWH-Automation Solution 3.0
• Tripwire
DWH Foundation Accelerator
Analysis
Source Analysis Automation
• Tripwire
DWH Foundation Accelerator
Development
Etl-Code
Generation
• Tripwire
DWH Foundation Accelerator
Testing
Automated
Data Validation
• Redbridge
Lifecycle Management
Automated Release
Management
• Oracle Enterprise Metada
Management
Impact Analysis
Enterprise Metadata
Management
Raw and Business DV
¡  In the foundation layer there are actually 2 persistent layers
(typically stored in 1 schema)
¡  RDV : Raw integration – none to simple business key integration
– the data does not represent common business rules
¡  BDV : Business Data Vault
¡  Business rules are applied
¡  Business key integration takes place
¡  New Business Concept introduced
¡  Data Virtualization of exiting business concepts in the
Raw Vault – Do not persist objects that already exist in
the Raw Data Vault
Foundation Area : The internal Layers
Multiple speed Implementation
¡  The Raw and the Business Data Vault area can be built at different speeds
because :
¡  The RAW or Source based Data Vault is :
¡  A technical implementation based on source systems and only requires a
source analysis = Single version of the fact
¡  Data Warehouse automation can be used as the target structure is a direct
representation of the source.
¡  The Business Data Vault is :
¡  A Business based implementation that requires functional and technical
analysis to understand business requirements = Single or multiple versions of
the truth
¡  New Business Concepts can be created (New Hubs) but implementation
experience show typically link tables between existing Source Business
concepts (source based Hubs) support requirements for 90%.
¡  The multiple speed approach supports better functional and technical
analysis when the raw data vault data is already available.
Rule 6 – Put the right business logic in the right
layer.
¡  If you do not standardize than you will have to document
everything
¡  Supports the multiple speed approach
¡  Increases the ability to change without high impact.
¡ 
Parameters to define where to place which Business logic :
¡  Stability : Is this logic likely to change a lot over time
¡  Scope : Enterprise wide, Departmental, User specific
¡  Type : Conditional, Calculation, Aggregation, Data Quality Check, …
¡  Result : Factual, Master Data
For questions :
Piet De Windt
piet.de.windt@tripwiresolutions.be
+32 473 99 99 89
Everything you need to build
something exceptional

More Related Content

What's hot

Apache Hadoop India Summit 2011 talk "Data Integration on Hadoop" by Sanjay K...
Apache Hadoop India Summit 2011 talk "Data Integration on Hadoop" by Sanjay K...Apache Hadoop India Summit 2011 talk "Data Integration on Hadoop" by Sanjay K...
Apache Hadoop India Summit 2011 talk "Data Integration on Hadoop" by Sanjay K...
Yahoo Developer Network
 

What's hot (20)

Testing the Data Warehouse—Big Data, Big Problems
Testing the Data Warehouse—Big Data, Big ProblemsTesting the Data Warehouse—Big Data, Big Problems
Testing the Data Warehouse—Big Data, Big Problems
 
Deliver Trusted Data by Leveraging ETL Testing
Deliver Trusted Data by Leveraging ETL TestingDeliver Trusted Data by Leveraging ETL Testing
Deliver Trusted Data by Leveraging ETL Testing
 
Modern Data Platforms
Modern Data Platforms Modern Data Platforms
Modern Data Platforms
 
Data Science Operationalization: The Journey of Enterprise AI
Data Science Operationalization: The Journey of Enterprise AIData Science Operationalization: The Journey of Enterprise AI
Data Science Operationalization: The Journey of Enterprise AI
 
Global ai conf_final
Global ai conf_finalGlobal ai conf_final
Global ai conf_final
 
Talend MDM
Talend MDMTalend MDM
Talend MDM
 
Rethink Your Data Governance - POPI Act Compliance Made Easy with Data Virtua...
Rethink Your Data Governance - POPI Act Compliance Made Easy with Data Virtua...Rethink Your Data Governance - POPI Act Compliance Made Easy with Data Virtua...
Rethink Your Data Governance - POPI Act Compliance Made Easy with Data Virtua...
 
Neo4j GraphTalks - Einführung in Graphdatenbanken
Neo4j GraphTalks - Einführung in GraphdatenbankenNeo4j GraphTalks - Einführung in Graphdatenbanken
Neo4j GraphTalks - Einführung in Graphdatenbanken
 
Modernize & Automate Analytics Data Pipelines
Modernize & Automate Analytics Data PipelinesModernize & Automate Analytics Data Pipelines
Modernize & Automate Analytics Data Pipelines
 
Postgres Vision 2018: How to Consume your Database Platform On-premises
Postgres Vision 2018: How to Consume your Database Platform On-premisesPostgres Vision 2018: How to Consume your Database Platform On-premises
Postgres Vision 2018: How to Consume your Database Platform On-premises
 
Data Services and the Modern Data Ecosystem (ASEAN)
Data Services and the Modern Data Ecosystem (ASEAN)Data Services and the Modern Data Ecosystem (ASEAN)
Data Services and the Modern Data Ecosystem (ASEAN)
 
Using OBIEE and Data Vault to Virtualize Your BI Environment: An Agile Approach
Using OBIEE and Data Vault to Virtualize Your BI Environment: An Agile ApproachUsing OBIEE and Data Vault to Virtualize Your BI Environment: An Agile Approach
Using OBIEE and Data Vault to Virtualize Your BI Environment: An Agile Approach
 
How to Realize an Additional 270% ROI on Snowflake
How to Realize an Additional 270% ROI on SnowflakeHow to Realize an Additional 270% ROI on Snowflake
How to Realize an Additional 270% ROI on Snowflake
 
Webinar: It's the 21st Century - Why Isn't Your Data Integration Loosely Coup...
Webinar: It's the 21st Century - Why Isn't Your Data Integration Loosely Coup...Webinar: It's the 21st Century - Why Isn't Your Data Integration Loosely Coup...
Webinar: It's the 21st Century - Why Isn't Your Data Integration Loosely Coup...
 
DataStax: Making a Difference with Smart Analytics
DataStax: Making a Difference with Smart AnalyticsDataStax: Making a Difference with Smart Analytics
DataStax: Making a Difference with Smart Analytics
 
PgConf 2018 - Postgres in a World of DevOps
PgConf 2018 - Postgres in a World of DevOpsPgConf 2018 - Postgres in a World of DevOps
PgConf 2018 - Postgres in a World of DevOps
 
Apache Hadoop India Summit 2011 talk "Data Integration on Hadoop" by Sanjay K...
Apache Hadoop India Summit 2011 talk "Data Integration on Hadoop" by Sanjay K...Apache Hadoop India Summit 2011 talk "Data Integration on Hadoop" by Sanjay K...
Apache Hadoop India Summit 2011 talk "Data Integration on Hadoop" by Sanjay K...
 
Altis AWS Snowflake Practice
Altis AWS Snowflake PracticeAltis AWS Snowflake Practice
Altis AWS Snowflake Practice
 
The Power Of Snowflake for SAP BusinessObjects
The Power Of Snowflake for SAP BusinessObjectsThe Power Of Snowflake for SAP BusinessObjects
The Power Of Snowflake for SAP BusinessObjects
 
Snowflake: The most cost-effective agile and scalable data warehouse ever!
Snowflake: The most cost-effective agile and scalable data warehouse ever!Snowflake: The most cost-effective agile and scalable data warehouse ever!
Snowflake: The most cost-effective agile and scalable data warehouse ever!
 

Viewers also liked

Information Lifecycle Governance Leader Reference Guide
Information Lifecycle Governance Leader Reference GuideInformation Lifecycle Governance Leader Reference Guide
Information Lifecycle Governance Leader Reference Guide
Dan D'Angelo
 
Estrategia Information lifecycle Management
Estrategia Information lifecycle ManagementEstrategia Information lifecycle Management
Estrategia Information lifecycle Management
Jaime Contreras
 
Ilm library information lifecycle management best practices guide sg247251
Ilm library information lifecycle management best practices guide sg247251Ilm library information lifecycle management best practices guide sg247251
Ilm library information lifecycle management best practices guide sg247251
Banking at Ho Chi Minh city
 

Viewers also liked (20)

Metadaten und Data Vault (Meta Vault)
Metadaten und Data Vault (Meta Vault)Metadaten und Data Vault (Meta Vault)
Metadaten und Data Vault (Meta Vault)
 
Das modulare DWH-Modell - DOAG SIG BI/DWH 2010 - OPITZ CONSULTING - ArnoTigges
Das modulare DWH-Modell - DOAG SIG BI/DWH 2010 - OPITZ CONSULTING - ArnoTiggesDas modulare DWH-Modell - DOAG SIG BI/DWH 2010 - OPITZ CONSULTING - ArnoTigges
Das modulare DWH-Modell - DOAG SIG BI/DWH 2010 - OPITZ CONSULTING - ArnoTigges
 
Part 4 - Data Warehousing Lecture at BW Cooperative State University (DHBW)
Part 4 - Data Warehousing Lecture at BW Cooperative State University (DHBW)Part 4 - Data Warehousing Lecture at BW Cooperative State University (DHBW)
Part 4 - Data Warehousing Lecture at BW Cooperative State University (DHBW)
 
Hadoop 2.0 - The Next Level
Hadoop 2.0 - The Next LevelHadoop 2.0 - The Next Level
Hadoop 2.0 - The Next Level
 
Part 3 - Data Warehousing Lecture at BW Cooperative State University (DHBW)
Part 3 - Data Warehousing Lecture at BW Cooperative State University (DHBW)Part 3 - Data Warehousing Lecture at BW Cooperative State University (DHBW)
Part 3 - Data Warehousing Lecture at BW Cooperative State University (DHBW)
 
Creating Better Customer Experiences Online (with Top Tasks) presented by Ger...
Creating Better Customer Experiences Online (with Top Tasks) presented by Ger...Creating Better Customer Experiences Online (with Top Tasks) presented by Ger...
Creating Better Customer Experiences Online (with Top Tasks) presented by Ger...
 
Pedro De Bruyckere Meetup Presentation
Pedro De Bruyckere Meetup PresentationPedro De Bruyckere Meetup Presentation
Pedro De Bruyckere Meetup Presentation
 
How business analysts are catalysts for business change
How business analysts are catalysts for business changeHow business analysts are catalysts for business change
How business analysts are catalysts for business change
 
3D printing en korte keten recyclage (Evi Swinnen, timelab)
3D printing en korte keten recyclage (Evi Swinnen, timelab)3D printing en korte keten recyclage (Evi Swinnen, timelab)
3D printing en korte keten recyclage (Evi Swinnen, timelab)
 
Google Glass UX Best Practices Presentation by Litrik De Roy (@litrik) at the...
Google Glass UX Best Practices Presentation by Litrik De Roy (@litrik) at the...Google Glass UX Best Practices Presentation by Litrik De Roy (@litrik) at the...
Google Glass UX Best Practices Presentation by Litrik De Roy (@litrik) at the...
 
Smarter Eduction - Higher Education Summit 2011 - D Watt
Smarter Eduction - Higher Education Summit 2011 - D WattSmarter Eduction - Higher Education Summit 2011 - D Watt
Smarter Eduction - Higher Education Summit 2011 - D Watt
 
Data Vault Introduction
Data Vault IntroductionData Vault Introduction
Data Vault Introduction
 
Information Lifecycle Governance Leader Reference Guide
Information Lifecycle Governance Leader Reference GuideInformation Lifecycle Governance Leader Reference Guide
Information Lifecycle Governance Leader Reference Guide
 
Trends for 2014
Trends for 2014Trends for 2014
Trends for 2014
 
Estrategia Information lifecycle Management
Estrategia Information lifecycle ManagementEstrategia Information lifecycle Management
Estrategia Information lifecycle Management
 
Information Lifecycle Management
Information Lifecycle ManagementInformation Lifecycle Management
Information Lifecycle Management
 
Creating a Smarter Shopping Experience with IBM Solutions at Carter's
Creating a Smarter Shopping Experience with IBM Solutions at Carter'sCreating a Smarter Shopping Experience with IBM Solutions at Carter's
Creating a Smarter Shopping Experience with IBM Solutions at Carter's
 
Het huis de school van morgen (Martine Tempels, Telenet)
Het huis de school van morgen (Martine Tempels, Telenet)Het huis de school van morgen (Martine Tempels, Telenet)
Het huis de school van morgen (Martine Tempels, Telenet)
 
Ilm library information lifecycle management best practices guide sg247251
Ilm library information lifecycle management best practices guide sg247251Ilm library information lifecycle management best practices guide sg247251
Ilm library information lifecycle management best practices guide sg247251
 
Experiences from a Data Vault Pilot Exploiting the Internet of Things
Experiences from a Data Vault Pilot Exploiting the Internet of ThingsExperiences from a Data Vault Pilot Exploiting the Internet of Things
Experiences from a Data Vault Pilot Exploiting the Internet of Things
 

Similar to Real-life Customer Cases using Data Vault and Data Warehouse Automation

Feature Store as a Data Foundation for Machine Learning
Feature Store as a Data Foundation for Machine LearningFeature Store as a Data Foundation for Machine Learning
Feature Store as a Data Foundation for Machine Learning
Provectus
 
Creating Your Data Governance Dashboard
Creating Your Data Governance DashboardCreating Your Data Governance Dashboard
Creating Your Data Governance Dashboard
Trillium Software
 
Experiences from a Data Vault Pilot Exploiting the Internet of Things
Experiences from a Data Vault Pilot Exploiting the Internet of ThingsExperiences from a Data Vault Pilot Exploiting the Internet of Things
Experiences from a Data Vault Pilot Exploiting the Internet of Things
GuyVanderSande
 

Similar to Real-life Customer Cases using Data Vault and Data Warehouse Automation (20)

Feature Store as a Data Foundation for Machine Learning
Feature Store as a Data Foundation for Machine LearningFeature Store as a Data Foundation for Machine Learning
Feature Store as a Data Foundation for Machine Learning
 
Meetup 25/04/19: Big Data
Meetup 25/04/19: Big DataMeetup 25/04/19: Big Data
Meetup 25/04/19: Big Data
 
Self-Service Analytics with Guard Rails
Self-Service Analytics with Guard RailsSelf-Service Analytics with Guard Rails
Self-Service Analytics with Guard Rails
 
2015 02 12 talend hortonworks webinar challenges to hadoop adoption
2015 02 12 talend hortonworks webinar challenges to hadoop adoption2015 02 12 talend hortonworks webinar challenges to hadoop adoption
2015 02 12 talend hortonworks webinar challenges to hadoop adoption
 
Creating Your Data Governance Dashboard
Creating Your Data Governance DashboardCreating Your Data Governance Dashboard
Creating Your Data Governance Dashboard
 
Querona Presentation 2018
Querona Presentation 2018Querona Presentation 2018
Querona Presentation 2018
 
Experiences from a Data Vault Pilot Exploiting the Internet of Things
Experiences from a Data Vault Pilot Exploiting the Internet of ThingsExperiences from a Data Vault Pilot Exploiting the Internet of Things
Experiences from a Data Vault Pilot Exploiting the Internet of Things
 
Data Engineer's Lunch #85: Designing a Modern Data Stack
Data Engineer's Lunch #85: Designing a Modern Data StackData Engineer's Lunch #85: Designing a Modern Data Stack
Data Engineer's Lunch #85: Designing a Modern Data Stack
 
Amit_Kumar_CV
Amit_Kumar_CVAmit_Kumar_CV
Amit_Kumar_CV
 
Big Data Expo 2015 - Hortonworks Common Hadoop Use Cases
Big Data Expo 2015 - Hortonworks Common Hadoop Use CasesBig Data Expo 2015 - Hortonworks Common Hadoop Use Cases
Big Data Expo 2015 - Hortonworks Common Hadoop Use Cases
 
Optimizing your Modern Data Architecture - with Attunity, RCG Global Services...
Optimizing your Modern Data Architecture - with Attunity, RCG Global Services...Optimizing your Modern Data Architecture - with Attunity, RCG Global Services...
Optimizing your Modern Data Architecture - with Attunity, RCG Global Services...
 
Shraddha Sharma
Shraddha SharmaShraddha Sharma
Shraddha Sharma
 
Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)
Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)
Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)
 
Making the Case for Legacy Data in Modern Data Analytics Platforms
Making the Case for Legacy Data in Modern Data Analytics PlatformsMaking the Case for Legacy Data in Modern Data Analytics Platforms
Making the Case for Legacy Data in Modern Data Analytics Platforms
 
The Shifting Landscape of Data Integration
The Shifting Landscape of Data IntegrationThe Shifting Landscape of Data Integration
The Shifting Landscape of Data Integration
 
VMworld 2013: Building the Management Stack for Your Software Defined Data Ce...
VMworld 2013: Building the Management Stack for Your Software Defined Data Ce...VMworld 2013: Building the Management Stack for Your Software Defined Data Ce...
VMworld 2013: Building the Management Stack for Your Software Defined Data Ce...
 
Is it sensible to use Data Vault at all? Conclusions from a project.
Is it sensible to use Data Vault at all? Conclusions from a project.Is it sensible to use Data Vault at all? Conclusions from a project.
Is it sensible to use Data Vault at all? Conclusions from a project.
 
Canadian Experts Discuss Modern Data Stacks and Cloud Computing for 5 Years o...
Canadian Experts Discuss Modern Data Stacks and Cloud Computing for 5 Years o...Canadian Experts Discuss Modern Data Stacks and Cloud Computing for 5 Years o...
Canadian Experts Discuss Modern Data Stacks and Cloud Computing for 5 Years o...
 
Open Sourcing GemFire - Apache Geode
Open Sourcing GemFire - Apache GeodeOpen Sourcing GemFire - Apache Geode
Open Sourcing GemFire - Apache Geode
 
An Introduction to Apache Geode (incubating)
An Introduction to Apache Geode (incubating)An Introduction to Apache Geode (incubating)
An Introduction to Apache Geode (incubating)
 

More from Patrick Van Renterghem

More from Patrick Van Renterghem (20)

Ethical AI at VDAB, presented by Vincent Buekenhout (Ethical AI Lead, VDAB) a...
Ethical AI at VDAB, presented by Vincent Buekenhout (Ethical AI Lead, VDAB) a...Ethical AI at VDAB, presented by Vincent Buekenhout (Ethical AI Lead, VDAB) a...
Ethical AI at VDAB, presented by Vincent Buekenhout (Ethical AI Lead, VDAB) a...
 
Implementing error-proof, business-critical Machine Learning, presentation by...
Implementing error-proof, business-critical Machine Learning, presentation by...Implementing error-proof, business-critical Machine Learning, presentation by...
Implementing error-proof, business-critical Machine Learning, presentation by...
 
Building Trust and Explainability into Chatbots: the Partena Ziekenfonds Busi...
Building Trust and Explainability into Chatbots: the Partena Ziekenfonds Busi...Building Trust and Explainability into Chatbots: the Partena Ziekenfonds Busi...
Building Trust and Explainability into Chatbots: the Partena Ziekenfonds Busi...
 
AI & Ethics: The Belgian Industry Vision & Initiatives, presentation by Jelle...
AI & Ethics: The Belgian Industry Vision & Initiatives, presentation by Jelle...AI & Ethics: The Belgian Industry Vision & Initiatives, presentation by Jelle...
AI & Ethics: The Belgian Industry Vision & Initiatives, presentation by Jelle...
 
Responsible AI: An Example AI Development Process with Focus on Risks and Con...
Responsible AI: An Example AI Development Process with Focus on Risks and Con...Responsible AI: An Example AI Development Process with Focus on Risks and Con...
Responsible AI: An Example AI Development Process with Focus on Risks and Con...
 
Fairness and Transparency: Algorithmic Explainability, some Legal and Ethical...
Fairness and Transparency: Algorithmic Explainability, some Legal and Ethical...Fairness and Transparency: Algorithmic Explainability, some Legal and Ethical...
Fairness and Transparency: Algorithmic Explainability, some Legal and Ethical...
 
How obedient digital twins and intelligent beings contribute to ethics and ex...
How obedient digital twins and intelligent beings contribute to ethics and ex...How obedient digital twins and intelligent beings contribute to ethics and ex...
How obedient digital twins and intelligent beings contribute to ethics and ex...
 
He Said, She Said: Finding and Fixing Bias in NLP (Natural Language Processin...
He Said, She Said: Finding and Fixing Bias in NLP (Natural Language Processin...He Said, She Said: Finding and Fixing Bias in NLP (Natural Language Processin...
He Said, She Said: Finding and Fixing Bias in NLP (Natural Language Processin...
 
Introduction to Bias in Machine Learning, presented by Matthias Feys, CTO @ M...
Introduction to Bias in Machine Learning, presented by Matthias Feys, CTO @ M...Introduction to Bias in Machine Learning, presented by Matthias Feys, CTO @ M...
Introduction to Bias in Machine Learning, presented by Matthias Feys, CTO @ M...
 
Business Case: Ozitem Groupe, where 80% of the company is working remotely. R...
Business Case: Ozitem Groupe, where 80% of the company is working remotely. R...Business Case: Ozitem Groupe, where 80% of the company is working remotely. R...
Business Case: Ozitem Groupe, where 80% of the company is working remotely. R...
 
Digital Workplace Case Study: How the Municipality of Duffel successfully swi...
Digital Workplace Case Study: How the Municipality of Duffel successfully swi...Digital Workplace Case Study: How the Municipality of Duffel successfully swi...
Digital Workplace Case Study: How the Municipality of Duffel successfully swi...
 
Unleashing the Full Potential of People, Teams and SOLVAY, presented by Bruce...
Unleashing the Full Potential of People, Teams and SOLVAY, presented by Bruce...Unleashing the Full Potential of People, Teams and SOLVAY, presented by Bruce...
Unleashing the Full Potential of People, Teams and SOLVAY, presented by Bruce...
 
The Building Blocks of a Digital Workplace, presented by Sam Marshall at the ...
The Building Blocks of a Digital Workplace, presented by Sam Marshall at the ...The Building Blocks of a Digital Workplace, presented by Sam Marshall at the ...
The Building Blocks of a Digital Workplace, presented by Sam Marshall at the ...
 
Engie's Digital Workplace and "Connecting the company" business case, present...
Engie's Digital Workplace and "Connecting the company" business case, present...Engie's Digital Workplace and "Connecting the company" business case, present...
Engie's Digital Workplace and "Connecting the company" business case, present...
 
Face your communication challenges when implementing a digital workplace, bas...
Face your communication challenges when implementing a digital workplace, bas...Face your communication challenges when implementing a digital workplace, bas...
Face your communication challenges when implementing a digital workplace, bas...
 
The first steps in Recticel's Digital Workplace program by Kenneth Meuleman (...
The first steps in Recticel's Digital Workplace program by Kenneth Meuleman (...The first steps in Recticel's Digital Workplace program by Kenneth Meuleman (...
The first steps in Recticel's Digital Workplace program by Kenneth Meuleman (...
 
Presentation by Dave Geentjens at the "Successful Digital Workplace Adoption"...
Presentation by Dave Geentjens at the "Successful Digital Workplace Adoption"...Presentation by Dave Geentjens at the "Successful Digital Workplace Adoption"...
Presentation by Dave Geentjens at the "Successful Digital Workplace Adoption"...
 
Tim scottkoenverheyenpresentation
Tim scottkoenverheyenpresentationTim scottkoenverheyenpresentation
Tim scottkoenverheyenpresentation
 
Presentation by Ivan Schotsmans (DV Community) at the Data Vault Modelling an...
Presentation by Ivan Schotsmans (DV Community) at the Data Vault Modelling an...Presentation by Ivan Schotsmans (DV Community) at the Data Vault Modelling an...
Presentation by Ivan Schotsmans (DV Community) at the Data Vault Modelling an...
 
Presentation by Luc Delanglez (DataLumen) at the Data Vault Modelling and Dat...
Presentation by Luc Delanglez (DataLumen) at the Data Vault Modelling and Dat...Presentation by Luc Delanglez (DataLumen) at the Data Vault Modelling and Dat...
Presentation by Luc Delanglez (DataLumen) at the Data Vault Modelling and Dat...
 

Recently uploaded

Exploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptxExploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptx
DilipVasan
 
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Valters Lauzums
 
一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理
cyebo
 
一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理
pyhepag
 
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
pyhepag
 
一比一原版麦考瑞大学毕业证成绩单如何办理
一比一原版麦考瑞大学毕业证成绩单如何办理一比一原版麦考瑞大学毕业证成绩单如何办理
一比一原版麦考瑞大学毕业证成绩单如何办理
cyebo
 
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotecAbortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Riyadh +966572737505 get cytotec
 

Recently uploaded (20)

Exploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptxExploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptx
 
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
 
AI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdfAI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdf
 
Artificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdfArtificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdf
 
Webinar One View, Multiple Systems No-Code Integration of Salesforce and ERPs
Webinar One View, Multiple Systems No-Code Integration of Salesforce and ERPsWebinar One View, Multiple Systems No-Code Integration of Salesforce and ERPs
Webinar One View, Multiple Systems No-Code Integration of Salesforce and ERPs
 
一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理
 
一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理
 
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdfGenerative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
 
Easy and simple project file on mp online
Easy and simple project file on mp onlineEasy and simple project file on mp online
Easy and simple project file on mp online
 
How I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prisonHow I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prison
 
2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting
 
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
 
Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)
 
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
 
Pre-ProductionImproveddsfjgndflghtgg.pptx
Pre-ProductionImproveddsfjgndflghtgg.pptxPre-ProductionImproveddsfjgndflghtgg.pptx
Pre-ProductionImproveddsfjgndflghtgg.pptx
 
一比一原版麦考瑞大学毕业证成绩单如何办理
一比一原版麦考瑞大学毕业证成绩单如何办理一比一原版麦考瑞大学毕业证成绩单如何办理
一比一原版麦考瑞大学毕业证成绩单如何办理
 
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotecAbortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
 
Supply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflictSupply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflict
 
basics of data science with application areas.pdf
basics of data science with application areas.pdfbasics of data science with application areas.pdf
basics of data science with application areas.pdf
 
2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Call2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Call
 

Real-life Customer Cases using Data Vault and Data Warehouse Automation

  • 1. 1 Real life customer cases using Data Vault and Data Warehouse automation Dirk Vermeiren – Partner Tripwire Solutions
  • 2.
  • 3. Historical – Milestone projects Health Sector1 2009 Data Vault
  • 4. Rule 1 – Do not implement a Data Vault DWH without DWH Automation ¡ Why ? ¡ You have 3 to 4 times more objects than 3NF, meaning much more manual development work. ¡ Data Vault objects have generic logic per type (HUB, SATELLITE & LINK) and there are lots of them. ¡ Therefore code generation can be used to deliver higher development speeds.
  • 5. Rule 2 – Do not create a DV model that holds the single version of the Truth in your first layer of your DWH ¡ Why ? ¡ Single version of the Truth : ¡ Is defined by business and changes faster over time than the source systems ¡ The single version of the truth is a myth. ¡ As business definitions changes over time you will also get multiple versions of the truth over time
  • 6. Rule 3 – Do not limit what you record in the DV based on user requirements ¡ Why ? ¡ End users can not predict everything they need and they want to be able change their mind on what is needed.
  • 7. Historical – Milestone projects Health Sector1 2009 Data Vault Health Sector2 2010 Data Vault DWH Automation 1.0
  • 8. Healthcare Sector Project 2 ¡  Create a foundation layer the holds : ¡  the Single version of the Facts = Stores data in Source system format ¡  Atomic level data ¡  All data from source except for Interface or other technical tables ¡  All History of change ¡  Integrates data across sources ¡  Use a data Vault modeling which is flexible and resilient to change. ¡  Use etl-generation = OWB OMB-code generation ¡  Important : Reuse of investment of existing ETL-tool is important so the automation tool should generate Mappings, not replace the existing tool. ¡  Create a presentation layer ¡  Generate the incremental logic from foundation to presentation layer. ¡  Manual development. ¡  That structures the data in a way end users understands it.
  • 9. Data Flow – HC Sector Project 2
  • 10. Rule 4 – Do not automate incremental logic towards PL ¡ Why ? ¡ Generic increment logic can not take in account that there are driving tables, which means all tables are driving tables in the load logic and this has a huge impact on the performance. ¡ Exception : ¡ Use Engineered systems to run this logic ¡ Use Engineered systems & in Memory technology to virtualize the Presentation Layer.
  • 11. Rule 5 – Do not implement all business logic From Foundation layer to Presentation layer ¡ Atomic level objects that do not exist in the source should not be placed in the presentation layer ¡ Why ? ¡ They are typically used in business logic to build multiple Presentation Layer object ¡ Best to persist them before the PL, otherwise you have to implement the logic to load them, multiple times
  • 12. Historical – Milestone projects Health Sector1 2009 Data Vault Health Sector2 2010 Data Vault DWH Automation 1.0
  • 13. Bank project ¡ Introduced new DV features in DWH Automation tool ¡ Transactional Links ¡ Same as Logic ¡ Splitting Satellites over Multiple Satellites ¡ More customization so more customer standards could be supported
  • 14. Historical – Milestone projects Health Sector1 2009 Data Vault Health Sector2 2010 Data Vault DWH Automation 1.0
  • 15. The Agile Information Factory i Architecture & Approach q  Innovation ü  Supports all new concepts in Information management q  Delivers Value ü  Agility ü  Cost Reduction q  Best Practices ü  Reuse of approach & solutions q  Oracle Platform ü  Uses Integrated Software/Hardware stack of Oracle
  • 16. DWH-Automation Solution 3.0 • Tripwire DWH Foundation Accelerator Analysis Source Analysis Automation • Tripwire DWH Foundation Accelerator Development Etl-Code Generation • Tripwire DWH Foundation Accelerator Testing Automated Data Validation • Redbridge Lifecycle Management Automated Release Management • Oracle Enterprise Metada Management Impact Analysis Enterprise Metadata Management
  • 18. ¡  In the foundation layer there are actually 2 persistent layers (typically stored in 1 schema) ¡  RDV : Raw integration – none to simple business key integration – the data does not represent common business rules ¡  BDV : Business Data Vault ¡  Business rules are applied ¡  Business key integration takes place ¡  New Business Concept introduced ¡  Data Virtualization of exiting business concepts in the Raw Vault – Do not persist objects that already exist in the Raw Data Vault Foundation Area : The internal Layers
  • 19. Multiple speed Implementation ¡  The Raw and the Business Data Vault area can be built at different speeds because : ¡  The RAW or Source based Data Vault is : ¡  A technical implementation based on source systems and only requires a source analysis = Single version of the fact ¡  Data Warehouse automation can be used as the target structure is a direct representation of the source. ¡  The Business Data Vault is : ¡  A Business based implementation that requires functional and technical analysis to understand business requirements = Single or multiple versions of the truth ¡  New Business Concepts can be created (New Hubs) but implementation experience show typically link tables between existing Source Business concepts (source based Hubs) support requirements for 90%. ¡  The multiple speed approach supports better functional and technical analysis when the raw data vault data is already available.
  • 20. Rule 6 – Put the right business logic in the right layer. ¡  If you do not standardize than you will have to document everything ¡  Supports the multiple speed approach ¡  Increases the ability to change without high impact. ¡  Parameters to define where to place which Business logic : ¡  Stability : Is this logic likely to change a lot over time ¡  Scope : Enterprise wide, Departmental, User specific ¡  Type : Conditional, Calculation, Aggregation, Data Quality Check, … ¡  Result : Factual, Master Data
  • 21.
  • 22. For questions : Piet De Windt piet.de.windt@tripwiresolutions.be +32 473 99 99 89 Everything you need to build something exceptional