Original: Lean Data Model Storming for the Agile Enterprise

Lean Data Modelstorming
For The Agile Enterprise
Daniel Upton
www.DecisionLab.Net
dupton@DecisionLab.Net
linkedin.com/in/danielupton

The Business Intelligence Promise: Smarter, more fact-based
decision-making as an everyday routine.

Traditional Architectural Trade-Off:
Do you want it quickly, fully featured, or
with high quality? Pick two.
Daniel Upton
DecisionLab dupton@decisionlab.Net

Q: Why does a Data Warehouse Take So Long?
A: Many functional interdependencies result in
mostly sequential tasking
Daniel Upton
Deliver Data

Especially these tasks: Nothing delivered
for multiple sprints. Non-agile.
Daniel Upton
Deliver Data

Sequential Development
Daniel Upton

Gantt View: End to End ModelStormed
Lean Data Hubs Enable Fast Delivery
Daniel Upton
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data

Lean Core Principles
• Focus on customer
• Eliminate waste
• Deliver as fast as possible
• Decide as late as possible
• Optimize the whole
– Sub-optimize some parts
• Modularize, automate, and re-use
Daniel Upton dupton@decisionlab.Net
DecisionLab

End to End ModelStorming
• Dimensional Modelstorming
• Goal: Quickly express a complete logical star schema data model proven to fully satisfy a user
data story’s narrative and it’s detailed acceptance criteria.
– Reference: Agile Data Warehouse Design: from Whiteboard to Star Schema (book by Lawrence Corr &
James Stagnitto)
• Lean Data Modelstorming
• Goal: Quickly express a complete lean data model, mapped upstream directly from actual source
data and proven to then supply this source data downstream directly into a data presentation layer
such as a Star Schema.
– An Original Concept by Daniel Upton (Presenter: 10/15/2016, SoTec Conference)
DecisionLab

Dimensional ModelStorming
In stakeholder meeting…
• Question: Who does what?
– Who, what, where, when, how, why, how much?
• Answers: User Information Stories
• On a whiteboard, draw and refine…
– Event Model Storm
– Dimension ModelStorm
– Hours Later… Event Matrix
Source: Agile Data Warehouse Design: from Whiteboard to Star Schema (book by Corr & Stagnitto)
DecisionLab

User Information Stories
• Alumni Development specialists must know which
donors donate how much and when, in order to ensure
donors receive recognition and benefits.
• Institutional Research must know how many students
enrolled weekly, with what program and standing, to
ensure University meets qualifications for student loans.
• Auditors must know what family relationships exist
between donors and students, and student standing, to
monitor for conflicts of interest and uphold reputation.
DecisionLab

Event ModelStorms
DecisionLab

Dimension ModelStorm
DecisionLab

Events Matrix
DecisionLab

End to End ModelStorm: Lean Data Overview
Daniel Upton
DecisionLab.Net dupton@decisionlab.Net

Traditional Enterprise DW:
Entities = Relations with Dependencies
Daniel Upton

Star Schema: Facts w/ Business Rules Fully
Dependent on Dimensions w/ Business Rules
Daniel Upton

Perfect World Data Flow: Direct to Star Schema
Daniel Upton

Perfect World Data Flow: EDW and Data Marts
Daniel Upton

Daniel Upton, DecisionLab
Real World Data Flow: EDW and Data Marts

Daniel Upton
DecisionLab
Real World Data Flow: EDW and Data Marts

Daniel Upton
DecisionLab
Real World: Even Big Ships Are At Risk

Daniel Upton
DecisionLab
EDW’s are at Risk from Business Volatility.

EDW Development Takes a Long Time
Daniel Upton

Long ago, many decided not to wait for an EDW
Daniel Upton

Traditional Data Mart (Star Schema)
Daniel Upton

Standardizing
Directly on
Star Schema
Forces Tight
Restrictions
on Incoming
Data
Daniel Upton

Real World Data Flow: Direct to Star Schema
Daniel Upton

What’s wrong here?
Daniel Upton

Direct to Star: Begins well, but harder to
sustain over time.
Daniel Upton

Real World Data Flow: Direct to Star Schema
Daniel Upton, DecisionLab.Net

Single Version of the Truth (SVOT)
Source data is reinterpreted and massaged
into a new data model that fixes core truths
about the data, it’s relationships, and the
business, so that one record, one field, one
table, contains THE authoritative data.
• Is SVOT easy? …achievable? …aspirational?
• Tasks for SVOT: Lengthy requirements analysis and successful negotiation
with many stakeholders across an enterprise, then intensive data modeling,
then custom ETL coding, while hoping SVOT remains fixed.
Daniel Upton

Assumptions vs. Experience
• Perfect World Assumption:
• SVOT is universally accurate and stable. Given the amount
of work to achieve SVOT, it needs to be.
• Real World Experience:
• SVOT accuracy can vary from ‘fair to excellent’, it’s scope is
often far from universal across an entire business, and
business changes occur faster than the DW can keep up.
Daniel Upton

Disclosure
• Seemingly trivial changes in the business may require non-trivial
changes to the ETL code and reports.
• Non-trivial business changes often require changes to the
underlying SVOT data model, the ETL code, and reports.
• Major business changes, or just many small ones over time often
leave a SVOT DW in a perpetual state of disarray, with a never-
ending list of critical issues and an excessive focus
on ‘break – fix – redeploy’.
Daniel Upton

SVOT is worth pursuing, but with a different playbook.
Resolution: The DW structure must not be dependent on a static
interpretation of truth. It must not break when new rules and analyses
need to be applied to data, when inaccuracies are discovered late, or
when data sources or business processes change.
Lean Data Warehouse overcomes this big challenge directly by loosely
coupling diverse source data to insulate the DW from changes, by
easily storing all data as it changes over time, and by delaying
decisions on business rules, SVOT, data quality, reporting or analytics
until after the horses are in the barn.
Daniel Upton

Questions?
Daniel Upton

Lean Data Principles
• Eliminate waste: For in-scope source tables, instantiate and load all
records from all attribute and key fields.
• Deliver Fast: Generic design pattern for quickly historizing source data.
• Decide Late:
– Write code for business rules just downstream of Lean Data Hubs, in
order to avoid hard-coding business rules into the core data load.
• Focus on Customer (Pragmatic Design): Scope, design, and load tables
purely based on business needs, regardless of functional constraints in
data sources.
Daniel Upton

Lean Data Principles
Here’s How: Optimize the Whole: The Lean Data Model must have…
• High Cohesion: Hubs have no functional dependencies to other Hubs, thus
can be scoped, design, loaded simultaneously or months or years later.
• Loose Coupling: Hubs link to other Hubs by association, never by
functional dependency (foreign key in a dependent table).
• Accept some Suboptimized Components to Achieve it: Models are
larger, associative links require an added 1-2 table joins for querying across
Hubs.
Daniel Upton

Lean Data Hubs
The Lean Data Hub is a critical architectural component in End-to-End
ModelStorming. It is fundamentally based on Data Vault architecture, with the
following specific Data Vault references:
– Super Charge Your Data Warehouse, by Dan Linstedt, co-edited by
Kent Graziano (2008-2011) http://LearnDataVault.Com
– Modeling the Agile Data Warehouse with Data Vault, by Hans Hultgren
(2012) New Hamilton Press
– Agile Data Warehousing for the Enterprise, by Ralph Hughes (2016)
Elsevier / Morgan Kaufman
Daniel Upton

Lean Data Hubs
Definition:
– “Pattern-based, history-tracking, modular data assets sourced
from highly disparate data, loosely coupled by common business
keys to join core business concepts (ensembles), and leaving
source data otherwise unchanged. It remains flexible and
cohesive, easily configured to support urgent changes in (a)
data sources, (b) business rules, and (c) reporting or analytics
requirements, and it’s loose-coupled design pattern inherently
supports fast, highly parallelized loading by eliminating
dependencies among core business concepts.”
- Daniel Upton, 10/15/2016, SoTec Conference
Daniel Upton

Lean Data Hubs
Definition (continued):
– Core Business Concept (CBC): Equivalent to an Entity in a 3rd
normal form normalized data model.
– Ensemble: Storage of a CBC in one Hub and all associated
Satellites.
– Modularity and Cohesion: Attained on two levels:
• Between Ensembles (Hubs): an associative (loosely-coupled) ensemble modeling
pattern eliminates all functional dependencies between ensembles (Hubs).
• Isolation of Business Rules from Core Data Layer: The virtualization of analytic or
business-rule transformations, downstream of the ensembles, and preferably as views,
prevents changes in those analytics or business rules from compromising the core
ensembles and their loading process.
Daniel Upton

A Lean Data Model Protects The Repository from
Volatile Business Rules That Cause ETL to Break
Daniel Upton

Lean Data Hubs: High Level Architecture
Daniel Upton

High Level View:
Lean Hubs Design is Modular
Daniel Upton

End to End ModelStorm: Lean Data Warehouse:
Detail
Daniel Upton

High Level Summary of Lean Data Modeling Steps:
Watch for the following details in the upcoming visual diagrams
* Addition of control fields
* Modification of primary keys
* Temporary duplication of tables
* Removal of excess fields
* Establishment of Hub vs. Satellite and the Hub-Satellite link
* Creation of Links
* Creation of Hub-Link relationships

Lean Data ModelStorming:
Step 1: Source Data – Fast Profile

Lean Data ModelStorming
Step 2: Prepare New Tables
a. Add two fields to top, set new PK, add two fields to bottom, then duplicate tables

Step 3: Modify for New Ensembles

Step 4: Working Data Model
Credits:
Lean Data Hubs model is based on Data Vault (aka. Hyper Normalized)
design pattern, with credits to these authors…
Super Charge Your Data Warehouse, Dan Linstedt, 2008
LearnDataVault.com (and other books by Mr. Linstedt)
Modeling the Agile Data Warehouse with Data Vault, Hans Hultgren, 2012,
New Hamilton
Agile Data Warehouse Design for the Enterprise, Ralph Hughes, 2016,
Elsevier Inc. (Mr. Hughes originated the term “Hyper Normalalized”.)

In 11th Hour,
Two User Stories Change
• Institutional Research must know how many students
enrolled weekly, with what program and standing and
relationship to a Counselor, to ensure University meets
qualifications for student loans.
• Auditors must know what family relationships exist
between donors and students and Counselors, and
student standing, to monitor for conflicts of interest and
uphold reputation.
DecisionLab

Revisit Step 1: Source Data – Fast Profile

Lean DW ModelStorming
Revisit Step 2: Prepare New Tables

11th Hour Lean ModelStorm Complete
High Cohesion and Loose Coupling
No functional dependencies between free-standing ensembles.
Delivered in ½the normal time. Load in parallel with existing loads.

Extensions and adaptations are all pattern-based.
No dependencies between Hubs (core business concepts), so existing
tables never need refactoring of existing tables, therefore multiple
teams can co-develop simultaneously within one Lean Data Warehouse
without interference with each other.

“Warehouse Your Data Now.
Add Rules and Relationships As Needed”
Daniel Upton

High Level View:
Lean Hubs Design is Modular Design
Daniel Upton

End to End ModelStorm: Lean Data Hubs
Daniel Upton

Lean Data Hubs: Detailed Architecture

With Lean Data Hubs as Infrastructure, we can
easily keep up with
ongoing change,
delivering quickly…
Daniel Upton
…and building data
stability just beneath the
changes.

Waiting for the EDW
Daniel Upton

Gantt: Nothing delivered
for multiple sprints. Non-agile.
Daniel Upton
Deliver Data

Critical Path Delays
Daniel Upton

Traditional Method vs. ModelStormed Lean Data Hubs:
Assume same resources, same levels of skill and effort.
Abbreviations denote chunks of work in Lean Data Hubs.
Daniel Upton

Daniel Upton
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
How should we sequence these little chunks of work?
This way?
Let’s take some hints from Lean Data Principles…
* Loose coupling. Many dependencies are gone now.
* Deliver as fast as possible.
…Ideas?

This way is faster and reflects actual
dependencies. Smaller chunks of work due
to fewer data dependencies.
Daniel Upton
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data

Deliver Data Every Sprint:
With same resources, skill levels and effort,
Modelstormed Lean Data Hubs support more rapid data delivery
Daniel Upton
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data
Deliver
Data

End to End ModelStorming and Lean Data Hubs
Daniel Upton

Lean Data Modelstorming
For The Agile Enterprise
Thank you!
Daniel Upton
www.DecisionLab.Net,
dupton@DecisionLab.Net
linkedin.com/in/danielupton

Daniel Upton

Original: Lean Data Model Storming for the Agile Enterprise

More Related Content

Viewers also liked

Similar to Original: Lean Data Model Storming for the Agile Enterprise

Recently uploaded

Original: Lean Data Model Storming for the Agile Enterprise