Temporal Snapshot Fact Tables

SQL Bits X

Temporal Snapshot Fact
Table
A new approach to data snapshots
Davide Mauri
dmauri@solidq.com
Mentor - SolidQ

EXEC sp_help „Davide Mauri‟
• Microsoft SQL Server MVP
• Works with SQL Server from 6.5, on BI from
2003
• Specialized in Data Solution Architecture,
Database Design, Performance Tuning, BI
• President of UGISS (Italian SQL Server UG)
• Mentor @ SolidQ
• Twitter: mauridb
• Blog: http://sqlblog.com/blogs/davide_mauri

Agenda
• The problem
• The possible «classic» solutions
• Limits of «classic» solutions
• A temporal approach
• Implementing the solution
• Technical & Functional Challenges
(and their resolution )
• Conclusions

The request
• Our customer (an insurance company)
needed a BI solution that would allow
them to do some “simple” things:
– Analyze the value/situation of all insurances
at a specific point in time
– Analyze all the data from 1970 onwards
– Analyze data on daily basis
• Absolutely no weekly or monthly aggregation

The environment
• On average they have 3.000.000
documents related to insurances that need
to be stored.
– Each day.
• Data is mainly stored in a DB2 mainframe
– Data is (sometimes) stored in a “temporal”
fashion
• For each document a new row is created each
time something changes
• Each row has “valid_from” and “valid_to” columns

Let‟s do the math
• To keep the daily snapshot
– of 3.000.000 documents
– for 365 days
– for 40 years

• A fact table of near 44 BILLION rows
would be needed

The possible solutions
• How can the problem be solved?
– PDW and/or Brute Force was not an option

• Unfortunately none of the three known
approaches can work here
– Transactional Fact Table
– Periodic Snapshot Fact Table
– Accumulating Snapshot Fact Table

• A new approach is needed!

The classic solutions
And why they cannot be used here

Transaction Fact Table
• One row per fact occurring at a certain point
in time
• We don‟t have facts for every day. For
example, for the document 123, no changes
were made on 20 August 2011
– Analysis on that date would not show the
document 123
– “LastNonEmpty” aggregation cannot help since
dates don‟t have children
– MDX can come to the rescue here but…
• Very high complexity
• Potential performance problems

Accumulating Snapshot F.T.
• One fact row per entity in the fact table
– Each row has a lot of date columns that store
all the changes that happen during the entity‟s
lifetime
– Each column date represents a point in time
where something changed
• The number of changes must be known in
advance
• With so many date columns it can be
difficult to manage the “Analysis Date”
11

Periodic Snapshot Fact Table
• The fact table holds a «snapshot» of all
data for a specific point in time
– That‟s the perfect solution, since doing a
snapshot each day would completely solve
the problem
– Unfortunately there is just too much data
– We need to keep the idea but reduce the
amount of data

BIA-406-S | Temporal Snapshot Fact Table 12

A temporal approach
And how it can be applied to BI

Temporal Data
• The snapshot fact table is a good starting
point
– Keeping a snapshot of a document for each
day is just a waste of space
– And also a big performance killer
• Changes to document don‟t happen too
often
– Fact tables should be temporal in order to
avoid data duplication
– Ideally, each fact row needs a “valid_from”
and “valid_to” column

Temporal Data
• With Temporal Snapshot Fact Table
(TSFT) each row represents a fact that
occurred during a time interval not at a
point in time.
• Time Intervals will be right-opened in order
to simplify calculations:
[valid_from, valid_to)
Which means:
valid_from <= x < valid_to

Temporal Data
• Now that the idea is set, we have to solve
several problems:
– Functional Challenges
• SSAS doesn‟t support the concept of intervals, nor
the between filter
• The user just wants to define the Analysis Date,
and doesn‟t want to deal with ranges
– Technical Challenges
• Source data may come from several tables and
has to be consolidated in a small number of TSFT
• The concept of interval changes the rules of the
“join” game a little bit…

Technical Challenges
• Source data may come from table with
different designs:
temporal tables
non-temporal tables
• Temporal Tables
– Ranges from two different temporal tables
may overlap
• Non-temporal tables
– There may be some business rules that
require us to «break» the existing ranges

• Before starting to solve the problem, let‟s
generalize it a bit (in order to be able to study
it theoretically)
• Imagine a company with this environment
– There are Items & Sub Items
– Items have 1 or more Sub Items
• Each Sub Item has a value in money
• Value may change during its lifetime
– Customers have to pay an amount equal to the
sum of all Sub Items of an Item
• Customers may pay when they want during the year,
as long as the total amount is paid by the end of the
year

• When joining data, keys are no longer
sufficient
• Time Intervals must also be managed
correctly, otherwise incorrect results may
be created
• Luckily time operators are well-known (but
not yet implemented)
– overlaps, contains, meets, ecc. ecc.

• CONTAINS (or DURING) Operator:

b1 e1

i1
b2 e2

i2

• Means:
b1<=b2 AND e1>=e2

• OVERLAPS Operator:
b1 e1

i1
b2 e2

i2

• Means:
b1<=e2 AND b2 <=e1
• Since we‟re using right-opened intervals, a
little change is needed here:
b1<e2 AND b2 <e1

• Before going any further it‟s best we start
to «see» the data we‟re going to use

2008 2009 2010

Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar

Item 1 A B C

Item 1.1 A B C D B C

Item 1.2 A B C D E C

Item 2 A B C D C
Item 2.1 A B C D C

• As you have seen, even with just three
tables, the joins are utterly complex!

• But bearing this in mind, we can now find
an easier, logically equivalent, solution.

• Drawing a timeline and visualizing the
result helps a lot here.


Range for one Item

Range for the Sub Item

Payment Dates Intervals will represent a
So, on the other way round,
time where nothing
dates represent a time in
changed
which something changed!

«Summarized» Status
1 2 3 4 5 6

• In simpler words we need to find the
minimum set of time intervals for each
entity
– The concept of "granularity" must be changed
in order to take into account the idea of time
intervals
– All fact tables, for each stored entity, must
have the same granularity for the time interval
• interval granularity must be the same “horizontally”
(across tables)
• interval granularity must be specific for each entity
(across rows: “vertically”)

• The problem can be solved by
«refactoring» data.
• Just like refactoring code, but working on
data
– Rephrased idea: “disciplined technique for
restructuring existing temporal data, altering
its internal structure without changing its
value”
• http://en.wikipedia.org/wiki/Code_refactoring

• Here‟s how:
• Step 1
– For each item, for
each temporal table,
union all the distinct
ranges

• Step 2
– For each item, take all the
distinct:
– Dates from the previous step
– Dates that are known to
«break» a range
– The first day of each year
used in ranges

• Step 3
– For each item, create
the new ranges, doing
the following steps
• Order the rows by date
• Take the «k» row and
the «k+1» row
• Create the range
[Datek, Datek+1)

The New LEAD operator is exactly
what we need! 

• When the time granularity is the day, the
right-opened range helps a lot here.
– Eg: let‟s say the at some point, for an entity,
you have the following dates that has to be
turned into ranges:

• With a closed range intervals can be
ambiguous, especially single day intervals:

– The meaning is not explicitly clear and we
would have to take care of it, making the ETL
phase more complex.
– It‟s better to make everything explicit

• So, to avoid ambiguity, with a closed range
the resulting set of intervals also needs to
take time into account:

– What granularity to use for time?
• Hour? Minute? Microseconds?
– Is just a waste of space
– Date or Int datatype cannot be used

• With a right-opened interval everything is
easier:

– Date or Int datatype are enough
– No time used, so no need to deal with it

• Step 4
– «Explode» all the temporal source tables so
that all data will use the same (new) ranges,
using the CONTAINS operator:

• The original Range:

Becomes the equivalent
set of ranges:

If we would have chosen a daily
snapshot, 425 rows would have been
generated…not only 12!
(That‟s a 35:1 ratio!)

• Now we have all the source data
conforming to a common set of intervals
– This means that we can just join tables
without having to do complex temporal joins

• All fact tables will also have the same
interval granularity

• We solved all the technical problems! 

Functional Challenges
• SSAS doesn‟t support time intervals but…

• Using some imagination we can say that
– An interval is made of 1 to “n” dates
– A date belongs to 1 to “n” intervals

• We can model this situation with a Many-
To-Many relationship!

• The «interval dimension» will then be
hidden and the user will just see the
«Analysis Date» dimension

• When such date is selected, all the fact
rows that have an interval containing that
date will be selected too, giving us the
solution we‟re looking for 


Date The user wants
Dimension to analyze data
on 10 Aug 2009

Fact Table
Factless DateRange
Date-DateRange Dimension

SSAS uses all
10 Aug is the found ranges
contained in two All the (three)
Ranges rows related to
those ranges are
read

The final recap
And a quick recipe to make everything easier

Steps for the DWH
• For each entity:
– Get all “valid_from” and “valid_to” columns
from all tables
– Get all the dates that will break intervals from
all tables
– Take all the gathered dates one time (remove
duplicates) and generate new intervals
– «Unpack» original tables in order to use the
newly generated intervals

Steps for the CUBE
• Generate the usual Date dimension
– Generate a «Date Range» dimension
– Generate a Factless Date-DataRange table
– Generate the fact table with reference to the
Date Range dimension
– Create the M:N relationship

BIA-406-S | Temporal Snapshot Fact Table 47

Conclusion and Improvements
Some ideas for the future

Conclusion and improvements
• The defined pattern can be applied each time
a daily analysis is needed and data is not
additive
– So each time you need to “snapshot” something

• The approach can also be used if source
data is not temporal, as long as you can turn
it into a temporal format

• Performance can be improved by
partitioning the cube by year or even by
month

Conclusion and improvements
• At the end is a very simple solution 

• Everything you‟ve seen is the summary of
several months of work of the Italian
Team. Thanks guys!

Temporal Snapshot Fact Tables

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Temporal Snapshot Fact Tables

Similar to Temporal Snapshot Fact Tables (20)

More from Davide Mauri

More from Davide Mauri (19)

Recently uploaded

Recently uploaded (20)

Temporal Snapshot Fact Tables

Editor's Notes