And its relation to the four dominant scientific
DWH-modeling concepts
Data warehousing in practice
Drs. S.F.J Otten
13-05...
Topics
 About me…
 Business Intelligence
 What is a Data warehouse (DWH)
 DWH – Design strategies
 Data-modeling
 Br...
About me…
 Education
 Highschool (MAVO)
 College (MBO ICT lvl.4)
 Univeristy of Applied
sciences (Avans
Hogeschool, Bu...
Business Intelligence
 Business Intelligence??
 “a way for organizations to understand their internal and external
envir...
What is a Data warehouse (1)
 Data warehouse?? (DWH)
 “a repository where all data relevant to the management of an
orga...
DWH – design strategies (1)
 Enterprise wide DWH-design (Imnon, 2002)
 DWH is designed by using a normalized enterprise ...
DWH – design strategies (2)
DWH – design strategies (3)
DWH – design strategies (3)
Inmon Kimball
 Subject-oriented
 Integrated
 Non-volatile
 Time-variant
 Top-Down
 Integ...
Data-modeling history
Data-modeling – Star/SF - concepts
Concepts
Star-/snowflake-schematic Golfarelli, M., Maio, D., & Rizzi, S. (1998)
Fact-ta...
Data-modeling - star-schematic
• Comprises of a
single fact-table
• Has N-
dimension-tables
• Each tuple in the
fact-table...
Data-modeling - snowflake-schematic
• A normalized
star-schematic
(3NF)
• Dimensions are
split up in to sub
dimensions
• L...
Data-modeling –Star/SF - ETL
• Conventional DWH-
architecture (Star-
/SF-schematic) for
populating a DWH
• RFC has a high
...
Data-modeling – Star/SF – ETL - P.O.A
 Two types of ETL:
 FULL ETL
 Complete transfer of all data in source-systems via...
Data-modeling – Star/SF – Case (1)
 DWH = Snowflake-architecture (3NF)
 Dimension-tables (DimItem,DimInvoice)
 Fact-tab...
Data-modeling – Star/SF – Case (2)
 Solution
 Identify column containing metric “NetValue” in the source-system
(require...
Data-modeling – Star/SF – Case (3)
 Introduce the new metric in your Sales-cube
 Refresh the data source / data source v...
Data-modeling – Datavault - Concepts
Concepts
Data vault (DV) Lindstedt, D., & Graziano, K. (2011)
Data vault The DataVaul...
Data-modeling – Datavault - Schematic
• Comprises of N-
Hub-/Link-
/Satellite-tables
• Hybrid between
3NF/Star-
schematic
...
Date-modeling – Datavault - ETL
• Datavault-ETL-
architecture for
populating a
datavault.
• RFC has no
impact on
existing ...
Data-modeling – Datavault – ETL –
P.O.A
 Two types of ETL:
 FULL ETL
 Complete transfer of all data in source-systems v...
Data-modeling – Datavault – Case (1)
 DWH = Datavault-architecture
 Hub-tables (H_Product,H_Customer,H_Order)
 Link-tab...
Data-modeling – Datavault – Case (2)
 Solution
 Identify column containing metric “NetValue” in the source-system (requi...
Data-modeling – Datavault – Case (3)
Data-modeling – Datavault – Case (4)
 Datavault does not store data in a structure that is suited for
usage in a datacube...
Data-modeling – Anchormodeling -
concepts
Concepts
Anchor modeling (AM) Rönnbäck (2010)
Anchor modeling Anchor modeling is...
Data-modeling – anchormodeling -
schematic
• 6NF-modeling
• Assumption of
AM is that data
changes over
time
• Future proof...
Data-modeling – anchormodeling - ETL
 ETL-procedure has many similarities with DV-ETL-ing
 In DV first the HUBS are fill...
Data-modeling – anchormodeling –
ETL – P.O.A
 In an ANCHOR only the surrogate key is stored.While with
DV in a HUB the su...
BREAK
Practical examples
 Star /SF-schematic
 ETL
 DWH
 Datavault
 ETL
 DWH
 Anchor Modeling
 ETL
 DWH
Summary (1)
 Two main DWH-design-strategies
 Enterprise wide DWH-design
 DWH is designed by using a normalized enterpri...
Summary (2)
 Four main Data-modeling-techniques
 Star-/Snowflake were introduced in the 80’s
 Star-/Snowflake requires ...
Summary (3)
 So, which data-modeling technique comes out as the
winner…
 Well, None, they can co-exist and you should ch...
Thank you
 @Linkedin : http://nl.linkedin.com/in/sjorsotten
 @mail : Sjors.Otten@csb.com
Upcoming SlideShare
Loading in …5
×

BI - Data warehousing in practice

1,540 views

Published on

this presentation covers the following:

* Data warehouse-design strategies
* Data warehouse-modeling techniques
* the points of attention when building ETL-procedures for one of these Data warehouse-modeling techniques

Published in: Technology, Business

BI - Data warehousing in practice

  1. 1. And its relation to the four dominant scientific DWH-modeling concepts Data warehousing in practice Drs. S.F.J Otten 13-05-2014
  2. 2. Topics  About me…  Business Intelligence  What is a Data warehouse (DWH)  DWH – Design strategies  Data-modeling  Brief history in data modeling  Star-schematic  Snowflake-schematic  Datavault  Anchormodeling  Pratical examples  Summary
  3. 3. About me…  Education  Highschool (MAVO)  College (MBO ICT lvl.4)  Univeristy of Applied sciences (Avans Hogeschool, Business Informatics; BSc)  Utrecht University (MBI; MSc)  Utrecht University (Dissertation on BI,DM,PPM; PhD)  Carreer till now..  CSB-System BV/GmbH (privatly held, 500-1000 employees globally) (2010- present)  BI-consultant/architect (Microsoft BI stack)  SQL-Programmer  Expert-role at programmingdepartment for BI-development at HQ  Semantic development
  4. 4. Business Intelligence  Business Intelligence??  “a way for organizations to understand their internal and external environment through the systematic acquisition,collation,analysis, interpretation and exploitation of information” (Watson & Wixom, 2007).
  5. 5. What is a Data warehouse (1)  Data warehouse?? (DWH)  “a repository where all data relevant to the management of an organization is stored and from which knowledge emerges.” (March & Hevner, 2007)  “A data warehouse is a subject-oriented,integrated,time-variant, nonvolatile collection of data in support of management’s decision- making process.”(Inmon, 1992)  Different definitions same goal;  provide data in such a way that it has meaning and can be used in all levels of an organization as input for a decision-making- process
  6. 6. DWH – design strategies (1)  Enterprise wide DWH-design (Imnon, 2002)  DWH is designed by using a normalized enterprise data model From the EDWH data marts for specific business domains are derived  Data mart design (Kimball, 2002)  Hybrid strategy (top-down & bottom-up) for DWH-design  Create datamarts in a bottom-up fasion  Datamart-design conforms to a top-down skeleton/framwork-design which is called the “data warehouse bus”  The EDW = the union of the conformed datamarts
  7. 7. DWH – design strategies (2)
  8. 8. DWH – design strategies (3)
  9. 9. DWH – design strategies (3) Inmon Kimball  Subject-oriented  Integrated  Non-volatile  Time-variant  Top-Down  Integration via assumed Enterprise data model (EDM / 3NF)  Datamarts are derived from EDW  Business-process-oriented  Bottom-up /evolutionary  Dimensional modeling (star- schematic)  Integration via conformed dimensions  Star-schematic enforces query semantics  The sum of the datamarts = the EDW
  10. 10. Data-modeling history
  11. 11. Data-modeling – Star/SF - concepts Concepts Star-/snowflake-schematic Golfarelli, M., Maio, D., & Rizzi, S. (1998) Fact-table A fact is a focus of interest for the decision- making process; typically, it models an event occurring in the enterprise world (e.g., sales and shipments) Dimension-table Dimensions are discrete attributes which determine the minimum granularity adopted to represent facts; typical dimensions for the sale fact are product, store and date Hierarchy Discrete dimension attributes linked by -to- one relationships, and determine how facts may be aggregated and selected significantly for the decision-making process.
  12. 12. Data-modeling - star-schematic • Comprises of a single fact-table • Has N- dimension-tables • Each tuple in the fact-table has a pointer (FK) to each of the dimension-tables • Each dimension- table has columns that correspond to attributes of the specific dimensions(Chaudh uri & Dayal, 1997)
  13. 13. Data-modeling - snowflake-schematic • A normalized star-schematic (3NF) • Dimensions are split up in to sub dimensions • Lesser FK’s in fact-table • Easier maintenance • Possibly better performance due to lesser joins
  14. 14. Data-modeling –Star/SF - ETL • Conventional DWH- architecture (Star- /SF-schematic) for populating a DWH • RFC has a high impact on existing ETL-practice/package and DWH (i.e. request for a new metric) = re-engineering  • Introduction of a new IT-system causes serious rework and headaches 
  15. 15. Data-modeling – Star/SF – ETL - P.O.A  Two types of ETL:  FULL ETL  Complete transfer of all data in source-systems via ETL-packages  Incremental ETL  After FULL ETL , incremental ETL determines the delta and loads it into the DWH.The loading can be :  INSERT records that are not present in the DWH  UPDATE records that have changed values in certain columns o Requires UPDATE-statements need to take into account the keys (primary and foreign) that uniquely identify a record in a table and execute the UPDATE-statement); risky if not entirely clear what the unique identifier is.
  16. 16. Data-modeling – Star/SF – Case (1)  DWH = Snowflake-architecture (3NF)  Dimension-tables (DimItem,DimInvoice)  Fact-table (FactSalesStatistics)  ETL comprises a FULL and INCREMENTAL-load  Client A sends an RFC for an addition in the sales-overview.  Addition = metric “NetValue” per item per invoice  Additional req= metric “NetValue” is present for future data and also for data allready residing in the sales-overview  How would you guys, as future Business-/Technical-consultants / researches approach this case??
  17. 17. Data-modeling – Star/SF – Case (2)  Solution  Identify column containing metric “NetValue” in the source-system (requires in-depth analysis of transactional system)  Add column to fact table “FactSalesStatistics” ([NetValue] [decimal] (x,y) NULL)  Revert to appropriate ETL-package;  Adjust the source-query / source-columns to include the identified column (metric)  Adjust the function that determines the Delta (add identified column)  Adjust the INSERT-command to write the value from the identified source- column  metric “NetValue” in fact-table “FactSalesStatistics”  Adjust the UPDATE-command to update the metric “NetValue” with the value from the identified source-column for the existing data in table “FactSalesStatistics”  VALIDATE…VALIDATE…VALIDATE…the ERP-data and DWH- data (especially in the beginning)
  18. 18. Data-modeling – Star/SF – Case (3)  Introduce the new metric in your Sales-cube  Refresh the data source / data source view to get the metric “NetValue” in the cube-server-environment  Add measure simply by adding the metric in a measuregroup in the sales-cube  Process the cube and the metric should be available for all end- users
  19. 19. Data-modeling – Datavault - Concepts Concepts Data vault (DV) Lindstedt, D., & Graziano, K. (2011) Data vault The DataVault is a detail oriented, historical tracking and uniquely linked set of normalized tables that support one or more functional areas of business. It is scalable and flexible Hub The Hub is intended to represent major identifiable concepts-entities of interest from the real world. It is required that every Hub entity can be denoted by a unique identifier Link The Link represents relationship among Concepts. Both, Hubs and Links may be involved in such relationships Satellite The Satellite is used to associate a Hub (or a Link) with (a data model) attribute
  20. 20. Data-modeling – Datavault - Schematic • Comprises of N- Hub-/Link- /Satellite-tables • Hybrid between 3NF/Star- schematic • Scalable/Flexible • 100% of the data, 100% of the time • Fairly new to DWH-world • Used by large organizations (i.e. D.O.D, ABN-AMRO)
  21. 21. Date-modeling – Datavault - ETL • Datavault-ETL- architecture for populating a datavault. • RFC has no impact on existing ETL- practice/package and DWH; no re-engineering  • Introduction of new IT-system does not cause headaches 
  22. 22. Data-modeling – Datavault – ETL – P.O.A  Two types of ETL:  FULL ETL  Complete transfer of all data in source-systems via ETL-packages  Decomposition of existing tables in to Hubs, Links, and Satellites  Incremental ETL  After FULL ETL , incremental ETL determines the delta and loads it into the DWH.The loading can be :  INSERT records that are not present in the DWH  END-DATING records that are not valid anymore  There is no UPDATING of metric columns in Datavault. Only an End-date update is required
  23. 23. Data-modeling – Datavault – Case (1)  DWH = Datavault-architecture  Hub-tables (H_Product,H_Customer,H_Order)  Link-tables (L_SalesOrder)  Satellite-tables (S_Product_1,S_SalesOrder_1,S_Customer_1)  ETL comprises a FULL and INCREMENTAL-load  ClientA sends an RFC for an addition in the sales-overview.  Addition = metric “NetValue” per item per order  Additional req= metric “NetValue is present for future data and also for data allready residing in the sales-overview  How would you guys, as future Business-/Technical-consultants / researches approach this case??
  24. 24. Data-modeling – Datavault – Case (2)  Solution  Identify column containing metric “NetValue” in the source-system (requires in- depth analysis of transactional system)  Create a new table in the DWH called S_SalesOrder_2 (ProductID,CustomerID,OrderID,LoadDate,NetValue,MD5,Source,EndDate)  Create a new ETL-package  Provide the source-query/ source-columns including the new metric “NetValue”  Create the function that determines the Delta (Keyfields &identified column)  Create the INSERT-command to write the value from the identified source- column  metric “NetValue” in satellite S_SalesOrder_2 with additional values for “ProductID,CustomerID,OrderID,LoadDate,MD5,Source)  Optional: Create EndDate-function (with the help of staging-tables)  VALIDATE…VALIDATE…VALIDATE…the ERP-data and DWH-data (especially in the beginning)
  25. 25. Data-modeling – Datavault – Case (3)
  26. 26. Data-modeling – Datavault – Case (4)  Datavault does not store data in a structure that is suited for usage in a datacube.  A datacube needs a Star-/SF-schematic. Hence, data marts or a “Business vault” is created.  introducting new data in the cube, by using a data mart, is the same as for a Star-/SF-schematic DWH
  27. 27. Data-modeling – Anchormodeling - concepts Concepts Anchor modeling (AM) Rönnbäck (2010) Anchor modeling Anchor modeling is an agile information modeling technique that offers non- destructive extensibility mechanisms. Anchor An anchor represents a set of entities. Attribute Attributes are used to represent properties of anchors Tie tie represents an association between two or more anchor entities and optional knot entities Knot knot is used to represent a fixed, typically small, set of entities that do not change over time
  28. 28. Data-modeling – anchormodeling - schematic • 6NF-modeling • Assumption of AM is that data changes over time • Future proof • Evolution of data model is done through extensions • Modulair • Agile • Bottom up
  29. 29. Data-modeling – anchormodeling - ETL  ETL-procedure has many similarities with DV-ETL-ing  In DV first the HUBS are filled, followed by the LINKS and to finish it of the SATELLITES are filled.  With AM at first the ANCHORS are populated, followed by theTIES and ATTRIBUTES  In addition a metadata-repository is filled with each ETL-run  Like DV, there are only INSERT-statements and END- DATING-procedures.  NO UPDATE-statement  DELETE-statement is only performed when errornous data is loaded for a given batch
  30. 30. Data-modeling – anchormodeling – ETL – P.O.A  In an ANCHOR only the surrogate key is stored.While with DV in a HUB the surrogate key and businesskey are stored together  How is this resolved in an ETL-environment?  Well, when implementing anAM in a database, views are created for each anchor (comprising the anchor and attributes) with an insert-trigger  We can simply populate the anchor and attributes through the view created by the online modeler.  Additional attributes can be loaded in parallel like in DV. For each of those attributes the surrogatekey is resolved by referencing the businesskey-attribute.
  31. 31. BREAK
  32. 32. Practical examples  Star /SF-schematic  ETL  DWH  Datavault  ETL  DWH  Anchor Modeling  ETL  DWH
  33. 33. Summary (1)  Two main DWH-design-strategies  Enterprise wide DWH-design  DWH is designed by using a normalized enterprise data model  From the EDWH data marts for specific business domains are derived  Data mart Design  Create datamarts in a bottom-up fasion  Datamart-design conforms to a top-down skeleton/framwork-design which is called the “data warehouse bus”  The EDW = the union of the conformed datamarts
  34. 34. Summary (2)  Four main Data-modeling-techniques  Star-/Snowflake were introduced in the 80’s  Star-/Snowflake requires re-engineering when introducing new metrics or systems at the source (ETL/DWH). High impact  Not Agile, specs are determined beforehand, traditional way of system development  deliver results slow  hard to expand existing  Datavault / anchor-modeling introduced in early/mid 00’s  Flexible, Scalable data-model, requires no re-engineering when introducing new metrics or systems at the source (ETL/DWH), simply extend/expand. Little to no impact  Agile  fast developemt track due to iterative development start small  deliver results fast Expand  Scale without effort
  35. 35. Summary (3)  So, which data-modeling technique comes out as the winner…  Well, None, they can co-exist and you should choose the one that is suited for your needs,demands, skillset etc.  It is merely a tool for acieving your goal
  36. 36. Thank you  @Linkedin : http://nl.linkedin.com/in/sjorsotten  @mail : Sjors.Otten@csb.com

×